Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
1answer
74 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
2answers
88 views

Robots.txt Blocking Blog Tags?

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled? User-agent: ...
2
votes
2answers
900 views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
208 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
1answer
340 views

Googlebot can't access my site when crawling from rootdomain

I can't explain why I get this message for my rootdomain result in Google : trocmalin.com/ A description for this result is not available because of this site's robots.txt – learn more. ...
2
votes
1answer
235 views

What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]

Possible Duplicate: How to Remove URLs from Google Search Engine In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
2
votes
2answers
132 views

Robots.txt syntax -> Can this be done?

When creating a robots.txt, are the following the same? # Slurp User-agent: Slurp Disallow: / and # Slurp User-agent: Slurp Disallow: / It seems like for someone disallowing a large number of ...
2
votes
1answer
148 views

How can I let robots access my site?

I have a webpage on my university's server, which means I cannot access the root directory and change files there. Is there some other way to give access to robots to my page? Thanks.
2
votes
3answers
883 views

What is a good robots.txt for WP?

What is the "best" setup for robots.txt? I'm using the following permalink structure in Wordpress: /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
2answers
45 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
2
votes
2answers
273 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
2
votes
2answers
268 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
2answers
141 views

How to block pagination when the URL is just one number?

I'm trying to avoid duplicate content on a website by blocking pagination via robots.txt, but it turns out the URL doesn't have a parameter passed into the URL I can simply block, it's just a number. ...
1
vote
2answers
300 views

robots.txt ignored

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories. By default it had Disallow: ...
1
vote
2answers
99 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
1
vote
3answers
228 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
6answers
93 views

Uploading a non-finished website

I have a pretty basic question. I developed a neat little website which I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
1
vote
3answers
206 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
3answers
444 views

Preventing Spiders on One Subdomain?

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
1
vote
3answers
8k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
1
vote
4answers
446 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
3answers
119 views

robots.txt file Help needed

What if I put same robots.txt file in multiple directories and folders? User-agent: * Disallow: /otherdirname/dirname2/ Can it'll affect anyhow ?? And how much time it generally takes to refine ...
1
vote
2answers
132 views

Problem with my robots.txt

I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
1answer
2k views

Strange robots.txt - how and why did it get there?

I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
1
vote
1answer
968 views

Different robots.txt for two different domains point to same folder

I have the following two domains: domain.com test.domain.com Both point to same folder which is "public_html". What I want is a different robots.txt file for each domain. So when someone ...
1
vote
3answers
221 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
1
vote
2answers
280 views

Can search engine robots read file with permission 640?

I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt. As per the following google group ...
1
vote
2answers
485 views

Robots.txt not working because my site is still searchable

I put a robots.txt in my server This is the code that I putted in User-agent: * Disallow: / I just want my entire site won't be searchable in google. I also put in the NO INDEX, NO FOLLOW on the ...
1
vote
5answers
397 views

Should I add a “nofollow” attribute to download links, or disallow the URLs in robots.txt?

I have a download link very similar to Opera's one - it's just a script that sends the file. It doesn't have an extension and there's no obvious way to tell that it's actually a download link. So ...
1
vote
2answers
113 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt
1
vote
3answers
61 views

Google is not crawling and indexing my site after updating my robots.txt file

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my ...
1
vote
3answers
208 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
1
vote
2answers
114 views

Handling duplicate content caused by 'new' smart-url structure

I used to have an URL structure like this: mydomain.com/search.php?key=value which I've since changed to: mydomain.com/key/value. I've changed all the internal links in my site, but Google ...
1
vote
1answer
143 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
2answers
159 views

Robots.txt and specific file inclusion

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
1
vote
3answers
52 views

How to handle URL parameter for mobile site

I have a phpBB forum with a skin for mobile users. Some javascript detects your device/resolution and sends you to the mobile skin or not. The URL's remain the same, it's based on a cookie. People ...
1
vote
3answers
51 views

Making a dynamic robots.txt but system is appending forward slash

I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I ...
1
vote
1answer
25 views

robots.txt could be overridden by another robots.txt in sub folders?

robots.txt is supposed to go in the root directory, I know. But, could it be overridden by another robots.txt inside a sub directory?
1
vote
2answers
96 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
1answer
127 views

Google lists redirected URL instead of target URL

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www. To clean up the search results, we've added an exclusion in robots.txt for the old hostname and ...
1
vote
3answers
105 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
1
vote
1answer
42 views

If I block a page in robots.txt would I still be able to see the page in my browser? [closed]

What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
1
vote
1answer
162 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
1answer
150 views

Should a single page fetcher like TinEye obey robots.txt?

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
1answer
109 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
1
vote
1answer
72 views

What should I do when the maximum sitemap size limit is reached according to Google Webmaster Tools?

We have an existing sitemap with the URL: http://example.com/sitemap/sitemap.xml It was submitted to Google Webmaster Tools a long time ago and almost all the pages were indexed, but recently we ...
1
vote
1answer
39 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
1
vote
1answer
92 views

Sudden Increase in “Blocked from robots”

In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots". There's been no spike in Crawl Errors since this happened a week and ...
1
vote
1answer
50 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...