Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My robots.txt is:

User-agent: *
Disallow: /

Sitemap:  http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz

It has been the same for well over a week now, but when I go to fetch Google, I am still getting the same error message: Googlebot is blocked from http://www.example.com/.

According to the Google Webmaster Tools, it may take up to just two days for an updated robots.txt file to be refreshed.

A few of my pages are being crawled and I have indexed them, but why are some still being denied by robots.txt? My site settings allow search engines to index, how long does the Googlebot take to see I have allowed all my site to be crawled?

share|improve this question

Your robots.txt own these lines:

User-agent: *
Disallow: /

It indicates to Googlebot and other search spiders to not index your website.

Just put these lines instead and wait:

User-agent: *
Allow: /

That way, all search spiders (Googlebot included) will index your website.

share|improve this answer

I can confirm that your robots.txt and HTML are not blocking Google from scanning your site, so whats the problem? In short... time.

Google may state that it takes two days for Google to take note of a robots.txt but in no way does this imply that all these pages blocked previously will be unblocked and instantly crawled.

Now that Google can crawl your site it may take a few visits to gather the data to index into their engine. This varies and can be one to two weeks, but sometimes sadly it does take longer. You just need to be patient nothing else you do in Google Webmaster Tools will speedup this process.

It's also important to note that even if Google visits your site say for example on the front page it doesn't necessary capture all the data they may want, sometimes it takes a few crawls. Again, just sit back and get some lemonade. ;)

share|improve this answer
Thank you for answering my question – user27707 May 4 '13 at 12:49

Your robots.txt tells all bots (User-agent: *) not (Disallow:) to crawl any URL starting with a slash (/), which would be all URLs, of course.

If you want to allow all bots crawling all your pages, remove the slash:

User-agent: *

Note that Allow: is not part of the original robots.txt specification, however, some/many crawlers recognize it. Disallow: and Allow: / would be equivalent in this case, so why not use the "standard" one.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.