Googlebot is blocked from my domain

Question

My robots.txt is:

User-agent: *
Disallow: /

Sitemap:  http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz

It has been the same for well over a week now, but when I go to fetch Google, I am still getting the same error message: Googlebot is blocked from http://www.example.com/.

According to the Google Webmaster Tools, it may take up to just two days for an updated robots.txt file to be refreshed.

A few of my pages are being crawled and I have indexed them, but why are some still being denied by robots.txt? My site settings allow search engines to index, how long does the Googlebot take to see I have allowed all my site to be crawled?

Answer 1

Your robots.txt own these lines:

User-agent: *
Disallow: /

It indicates to Googlebot and other search spiders to not index your website.

Just put these lines instead and wait:

User-agent: *
Allow: /

That way, all search spiders (Googlebot included) will index your website.

Answer 2

I can confirm that your robots.txt and HTML are not blocking Google from scanning your site, so whats the problem? In short... time.

Google may state that it takes two days for Google to take note of a robots.txt but in no way does this imply that all these pages blocked previously will be unblocked and instantly crawled.

Now that Google can crawl your site it may take a few visits to gather the data to index into their engine. This varies and can be one to two weeks, but sometimes sadly it does take longer. You just need to be patient nothing else you do in Google Webmaster Tools will speedup this process.

It's also important to note that even if Google visits your site say for example on the front page it doesn't necessary capture all the data they may want, sometimes it takes a few crawls. Again, just sit back and get some lemonade. ;)

Answer 3

Your robots.txt tells all bots (User-agent: *) not (Disallow:) to crawl any URL starting with a slash (/), which would be all URLs, of course.

If you want to allow all bots crawling all your pages, remove the slash:

User-agent: *
Disallow: 

Note that Allow: is not part of the original robots.txt specification, however, some/many crawlers recognize it. Disallow: and Allow: / would be equivalent in this case, so why not use the "standard" one.