My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to fetch Google, I am still getting the same error message: Googlebot is blocked from http://www.example.com/
.
According to the Google Webmaster Tools, it may take up to just two days for an updated robots.txt file to be refreshed.
A few of my pages are being crawled and I have indexed them, but why are some still being denied by robots.txt? My site settings allow search engines to index, how long does the Googlebot take to see I have allowed all my site to be crawled?