11

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt?

I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt.

Through some error, my robots.txt was replaced with:

User-agent: *
Disallow: /

And now all my content was removed from Google search results.

Obviously, I'm interested in correcting this as soon as possible. I already replaced the robots.txt, but I can't find a way to make Google update the cached version.

enter image description here

| improve this question | |
  • 1
    Just disallowing all your pages in robots.txt should generally not be enough to completely remove them from Google's results, as long as other sites still link to them. – Ilmari Karonen Apr 19 '12 at 16:37
  • Hmm its a tricky one. ZenCart URLs seem to confuse the robots.txt web crawler bot and before you know it, you have blocked URLs that you don't want to be blocked. My experience is that you are better off without robots.txt, but just keeping a clean web site. I lost many web rank places due to this robots.txt error blocking of valid URLs. Because ZenCart uses dynamic URLs it seems to confuse the robots.txt web crawler resulting in blocking of URLs that you don't expect to be blocked. Not sure if it relates to the disabling of a category in ZenCart and then moving products out of that category a – user26869 Apr 6 '13 at 14:44
10

You can't make them re-download your robots.txt when you want them to. Google will re-crawl it and use the new data whenever they feel it is appropriate for your site. They tend to crawl it regularly so I wouldn't expect it to take long for your updated file to be found and your pages re-crawled and re-indexed. Keep in mind that it may take some time after the new robots.txt file is found before your pages are re-crawled and even more time for them to reappear in Google's search results.

| improve this answer | |
1

I faced the same problem when I started my new website www.satyabrata.com on June 16.

I had a Disallow: / in my robots.txt, exactly like Oliver. There was also a warning message in Google Webmaster Tools about blocked URLs.

The problem was solved yesterday, June 18. I did the following. I am not sure which step worked.

  1. Health -> Fetch as Google: robots.txt and the home page. Then, submit to index.
  2. Settings -> Preffered domain: Display URL as www.satyabrata.com
  3. Optimization -> Sitemaps: Added XML sitemap.

The warning message about blocked URLs is gone now and a fresh robots.txt is shown downloaded in Google Webmaster Tools.

Presently, I have only two pages indexed in Google, the home page and robots.txt. I have 10 pages on the website. I hope the rest will get indexed soon.

| improve this answer | |
0

I had a problem where the images were moved to a separate CNAME server and a disallow was put on the images folder. The way I got it to clear was to have robots.txt retrieved in the Webmaster Tools read webpage as Google tool. Once it told me it had retrieved and read robots.txt, I submitted it. This broke a three month embargo on scanning images where Google reported it was reading the robots.txt file but wasn't changing its spidering to match the rules that were changed to allow the image folder. Within a week, images were being indexed again.

Might be worth a try. Google is known to occasionally get stuck and fail to reread the file.

| improve this answer | |
  • They re-read the file about 6 hours after I posted. Everything is back to normal by now. – Der Hochstapler May 6 '12 at 21:24
  • Whew! Back on track then! – Fiasco Labs May 6 '12 at 21:39
  • I tried asking webmaster tools to fetch robots.txt, it complained it was denied by robots.txt :). So apparently that trick will not work if you have robots.txt doing a full block. – studgeek Aug 2 '12 at 15:40
  • Same here... Request for robots.txt denied by robots.txt! Hah! – Kasapo Aug 27 '12 at 19:27
  • Whelp, if you put deny on the root then I guess you're kind of SOL. In my case, it was a subfolder that was being refused, so forcing a reread of robots.txt through the mechanisms provided actually worked. – Fiasco Labs Sep 9 '12 at 2:40
-1

I hope this link will help you crawl your websites: https://support.google.com/adsense/answer/10532?hl=en.

Remove / from your robots.txt file.

| improve this answer | |
  • That's not what this question is about – Der Hochstapler Oct 5 '13 at 16:32
-1

In my case the problem was that I was using a free DNS service called afraid.org.

(my free domain ended in .us.to)

Once I transitioned to a TLD, it started working.

| improve this answer | |
  • I don't see what DNS or having free site has to do with robots.txt or telling Google to refetch it. – Stephen Ostermiller♦ Apr 5 '18 at 21:36
  • @StephenOstermiller: I don't see either, but the fact is, this helped in my case. – Stefan Monov Apr 6 '18 at 6:47
  • It helped Google check your robots.txt? – Stephen Ostermiller♦ Apr 6 '18 at 6:53
  • @StephenOstermiller: Yes. – Stefan Monov Apr 6 '18 at 17:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.