The old robots.txt alerts me with warning about locked URL. With the new one, the URL passes the tester in Google Webmaster. But when I resend the sitemap I get the same error!

  • Google reads the robots.txt file every 24 plus hours since the last time it read your robots.txt. Using Google Search Console will use the last one it has. You will just have to wait. Do not try and hurry Google up. It will not be long before your updated robots.txt file can be seen and used. – closetnoc Sep 25 '15 at 16:11

It's not really an exact answer to your question but this may help you more: If I remember correctly I have successfully used a manual "Fetch as Google" at https://www.google.com/webmasters/tools/googlebot-fetch to force a refresh of robots.txt.

|improve this answer|||||
  • I have seen reports where that has not worked effectively. So now I wonder what the truth is. – closetnoc Sep 26 '15 at 0:19
  • 2
    I've also used the fetch facility in webmaster tools and it updated robots.txt in their cache almost instantly – MrCarrot Sep 26 '15 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.