Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This question already has an answer here:

My old robots.txt is

User-agent: *
Disallow: /

Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.

share|improve this question

migrated from serverfault.com May 20 at 0:34

This question came from our site for professional system and network administrators.

marked as duplicate by John Conde May 20 at 2:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
Why don't you change it? –  John Mee May 20 at 1:43
add comment

2 Answers 2

robots.txt is cached and will automatically be refreshed by google within a day, so passage of time will sort this.

Caching A robots.txt request is generally cached for up to one day, but may be cached longer in situations where refreshing the cached version is not possible (for example, due to timeouts or 5xx errors). The cached response may be shared by different crawlers. Google may increase or decrease the cache lifetime based on max-age Cache-Control HTTP headers.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=it

share|improve this answer
add comment

Googlebot will re-fetch robots.txt more often than most other files on your web server. You generally have to wait less than 24 hours. From Google's Documentation:

A robots.txt request is generally cached for up to one day, but may be cached longer in situations where refreshing the cached version is not possible (for example, due to timeouts or 5xx errors). The cached response may be shared by different crawlers. Google may increase or decrease the cache lifetime based on max-age Cache-Control HTTP headers.

I'm not sure what you mean when you say that your old robots.txt file is blocking you from fetching the new one manually. Robots don't obey robots.txt for the purpose of fetching robots.txt. Robots periodically fetch the robots.txt file, even if you were to put a Disallow: /robots.txt line in the file. There is no way to use robots.txt to prevent bots from checking robots.txt.

One way to force Googlebot to download a page immediately is to use the "Fetch as Google" feature in Google Webmaster Tools (its in the "Crawl" menu). You could use this feature to force Googlebot to fetch your new robots.txt file right away.

Webmaster Tools also has a "Blocked URLs" feature (also in the "Crawl" menu) that shows you what your current robots.txt file is and lets you test to see which URLs are blocked by it. You can change the robots.txt file in that tool to make sure that changes block and unblock the URLs that you expect.

share|improve this answer
    
I tried that but google gives me this error: Fetch as Google This is how Googlebot fetched the page. URL: syrup.mx/robots.txt Date: Monday, May 19, 2014 at 5:34:17 PM PDT Googlebot Type: Web Download Time (in milliseconds): The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed. You can find more information in the Help Center article about robots.txt. –  user220297 May 20 at 2:55
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.