Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In Google Webmaster Tools, I get an error:

Google couldn't crawl your site because we were unable to access your site's robots.txt file

The associated help says:

If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling disallowed URLs

My site 302 redirects all http traffic to HTTPS - so access to http://blah/robots.txt is 302 redirected to https://blah/robots.txt, so literally it doesn't return 200 or 404 as requested above.

My question - does Googlebot object to the 302 redirect when it attempts to access the robots.txt file?

Note: a lot of the server configuration is out of my control, and is configured this way due to corporate IT. I'm just the poor guy who needs to get this working in spite of the constraints.

share|improve this question
add comment

2 Answers 2

These quotes I have taken from Google.

File location & range of validity
The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number. Generally accepted protocols for robots.txt (and crawling of websites) are "http" and "https". On http and https, the robots.txt file is fetched using a HTTP non-conditional GET request.

and also

3xx (redirection)
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404.

You could/should maybe read the source at Google.

share|improve this answer
add comment

Most likely yes, as you are using a 302 redirect which is temporary redirect that Google recommends against using. You should be using a permanent 301 redirect to redirect from HTTP to HTTPS.

share|improve this answer
2  
I think moobot is answering "yes" to if Googlebot would object to the 302 redirect, and not the question in the title. –  dan Nov 20 '13 at 1:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.