In Google Webmaster Tools, I get an error:
Google couldn't crawl your site because we were unable to access your site's robots.txt file
The associated help says:
If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling disallowed URLs
My site 302 redirects all http traffic to HTTPS - so access to http://blah/robots.txt
is 302 redirected to https://blah/robots.txt
, so literally it doesn't return 200 or 404 as requested above.
My question - does Googlebot object to the 302 redirect when it attempts to access the robots.txt file?
Note: a lot of the server configuration is out of my control, and is configured this way due to corporate IT. I'm just the poor guy who needs to get this working in spite of the constraints.