This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web:
Our client migrated to HTTPS a few months ago, using their HTTP sites along their HTTPS sites. We told them to 301 redirect their HTTP sites to their corresponding HTTPS sites. So far everything was fine...
UNTIL we got an error essage in Google Webmaster Tools for
Googlebot encountered 5429 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 12.9%.
We asked their IT service provider to look into the issue, assuming that they made some mistake by setting the 301 redirect in the .htaccess file. However, they referred back to us stating that redirection for robots.txt might be discouraged by Google generally, see here, and that this might well be the issue. They recommend keeping the HTTP robots.txt with a 200.
I actually have never come across this issue so far. Have you any idea what might be causing the issue?
I figure if we do no longer redirect the HTTP robots.txt file, Googlebot might try to crawl the HTTP versions of the website. Which shouldn't be an issue really if all HTTP versions are 301 redirected to the HTTPS versions properly. It just doesn't feel right ;) I'm interested in fixing the issue (by finding out the cause) more than finding a quick fix.