Does Google respect a redirect header for robots.txt to a different file name?

Question

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found".

Does this mean that, if I create a 301 redirect for http://www.example.com/robots.txt to http://www.example.com/myrobots.txt, Google would parse the content served from http://www.example.com/myrobots.txt as being valid robots.txt content? Or does the fact that the result URL is not named robots.txt make this an invalid result, and therefore will be ignored?

Answer 1

Robots follow the Robots exclusion standard, also known as the Robots Exclusion Protocol. According to the WC3, robot directives must be specified in a file with the URI containing "/robots.txt", all in a lower case string, located in the root directory of your site. For more on that see: WC3 - The robots.txt file

As stated in there: The Robot will simply look for a "/robots.txt" URI on your site. Therefore, some robots may not recognize a robots.txt with a different file name or location.

In Google Webmaster Tools, you can view the URL to the robots.txt file for your site under Crawl->Blocked URLs, as covered here: Google Webmaster Tools - Block or remove pages using a robots.txt file (under the "Test a robots.txt file" down arrow).

If your robots.txt is missing, it likely did not find or recognize it. You can reconfirm this by listing your URL to test against and click "Test".