Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found".

Does this mean that, if I create a 301 redirect for http://www.example.com/robots.txt to http://www.example.com/myrobots.txt, Google would parse the content served from http://www.example.com/myrobots.txt as being valid robots.txt content? Or does the fact that the result URL is not named robots.txt make this an invalid result, and therefore will be ignored?

share|improve this question
    
Looks like your first example is the how it works –  John Conde Aug 1 '13 at 15:22

1 Answer 1

Robots follow the Robots exclusion standard, also known as the Robots Exclusion Protocol. According to the WC3, robot directives must be specified in a file with the URI containing "/robots.txt", all in a lower case string, located in the root directory of your site. For more on that see: WC3 - The robots.txt file

As stated in there: The Robot will simply look for a "/robots.txt" URI on your site. Therefore, some robots may not recognize a robots.txt with a different file name or location.

In Google Webmaster Tools, you can view the URL to the robots.txt file for your site under Crawl->Blocked URLs, as covered here: Google Webmaster Tools - Block or remove pages using a robots.txt file (under the "Test a robots.txt file" down arrow).

If your robots.txt is missing, it likely did not find or recognize it. You can reconfirm this by listing your URL to test against and click "Test".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.