3

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer.

example: https://www.mydomain.example/robots.txt redirects to https://differentdomain.example/example/robots.txt

  • Why would you want to do that? – John Conde Mar 11 at 11:51
  • 1
    I assume you want to know if search engines will still honor robots.txt for your domain, even if it hosted elsewhere? – Stephen Ostermiller May 7 at 11:13
3

Google allows the robots.txt 301 redirection you're talking about:

Google follows at least five redirect hops as defined by RFC 1945 for HTTP/1.0 and then stops and treats it as a 404.

https://developers.google.com/search/reference/robots_txt

I couldn't find any information about Bing's crawler.

It's my suspicion that smaller, poorly-written, non-search-engine bots might not be smart enough to follow the redirect, and might take the 301 status code as a blank check to hit your website.

If this is about keeping your robots.txt automatically up to date with the robots.txt on the other domain, I'd consider a reverse-proxy setup if possible.

On the other hand if this is about lessening the strain on your server by offloading your robots.txt to a CDN, well, your server is still going to have to serve all of those 301 status codes, which probably won't be much of a lighter load than just serving up the robots.txt file in the first place. And if serving your robots.txt file even tens of thousands of times per day is putting any real strain on your server, that's a surefire sign that there's an issue with your server.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.