5

I'm moving a site to a new domain, to a local tld. I already redirected everything successfully to the new domain, except the robots.txt. My question would be that is it needed to redirect the olddomain.com/robots.txt to newdomain.com/robots.txt with 301 redirect? Is there any common practice for this situation?

  • You should redirect everything... using a global mod_rewrite... but it is not required to redirect robots.txt as Google and Bing will always check the new domain for it, heck you can even submit it in Google Search Console. – Simon Hayter Jun 9 '17 at 20:10
  • @SimonHayter I didn't use Google Search Console, it's a quite new site. I realized the need of another domain in time. I'm thinking about to redirect the robtos.txt because I see in the logs that Google visits it regularly so it might index the new site faster if I redirect it. Or you think I don't have to really care about it? – rihe Jun 9 '17 at 20:17
  • It will not speed up the indexation process. – Simon Hayter Jun 9 '17 at 21:35
  • Please remember that 301 redirects are applied per page. Even if you redirect all pages to the new domain, robots still read and process your robots.txt in the old domain. – Osvaldo Jun 10 '17 at 10:45
3

This is really interesting question, It took me few minutes to search official statement from gary tweet..

When Google see 404/30X status, then Google will ignore your robots.txt completely. It is like not having a robots.txt at all.

But if Google see 50X status or server side errors, then Google will stop crawling your site, because may be you're busy to configure your server.

So do not redirect your robots.txt or if you do then make sure you've block same thing in new site robots.txt

  • 1
    This contradicts the information provided by Google: Redirects will generally be followed until a valid result can be found (or a loop is recognized). developers.google.com/search/reference/robots_txt – dimiguel Apr 8 at 23:00
  • Thanks @dimiguel. I think someone should make a test on robots.txt redirection by using Google robots.txt checker, to know weather they follow 301 redirection or not. They even say they use same user-agent as Googlebot use, but still I'm not sure weather the test will be 100% correct or not. – Goyllo Apr 9 at 4:56
0

The robots.txt on the old domain is now useless cause it helps google crawling the current domain website.

So if you correctly redirect olddomain.com/* to newdomain.com/* The robots.txt will be forwared to, and you should put approriate rules in your new robots.txt (change the content if you also change the cms or url format ...)

hope it helps ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.