8

A site I am working on moved a subdomain to another subdomain via a 301 redirect. However when checking robots.txt of the old subdomain, it has a robots.txt which disallows search engine web crawlers to crawl it.

Is this the right move? I believe no because the crawlers won't be able to crawl the old site to see the 301 status codes.

Improve this question
1
  • 3
    Was the disallow already in place before the move? Some websites do not wish to be in search engines. It is unclear which way the situation played out.
    – user64742
    Commented Jul 5, 2017 at 17:27

3 Answers 3

Reset to default
11

Your suspicions are right, with the exact reasoning that you mentioned. If you disallow robots from accessing the old site, they won't be able to see the 301 redirect.

Improve this answer
2
  • Would it also mean that link juice that was pointing to the old domain won't be transferred to the new domain because the robots.txt is not letting Gbot crawl?
    – mat boy
    Commented Jul 5, 2017 at 11:19
  • 1
    Yes, that is exactly what would happen if the robots.txt is blocking access to the bots.
    – Theo van der Zee
    Commented Jul 5, 2017 at 11:45
6

I believe no because the crawlers won't be able to crawl the old site to see the 301 status codes.

Yes, exactly - this is not the "right move". If you implement a 301 redirect, presumably to preserve SEO, then blocking the URLs that are being redirected is going prevent the search engines from seeing the redirect and indexing the new content.

Improve this answer
4
  • Would it also mean that link juice that was pointing to the old domain won't be transferred to the new domain because the robots.txt is not letting Gbot crawl?
    – mat boy
    Commented Jul 5, 2017 at 11:19
  • @mat boy, "link juice" is seldom updated if ever. It's not likely to ever be updated to the new domain with or without the robots.txt file, but keeping the 301 in place indefinitely, assures that the links will end up in the right place.
    – Octopus
    Commented Jul 5, 2017 at 21:40
  • 1
    @Octopus You seem to be describing "PageRank", not "link juice"? "link juice" is just a common/slang term describing the SEO benefit of linking from one page to another.
    – DocRoot
    Commented Jul 5, 2017 at 23:11
  • 1
    @matboy Yes, that's right, no "link juice" will be transferred to the new URL. Since the bot is blocked from seeing the redirect, the bot (search engine) does not know the page has moved to a different location. The old URLs will drop in the SERPs and eventually dissappear, whilst the new URLs will need to build their SEO from scratch.
    – DocRoot
    Commented Jul 5, 2017 at 23:17
0

I would keep the robots.txt as is, but make sure that the old and new pages have the appropriate version of Canonical tags. This will help the search engine make sense of what has gone on.

Improve this answer
1
  • 2
    How could the crawlers read the canonical tags on the webpage if you are telling them not to crawl the webpage?
    – mat boy
    Commented Jul 17, 2017 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.