0

We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs.

This causes Google bot to keep crawling old paths again and again and spending a lot of crawl quota on old URLs.

If we block old URLs via Robot.txt like this:

Disallow: /olddirectory/

This would mean Google won't crawl them but would they still pass link juice?

Thanks

1

You would lose the effect of the link/301 redirect if you block it in robots.txt. Google needs to know that the link has something at the end of it- ie. your web page- or it cannot know that the link is valid. Therefore the search engine must be able to follow the link all the way through to your page.

I have mentioned this on several similar answers before. What is often forgotten in these cases where a 301 redirect is used to preserve a link is that using a 301 redirect should be temporary. Certainly you can keep a 301 redirect for as long as you like, however, what typically happens is there comes a day where the idea of the 301 seems less desirable. If it is just a few links then they are no real bother, but when there are a bunch of them, then it is likely that one day the link/301 redirect glimmer will fade. It always happens.

I try and preempt this often where a site moves from an old domain name to a new one. I advise that the new domain must be able to stand on it's own as if the old domain did not exist. The same advice exists here. Before you tire too much of these links, try and replace the old link profile with a new one so that the old links do not matter any more. This is good advice for anyone who uses 301 redirects to preserve a link. There comes a time where you will want to remove the 301 redirect and you want to have already replaced it's value with new links. It becomes so much easier to say good bye to old rubbish that way.

  • Thanks. This is very useful. So will google ever register them in their DB or will keep following the old URLs as long as people are linking to them (old links) – KBS Sep 5 '14 at 3:58
  • @KBS As long as the link exists, Google will follow it. If the 301 redirect exists, Google will follow that. It is a simple system. If there is no page, then Google will get a 404 error. If there is a page, then Google will get a 200 success. As long as the link exists, Google will try and fetch the page get a 301 and then try and fetch the page at the new location. It cannot know if this time it will get a 404 error or a 200 success so it tries each and every time. – closetnoc Sep 5 '14 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.