2

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains and pages on the old site which have no similar comparable page on the new site. I want Google to stop wasting crawl budget on these pages. So a robots.txt would be in order, however, I also want these pages to be dropped out of the index and robots.txt may not do that. Would noindex, nofollow meta robots achieve both crawl budget waste and dropping out of the index?

  • If the page has been removed, just return a "404 Not Found" or (preferably) "410 Gone" status. Google doesn't index those. – Stephen Ostermiller Aug 3 '17 at 11:31
2

There are at least two things you can do with pages that are outdated: redirect them (usually via a 301 permanent redirect) or let them go to a 404, as Stephen said.

In most cases, I would redirect, for a couple of reasons:

  1. Your old pages and subdirectories/categories may carry link juice. You can use Moz's Open Site Explorer to see their inbound link profiles and authority (as perceived by Moz). You want to pass that link profile and authority to another page on your site.

  2. If those pages have been indexed, they may hang around in the index for a while before search engines remove them. You want users who may come across these pages to go to another page on your site.

  3. After a while, when search engines realize that your pages are redirecting, they will take the old pages out of the index.

When redirecting, your old and new pages don't have to be a 1:1 match. If the section is sort of related, you can still redirect - that is, if you think the new section will benefit the user who was looking for the old one.

If nothing fits, you can redirect to the home page, or to a contact page. You won't be penalized; at worst, the redirect will pass no link juice. (If you're really worried about it, you can use a 302 temporary redirect, but 301's do a better job of removing things from the index.)

It's also fine to let it go to a 404, if there is absolutely nothing that's semantically similar to the page you're taking down, or if that page has a poor backlink profile or got very little traffic. In that scenario, it'll get dropped from the index eventually too. I'd make sure that your 404 page is helpful to the user, displaying suggested links and the nav, for instance.

You don't want to use noindex, nofollow in this case, because there's no point in keeping this page on the server. The robots.txt solution is not a solution, because disallowing directories you've deleted anyway will create redundant rules, and as you said won't necessarily take it out of the index.

Whichever way you choose to go, run a site:example.com query now and again, and go deep into the SERPs. If you see pages you don't want still in the index, you can take a different course, and also request manual removal via Google Search Console and Bing Webmaster Tools.

  • Fantastic answer thanks. Helped confirm a lot to me. – mat boy Aug 4 '17 at 17:18
  • You're welcome - glad it was helpful! – Henry Visotski Aug 7 '17 at 4:34
2

what should I do with subdomains and pages on the old site which have no similar comparable page on the new site.

If the pages will stay live, Add a robots.txt file or add noindex, nofollow meta tag, this will stop this pages from being crawled.

Deleting the pages from the server will drop the pages from the index automatically.

Also, a well drafted 404 page with relevant links will help the users who still have this links saved.

  • Thanks for your help. I don't think a noindex, nofollow meta tag stops the pages from being crawled. Do you have a source for this? – mat boy Aug 4 '17 at 17:15
  • But using measures like meta tags and robot.txt file will stop search engines from listing your site on search results. – gabehou Aug 4 '17 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.