4

How do I remove legacy URLs that Google has decided to cache?

When we went live with our new site, we moved the old site to legacy.example.com for our staff to be able to access the old site. I'm guessing robots.txt wasn't updated and as such, Google decided to cache it.

Our DNS has a wildcard redirect, so I think Google may still think that this site exists because navigating to legacy.example.com will redirect the user back to our live page.

I would normally use robots.txt but legacy.example.com doesn't actually exist anymore.

What options do I have? I have validated the property legacy.example.com in the Google Search Console. Do I need to change my robots.txt on my live site to somehow remove legacy.example.com?

4
  • Do I need to change my robots.txt on my live site to somehow remove legacy.xxx.com? Oh! God no! What web server are you using?
    – closetnoc
    Commented Mar 22, 2018 at 22:47
  • @closetnoc we are using Nginx
    – Lock
    Commented Mar 22, 2018 at 22:53
  • Cool. I do not know how to configure that web server. Essentially, you will need the web server to 404 or 410 or 301 any requests for the sub-domain or better yet not respond at all. This is one reason why I am a firm believer in not using a catch-all site to capture valid requests in Apache. In my case, I would create the sub-domain in Apache so that any request could could be handled separately. If you can do that, that may be your only or best option. Cheers!!
    – closetnoc
    Commented Mar 22, 2018 at 23:26
  • You say "Our DNS has a wildcard redirect", but DNS cannot redirect. I think you mean that your DNS has a wildcard record (possibly with a CNAME) that points to your server. Redirects have to be configured on the webserver. Commented Mar 24, 2018 at 12:37

3 Answers 3

1

If you move old content to new URLs where you don't want it to be found in search engines you have a few options.

  • Use robots.txt to block the entire subdomain from crawling. In legacy.example.com/robots.txt put:

    User-Agent: *
    Disallow: /
    
  • Require password protection to access the content on that domain. You could implement basic authentication in .htaccess. Here are some instructions for doing so: http://www.htaccesstools.com/articles/password-protection/
  • Host the subdomain on a server on your company's intranet where it is not accessible to the outside world.

Other answers suggest using 301 redirects, but they don't sound appropriate to me. Search engines don't need them because the content isn't in its original location. If you implemented them, you would have to do it in a way such that employees can see the content and don't get redirected.

I also wouldn't recommend canonical tags. They could work OK if there is a one to one correspondence of pages on the legacy site and the current site. However, Google reports in Search Console that it is ignoring canonical tags and choosing your non-canonical URLs to index. For my sites that happens pretty frequently, so canonical tags are not nearly as powerful for solving this type of problem as they once were.

2
  • He should be blocking the subdomain with robots.txt, not the entire domain, correct? Commented Mar 25, 2018 at 12:47
  • @joshssalganik Yes, I edited it. Commented Mar 25, 2018 at 13:14
0

In order to remove legacy.example.com/page from the search results, and have those results appear as example.com/page, you'll want to use 301 redirects or rel=canonical tags.

If you 301 redirect legacy.example.com/page to example.com/page, then example.com/page will show up in the search results once Google crawls the legacy subdomain pages.

You can also add a rel=canonical tag to all of your legacy subdomain pages, or all of the pages on your site pointing to the URL without the legacy subdomain at example.com/page. This will instruct Google to rank your pages without the subdomain.

As of now, you'll have to do one or the other, because Google will see these pages as duplicate content and won't be sure of which ones to rank. The best bet is to use the 301 tool unless you need the legacy subdomain pages for some other reason.

4
  • It sounds like they keep the legacy content around on a different subdomain than it used to be on so employees to still access it. If they put in redirects, employees won't have access anymore. Since it isn't the original URLs for the content, I don't see why they shouldn't just block that whole subdomain in robots.txt. Commented Mar 24, 2018 at 12:39
  • ^^Completely agree with Stephen. Take the path of least resistance. Commented Mar 25, 2018 at 13:38
  • @Michael d - we are already using 301 redirects but Google is still showing it
    – Lock
    Commented Apr 10, 2018 at 3:54
  • If you're using 301 redirects then it's probably just that Google hasn't recrawled those pages yet. As long as the old pages are directing to the new URL then the issue should be resolved eventually. For larger sites, it may take a while for Google to recrawl all of your indexed pages. For sites with millions of indexed pages, Google may not ever recrawl every single page especially if there are crawl delays set in Search Console.
    – Michael d
    Commented Apr 10, 2018 at 4:26
0

If you have not already, verify your site with Google Webmaster Central. Be sure to very each version, which would include the following: - www. - non-www. - https://non-www (if applicable) - https://www. (if applicable)

Now chose which site is your preferred version 1. Click on the preferred version of your site, 2. Click on site settings (icon looks like a gearbox) 3. Then select your preferred domain.

  1. remain inside the same GWC account
  2. Click on Google Index, it will expand
  3. Next, Click on Remove URL's from Index.

This should help prevent these URL's from showing up in search engines, if implemented with some of the previous suggestions such as blocking the subdomain using robots.txt.

You could always use a tool like slack, which is free, and you can keep company resources available for whomever may need them, hard to advise without knowing more.

@StephenOstermiller Below are 2 images, which prove that it is possible to verify the domain & subdomain separately within Google Search Console:

Main Site: tjmfuneral site

Subdomain:

tjmfuneral blog (under construction)

3
  • Google search console will let you choose between www and no-www, but it doesn't have controls for other subdomain. It won't have options for legacy vs no-legacy. Commented Mar 25, 2018 at 13:18
  • If he verifies the subdomain, he would have the exact same controls as any other site. A subdomain is treated entirely like a seperate site, despite there being a prefix before the root url. Commented Mar 25, 2018 at 13:30
  • The choice would be between www.legacy.example.com and legacy.example.com. I don't see how that would help. Commented Mar 25, 2018 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.