3

I have multiple servers which host several sites, some of them being QA environments.

Unfortunately, Google has found its way to some of them, and it is displaying results which are in fact competing with the corresponding live site URLs, for multiple sites.

Is there a way I can tell google, via analytics or something similar, to simply delist all of the existing information for subdomains *.a.example.tld or *.b.example.tld?

|improve this question
  • Possible duplicate of How can I use robots.txt to disallow subdomain only? – Anuvesh Aug 28 '19 at 5:18
  • 1
    @BlunT Do you want to tell Google not to index sites under a subdomain, remove these sites from Google's SERPS, or both? – dan♦ Aug 28 '19 at 8:38
  • @Anuvesh I don't think that is a duplicate because this one is asking about multiple wildcard sub-subdomains, not just about the content on a single subdomain. – Stephen Ostermiller♦ Aug 28 '19 at 9:42
  • @Anuvesh That question is also about preventing crawling, whereas this question wants to remove search results (that have already been crawled and indexed). – MrWhite Aug 28 '19 at 14:45
  • @dan I want everything :) I want google to remove the indexing that it has already made, and prevent it from doing the same in the future! – BlunT Aug 28 '19 at 18:05
2

There is no setting in Google Analytics or Google search console that will tell Googlebot not to index an entire site, subdomain, or wildcard subdomain. You will need to change the development sites themselves, or configure their servers.

If the sites give errors to Googlebot, Google will naturally drop them from the search index as they are crawled. There are several ways to do this:

  • Password protect all the sites using digest authentication. Implementing it is only a few lines of configuration for Apache. Other servers such as IIS and nginx support it as well.
  • Configure your servers to return a "403 Forbidden" response to Googlebot
  • Place those servers on an internal network behind a firewall that is not accessable to the outside world and Google.
  • Retire your existing subdomains and create new ones. Instead of having a.example.tld, you could use q.example.tld. You would again have to change the subdomain when Googlebot discovered them again unless you also protected them further.

  • Use robots.txt on each site to disallow Googlebot from crawling each site. You'd need a robots.txt on each sub-subdomain such as site1.a.example.com/robots.txt and also site2.a.example.com/robots.txt with contents like:

    User-Agent: *
Disallow: /

    Note that Google may still choose to index some pages that are blocked by robots.txt. However, Google wouldn't see the words on the page, and just index them based on any links pointing to them. It is much less likely the pages would get indexed, and they would be unlikely to rank well. This would be a good solution to further protect your QA sites if you decided to change your subdomains.

  • Change every page on every QA site so that is contains a <meta name="robots" content="noindex"> tag. Then Google will never index those pages. You would need to do this with every page on every QA site. This solution is also not compatible with other solutions such as robots.txt because for these tags to be effective Googlebot has to be able to crawl to see the tags.
|improve this answer
  • 1
    "Use robots.txt on each site to disallow Googlebot" - Although this won't necessarily remove the existing results from the SERPs. – MrWhite Aug 28 '19 at 11:32
  • 1
    Another option (similar to the second bullet point) is to restrict access by IP. You can either whitelist your own IPs, or block the IPs of Googlebot (it appears they don’t publish an official list but there are websites listing them). – DisgruntledGoat Aug 28 '19 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.