How do I tell Google to ignore an entire fourth level wildcard subdomain with several sites?

Question

I have multiple servers which host several sites, some of them being QA environments.

Unfortunately, Google has found its way to some of them, and it is displaying results which are in fact competing with the corresponding live site URLs, for multiple sites.

Is there a way I can tell google, via analytics or something similar, to simply delist all of the existing information for subdomains *.a.example.tld or *.b.example.tld?

Answer 1

There is no setting in Google Analytics or Google search console that will tell Googlebot not to index an entire site, subdomain, or wildcard subdomain. You will need to change the development sites themselves, or configure their servers.

If the sites give errors to Googlebot, Google will naturally drop them from the search index as they are crawled. There are several ways to do this:

• Password protect all the sites using digest authentication. Implementing it is only a few lines of configuration for Apache. Other servers such as IIS and nginx support it as well.
• Configure your servers to return a "403 Forbidden" response to Googlebot
• Place those servers on an internal network behind a firewall that is not accessable to the outside world and Google.
• Retire your existing subdomains and create new ones. Instead of having a.example.tld, you could use q.example.tld. You would again have to change the subdomain when Googlebot discovered them again unless you also protected them further.

• Use robots.txt on each site to disallow Googlebot from crawling each site. You'd need a robots.txt on each sub-subdomain such as site1.a.example.com/robots.txt and also site2.a.example.com/robots.txt with contents like:

  User-Agent: *
  Disallow: /
  

  Note that Google may still choose to index some pages that are blocked by robots.txt. However, Google wouldn't see the words on the page, and just index them based on any links pointing to them. It is much less likely the pages would get indexed, and they would be unlikely to rank well. This would be a good solution to further protect your QA sites if you decided to change your subdomains.

• Change every page on every QA site so that is contains a <meta name="robots" content="noindex"> tag. Then Google will never index those pages. You would need to do this with every page on every QA site. This solution is also not compatible with other solutions such as robots.txt because for these tags to be effective Googlebot has to be able to crawl to see the tags.