Can Googlebot discover a new subdomain through a DNS CNAME chain?

Question

I have a website - www.example.com - which is listed on Google, domain verified in Search Console, sitemaps submitted, canonical urls all in line. This is my only domain that is "out there" in the wild.

www.example.com is configured in DNS using a CNAME which points to another domain, which finally points to an IP. As follows:

www.example.com -> CNAME = example.cloudhost.com -> A = 192.0.2.255

For dev/test/debugging reasons I have a binding on my web server which will respond to example.cloudhost.com, which means if you navigate to any url on that domain, you'll get the exact same responses as www.example.com, including all canonical tags pointing back to www.example.com

example.cloudhost.com has not been published ANYWHERE on the Internet, let alone given directly to Google. It is not exposed via any html tags, sitemaps, Search Console submissions, social media posts, 3rd party websites. Nothing.

--- edit ---

I now realise it could be leaked via http referrer or analytics trackers running on example.cloudhost.com when used for testing.

---

However, somehow Google has indexed example.cloudhost.com as a duplicate domain to www.example.com.

How did Googlebot get hold of example.cloudhost.com to start indexing it and is there any documentation of its methods? Does Googlebot look at DNS CNAME chains and speculatively try to crawl content on those intermediate domains?

Answer 1

I highly doubt that the CNAME is the only reference to example.cloudhost.com. There are many ways that URLs leak. See Can a URL that is not linked to from anywhere be discovered? for a more complete list but it includes:

• Emails with the link sent through Gmail
• Third party JS used on your site, especially analytics or ads which send data to Google
• External links from your site that send a referrer that is subsequently published by the other site.

It is possible that Googlebot looks at the hostname in the CNAME chain to discover the site. I've not seen evidence that Googlebot does this, but it wouldn't surprise me if it did. Googlebot does a lot of speculative crawling:

• Google watches new domain name registrations and tries to crawl all newly registered sites.
• Googlebot picks strings out of JS files that look like they might be URLs and crawls those. Pretty much any string literal with no spaces and a slash will get hit by Googlebot.
• Googlebot submits simple forms with some random options just to see if there is content behind the form.

The bottom line is that you can't rely on secrecy to keep your URLs out of Google. Googlebot is very good at finding private URLs. See Can secret URLs be used to protect files?

There are several ways to fix this:

1. Include canonical tags in each of your pages pointing to the full URL of the page on www.example.com. Then even if Googlebot crawls the cloud host URL, it will know to index the URL on your domain. Note that canonical tags won't stop Googlebot from crawling the entire alternate domain. Googlebot will even come back and crawl the alternate domain many times. The canonical tags will almost always cause Google to index each page on your preferred domain instead.
2. Reconfigure your server to redirect the cloud host URL to your domain. You said that you had it configured this way for testing and development, but presumably now that the domain is live, you don't need that. If you do still need it, you could do the redirect except for your own IP address.
3. Configure your server to serve different robots.txt files for the different host names. Disallow all crawling for the cloud host domain name.
4. Configure your server to password protect the site on the cloud host domain name. It is usually fairly easy to set up digest authentication that can do that.

Let me put it this way: I've always had Googlebot come crawl my development and staging URLs if I leave them open to the public. It doesn't matter how secret I think I've kept them. Even if I'm the only one that ever uses them, Googlebot seems to find them. I always protect them now, either with a password, or by doing my development on a private network.