1

I have a website - www.example.com - which is listed on Google, domain verified in Search Console, sitemaps submitted, canonical urls all in line. This is my only domain that is "out there" in the wild.

www.example.com is configured in DNS using a CNAME which points to another domain, which finally points to an IP. As follows:

www.example.com -> CNAME = example.cloudhost.com -> A = 192.0.2.255

For dev/test/debugging reasons I have a binding on my web server which will respond to example.cloudhost.com, which means if you navigate to any url on that domain, you'll get the exact same responses as www.example.com, including all canonical tags pointing back to www.example.com

example.cloudhost.com has not been published ANYWHERE on the Internet, let alone given directly to Google. It is not exposed via any html tags, sitemaps, Search Console submissions, social media posts, 3rd party websites. Nothing.

--- edit ---

I now realise it could be leaked via http referrer or analytics trackers running on example.cloudhost.com when used for testing.

---

However, somehow Google has indexed example.cloudhost.com as a duplicate domain to www.example.com.

How did Googlebot get hold of example.cloudhost.com to start indexing it and is there any documentation of its methods? Does Googlebot look at DNS CNAME chains and speculatively try to crawl content on those intermediate domains?

2

I highly doubt that the CNAME is the only reference to example.cloudhost.com. There are many ways that URLs leak. See Can a URL that is not linked to from anywhere be discovered? for a more complete list but it includes:

  • Emails with the link sent through Gmail
  • Third party JS used on your site, especially analytics or ads which send data to Google
  • External links from your site that send a referrer that is subsequently published by the other site.

It is possible that Googlebot looks at the hostname in the CNAME chain to discover the site. I've not seen evidence that Googlebot does this, but it wouldn't surprise me if it did. Googlebot does a lot of speculative crawling:

  • Google watches new domain name registrations and tries to crawl all newly registered sites.
  • Googlebot picks strings out of JS files that look like they might be URLs and crawls those. Pretty much any string literal with no spaces and a slash will get hit by Googlebot.
  • Googlebot submits simple forms with some random options just to see if there is content behind the form.

The bottom line is that you can't rely on secrecy to keep your URLs out of Google. Googlebot is very good at finding private URLs. See Can secret URLs be used to protect files?

There are several ways to fix this:

  1. Include canonical tags in each of your pages pointing to the full URL of the page on www.example.com. Then even if Googlebot crawls the cloud host URL, it will know to index the URL on your domain. Note that canonical tags won't stop Googlebot from crawling the entire alternate domain. Googlebot will even come back and crawl the alternate domain many times. The canonical tags will almost always cause Google to index each page on your preferred domain instead.
  2. Reconfigure your server to redirect the cloud host URL to your domain. You said that you had it configured this way for testing and development, but presumably now that the domain is live, you don't need that. If you do still need it, you could do the redirect except for your own IP address.
  3. Configure your server to serve different robots.txt files for the different host names. Disallow all crawling for the cloud host domain name.
  4. Configure your server to password protect the site on the cloud host domain name. It is usually fairly easy to set up digest authentication that can do that.

Let me put it this way: I've always had Googlebot come crawl my development and staging URLs if I leave them open to the public. It doesn't matter how secret I think I've kept them. Even if I'm the only one that ever uses them, Googlebot seems to find them. I always protect them now, either with a password, or by doing my development on a private network.

| improve this answer | |
  • 1
    You are correct, canonicals are just a hint, and Google is ignoring them more often than ever these days. They should be very effective eventually. The content is 100% the same, Google will see actual duplicates. Your main domain will eventually have a lot more reputation than the cloud host subdomain, so Google will see the canonical and it will be a no-brainer for their bot to obey it. Since your site is new, that may not be the case and Google may see content on the cloud host domain and being more likely to be authoritative. – Stephen Ostermiller Jul 2 at 12:19
  • 1
    Google rarely releases official documentation or talks about specifics of how their crawler works. At best we could run an experiment and see if Googlebot crawls a host name that we make up and are very careful to make sure is only ever mentioned in a DNS CNAME. I don't know of anybody that has done that experiment so far though. – Stephen Ostermiller Jul 2 at 12:22
  • 1
    How are you verifying that the cloud host subdomain is indexed? Are you using site: queries in Google? They don't show which domain is actually indexed. If you search for site:example.cloudhost.com Google will show results that are not actually indexed on that domain but which look like they are in that search result. If you search for phrases from the pages in quotes you should see which domain is actually indexed. You could also add the cloudhost subdomain to Google Search Console and get stats about what Google has actually indexed on it. – Stephen Ostermiller Jul 2 at 12:28
  • 2
    For a 20 year old domain and canonical URLs in place, I wouldn't worry about the reputation of your site suffering. Scraper sites pop up all the time with copies of sites and no canonicals. Google is usually good about preferring the original site and not penalizing it in any way. – Stephen Ostermiller Jul 2 at 12:29
  • 1
    re: verifying indexation. It actually popped up whilst searching for specific "quoted terms" rather than using site:example.cloudhost.com but that's super useful to know it will show things that aren't indexed! – theyetiman Jul 2 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.