2

I know Google can find domains by checking with registrars, but is it able to find new subdomains with no backlinks?

I suspect they are able to by checking DNS records, but can anyone confirm/deny?

  • It's not so easy to get all the subdomains from the DNS - unless you already know what you are looking for (catch-22). Most DNS servers block AXFR requests - which would be required to retrieve a list of the subdomains. – MrWhite Dec 9 '16 at 20:59
  • @w3dk You do not need to do AXFR. Any 'dig -all` style command will return what you want. There are code modules that do this extremely well in a formatted way so that code wise, getting DNS records for any domain is simple pimple. – closetnoc Dec 10 '16 at 2:37
  • To answer your question, most search engines will seek new domain names (to the search engine) and will follow links. That is it. At one point, there was a concept of deep web where search engines actively sought out information that it would not normally have access to. That notion failed to some extent except for javascript rendering, AJAX, and so forth that act as part of the user interface. And still, through all of that, search engines never really sought out sub-domains. I have not seen this behavior with Google at least. – closetnoc Dec 10 '16 at 2:43
  • How do I get a list of all subdomains of a domain? seems to indicate that DNS won't be able to give you a list of subdomains in pretty much any case. – Stephen Ostermiller Dec 10 '16 at 12:34
  • @closetnoc But how does dig -all do it? As far as I know... a DNS zone transfer (AXFR query) is the only way to reliably get a complete list of the subdomains? Other techniques would seem to use brute force or rely on spidered data, which is time consuming and not 100% reliable. (?) – MrWhite Dec 10 '16 at 12:44
1

Hard to say without doing case study, I will say Google will find out those pages if they consider them as popular/important pages (If those pages are getting zero visits then I will say Google will not going to gather any information from anywhere), so to do case study, create simple landing page and block that page via robots.txt so Googlebot don't crawl it then redirect your all visitor to subdomain (Actually you should not taste subdomain, instead test any webpage like example.com/secreat/xyz.html and don't link them anywhere, not in sitemap as well)).

But I think Google will index your content because Google may already using their product info (Chrome user history, Google public DNS etc) to index popular/important content.

We all know nofollow links are not crawled by Google directly. So whenever any user share any information to social media(Which use nofollow tag on all links), and if many of visitors read those content, then I confirm that Google will index that page faster, without checking your sitemap/links first.

Web are too big now, they may already implement many of crawlers other than based on backlinks, and Pagerank.

Google does not share so much information about crawling/ranking compare to 2003 - 2007, so can't confirm anything.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.