This is just a question of curiosity. Let's say I register for a new web domain and purchase a hosting account. I set up a website, and soon search index bots are visiting the site to learn more about it. How did those bots know how to find the site if I haven't linked to it from anywhere else on the web?

My guess would be that the search providers obtain new IPs from either the domain name registrar or the hosting provider and feed these to their spiders, but I've been unable to find a clear answer on this.

  • I apologize for posting what amounts to a duplicate question; mine was slightly more generic (not specific to Google) and I didn't find that answer when searching SE. – user45623 Feb 11 '16 at 4:35

Just a few, I expect the list to go on and on...

  • Backlinks
  • DNS/Name Servers
  • Manual Submission
  • Domain Whois (Published List or Cache)
  • 1
    Registrars also feed Google (at least). Or they used to. Google being a registrar may also give them an automatic advantage that we are not aware of. Of course there also the various webmaster tools that SEs offer and submission sites. – closetnoc Feb 10 '16 at 20:08
  • 1
    Domain Whois (Published List) :) – Simon Hayter Feb 10 '16 at 20:29
  • 1
    I am seeing Google hit new sites within about 10-20 minutes of registration in the past 3 years. – closetnoc Feb 10 '16 at 20:33
  • 2
    Most rogue spiders just crawl from the newly registered domain lists publicly available on several sites. That's where most SEO, web design, etc... spam comes from that we all get if not using a privacy/proxy service for our registrations. – dan Feb 11 '16 at 2:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.