1

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked to from anywhere then it shouldn't get crawled.

For example, let's say I own example.com and I make a new subdomain with a weird random name such as adayinthewoods.example.com and I throw a wordpress installation on there that I plan to use for testing purposes.

What would have to happen for google to start crawling and indexing this? Is google going to look at the whois records and see that I've added a subdomain in my DNS table and then start crawling it as a result? Is the fact I've installed wordpress on it making my installation "ping search engines"? How does that work? How do top level domains get crawled when it's a brand new domain? I assume the mechanism there is different from random subdomains.

What if I add a new page in my root folder called "noOneWillEverSeeThis.html", could that ever get crawled/indexed if it wasn't included in any sitemap and wasn't linked to from anywhere?

Would really appreciate a solid answer from someone that understands what's going on with this.

Thanks much

  • Google does not sniff DNS records for sub-domains (does sniff whois, but whois is not DNS), but it can get feeds for new domains created from some registrars (important to remember) though I am sure that is parent domains only. However, if someone creates a link or a citation (and this is important to remember), then Google may taste the site for an HTTP response. If a response is given, then Google will index the site. Now Google will examine DNS results for network quality but that is completely different. – closetnoc Jun 6 '15 at 21:59
4

Yes, Google can do that and it is best to assume that anything that is publicly available on the internet may be indexed by Google. Linked to or not.

Of course, if you don't link to it the chances of it being indexed go way down. However, Google uses a multitude of tools to gather URLs for indexing. Recently there was a news item about Dropbox links that had been shared (thus becoming publicly available) being indexed by Google because people clicked on links in the documents or put their URLs in Google search box.

It doesn't really matter how Google finds the link. The point is that it may.

So the bottom line is that if you don't want something to wind up in Google you must put a robots.txt file in place to keep Google out. Relying on obscurity is not advised.

2

Think of the linking as a chain reaction. Google won't link to domains if it has no way of accessing it or even finding it. If a friend advertises your URL on a popular forum site that Google always indexes, then there's a chance Google will scan your URL and possibly index it, thinking the link may be part of the site.

Is the fact I've installed wordpress on it making my installation "ping search engines"? How does that work?

I doubt content management systems will randomly ping search engines, but you can always check the source code and see if any code contains commands for opening remote URL's. Examples of such code in PHP (which is what wordpress uses) might include (in no particular order):

$data=file_get_contents("http://www.searchengine.com/submittoengine/data.cgi?whatever=whatever");

$remote=fopen("http://www.remote.com/upload.cgi?website=bla.com");

Or even curl functions which may include:

curl_exec($webdata);

A good way to see how wordpress behaves network wise is to create your own LAMP/WAMP setup. This means use Linux or Windows and install Apache, MySQL and PHP on one computer then disconnect from your actual internet and setup Apache so that you can access content when you type in one of the following URLs:

http://127.0.0.1/
http://localhost/

When apache is installed correctly for the first time regardless of the status of your real internet connection, you should see something like "It works" or something other than a "could not connect to remote server" type of message. Next, install wordpress and then see if it complains about the internet or pinging, etc. I bet it wouldn't.

Everything is best done by experimentation.

  • 2
    I looked further into this after posting and found that Wordpress among other CMSs do ping search engines on your behalf. And while I can't speak for all the other CMSs, I now know that wordpress does this by default whenever you publish a new post. This is explained in their Update Services doc here codex.wordpress.org/Update_Services and it says it is turned on by default on all installations. – vesperknight Jun 7 '15 at 12:33
  • So in your case, it all comes down to wordpress. If you made posts without the need of any CMS and posted a bunch of links in that post to new random domains by hand (for example, by directly modifying HTML source code), then google would never know about it unless the links got advertised somewhere, but now that Wordpress has an updating service, you can expect it to attempt to index almost everything you put into it. – Mike Jun 7 '15 at 15:48
-1

Nope if you sub domain was not having any high quality content or any link from anywhere (from main domain or from anywhere else) google was not going to index/crawl your random subdomains.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.