Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]

Question

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web.

For him, this means that the only way to not being indexed (for example when a website being developed is already online, but not ready to be used by customers yet) is to restrict the access to the files on the server with a password.

How can Google find a website without following links? If for some reason the site is discovered, isn't robots.txt and a global <meta name="robots" content="noindex"> (on every page) enough?

Answer 1

You never know who might be aware of your "hidden pages". Anybody could link to these pages from their site without you even knowing about it. Matt Cutts from Google has addressed this in one of his videos on YouTube.

It's also worth bearing in mind that any 3rd-party scripts you run on your site could report back the URL where they're running, as well as any internal links within the page. This could include Google Analytics, Affiliate Marketing, random jQuery plugins, or other UI scripts.

Answer 2

It's a good question, I'm not sure of the direct answer, but in the past I've had a subdomain with no links to it, i.e. dev.mysite.com being indexed (with mysite.com being an established site).

Of course they can also be discovered if you have Google Webmaster Tools setup.

If you don't want it to be discovered, I would say instruct the robots with robots.txt and meta noindex. For a full belt and braces approach, put a basic password lock on the site using .htaccess and .htpasswd.