Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

This question already has an answer here:

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web.

For him, this means that the only way to not being indexed (for example when a website being developed is already online, but not ready to be used by customers yet) is to restrict the access to the files on the server with a password.

How can Google find a website without following links? If for some reason the site is discovered, isn't robots.txt and a global <meta name="robots" content="noindex"> (on every page) enough?

share|improve this question

marked as duplicate by DisgruntledGoat, John Conde Jun 22 '13 at 12:28

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

add comment

2 Answers 2

up vote 3 down vote accepted

You never know who might be aware of your "hidden pages". Anybody could link to these pages from their site without you even knowing about it. Matt Cutts from Google has addressed this in one of his videos on YouTube.

It's also worth bearing in mind that any 3rd-party scripts you run on your site could report back the URL where they're running, as well as any internal links within the page. This could include Google Analytics, Affiliate Marketing, random jQuery plugins, or other UI scripts.

share|improve this answer
add comment

It's a good question, I'm not sure of the direct answer, but in the past I've had a subdomain with no links to it, i.e. dev.mysite.com being indexed (with mysite.com being an established site).

Of course they can also be discovered if you have Google Webmaster Tools setup.

If you don't want it to be discovered, I would say instruct the robots with robots.txt and meta noindex. For a full belt and braces approach, put a basic password lock on the site using .htaccess and .htpasswd.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.