Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
Is it possible for web crawlers to see static pages without following a link to them?

I have some URLs (some pdfs and static HTML files) in my website that I want only few people to know. These URLs dont have any pointing links from my website or any other source.

So my question is this: Can google crawl an URL (say some static HTML file) that has no incoming link?

share|improve this question

marked as duplicate by paulmorriss, danlefree Feb 25 '12 at 9:58

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public.

In addition You can always use the robot.txt file to tell Google not to crawl the URLs.

But this is just security by obfuscation, if you really want to protect them use a proper way (authentication/authorization)

share|improve this answer
+1 for adding the note about authentication/authorisation. The only real way to guarantee search engines won't be able to reach these pages is to have them in a secure area in such a way that even requesting the path directly triggers the authentication. – Zhaph - Ben Duguid Feb 24 '12 at 14:39

Yes, Google will find it somehow!

They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler.

So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also can't control what users post to social media sites and the like.

You can prevent it's inclusion in google's index though, robots.txt, a simple text file which sits in the www root directory of your server, will stop the GoogleBot in it's tracks.

House your non-google pages in a single directory and exclude like this:-

User-agent: *
Disallow: /your-directory-name/

As @Matteo and @Zaph note this isn't real protection and won't stop determined users finding your content. I use .htpasswd to block areas on my sites in conjunction with Coffee Cup Website Access Manager which outputs hashed htpasswd files and uploads them to your site, and provides multi user management.

share|improve this answer
Note that this doesn't stop people from snooping at your robots.txt file and having a look themselves at directories and files that you list there. Your best bet is probably to password protect the directory as well as mention it in robots.txt. – Andy Smith Feb 24 '12 at 15:00
I don't think this answer makes sense. How can "monitoring search history" help Google index pages if the files aren't being returned in search results? – DisgruntledGoat Feb 24 '12 at 23:51
@DisgruntledGoat in essence Google uses input from the search history in your Google account/toolbar/site search/analytics to 'train' its bot. Put another way, in some respects, you direct the crawler. – toomanyairmiles Feb 25 '12 at 0:16
@toomany but I don't see how that leads to discovery of URLs. Let's assume Google doesn't know about a URL. Then you search Google for something that would match that URL. Of course that URL is not returned as Google doesn't know about it. So how can it get the URL from your search history when it's not there? And Google Analytics (and Adsense for that matter) are completely separate, they don't share data. – DisgruntledGoat Feb 25 '12 at 0:40
@DisgruntledGoat User visits previously unknown URL (search history etc enabled) > Google discovers URL > Google indexes URL – toomanyairmiles Feb 25 '12 at 1:01

As long as there is no incoming link or anything that can point google towards the file (sitemap, open directory structure, etc..) then I believe that it shouldn't be indexed. alternatively you could put the files in a folder and block it through the robots file.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.