Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'd like to create a "private" website that clients can visit by typing in the provided direct URL (on a business card, for example), but is not password protected and would not be indexed by search engines. Is that possible?

share|improve this question

Probably not.

If any of your clients post the URL in just about anything, search engines will likely discover it. They may also find it while crawling domain registration sites and the like...

As Google states here:

If you need to keep confidential content on your server, save it in a password-protected directory. Googlebot and other spiders won't be able to access the content. This is the simplest and most effective way to prevent Googlebot and other spiders from crawling and indexing content on your site.

Even if you try to block crawling using robots.txt, noindex meta tags, or X-Robots-Tag HTTP headers, any mention of your site anywhere may result in crawling and indexing it by various search engines. As Google indicates:

It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index. For example, other sites may still link to it. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project, can appear in Google search results. In addition, while all respectable robots will respect the directives in a robots.txt file, some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password-protecting confidential information (see above).

share|improve this answer

The easiest way would be to use htaccess to send an HTTP header to tell the search engines not to index any of those pages. Put the following code in an htaccess file in your root web directory:

Header: "X-robots-tag: noindex"

This will apply this header to every page in that directory and subdirectory below it and prevent those pages from being indexed.

You can read more about this in Google's Webmaster Blog.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.