I am working on a web project that has nearly 100 thousand instant users and there is a webpage that we are using for test cases. There are no links pointing to it from other pages. It shouldn't be indexed by Google or any other search engines.

"noindex" can be used in this situation, I know but I wonder if Google (or any others) indexes this page, if I don't do anything to prevent it.


3 Answers 3


Googlebot (the web crawler of Google) can discover your page by different ways:

  • if it finds a link to your page from a web page (inside or outside your domain name)
  • if you submit the URL to your Google Webmaster Tools account through a sitemap.xml

In that case, the page can be indexed. In most of cases, Googlebot decides to index it (for example, it's not the case when your page has a lot of duplicate content with another page).

If you don't want this page indexed, just put this <meta name="robots" content="noindex, nofollow"> tag in your page and block the page with your robots.txt like this:

User-agent: Googlebot (or * for all web crawlers)
Disallow: /the-page

If web crawlers get to this page one way or another, and if it is not marked as noindex and it is not blocked in your robots.txt, then yes, they may decide to index it.

You may also use noarchive to make sure they don't keep a cached version of it.


No, Google doesn't automatically index your pages, if no other site links to them. You should put a robots.txt file on the root directory of your server, which can prevent the search engine from indexing the pages of a specific folder.

It could look like this:

User-agent: *
Disallow: /
  • 3
    It's better to use the NOINDEX robots tag on the pages because it might index pages even if it's not "allowed to crawl". Commented Aug 21, 2014 at 12:06
  • @Christoph Bühler: Allan S. Hansen is right and no, if a page has no connection, Google can't find it.
    – Zistoloen
    Commented Aug 21, 2014 at 12:36
  • Google can find pages without inbound links when they are in accessible sitemaps. Commented Aug 21, 2014 at 12:59
  • @JVerstry: This is what I said in my answer :-).
    – Zistoloen
    Commented Aug 21, 2014 at 15:20
  • Sorry... too much coffee I guess... lol Commented Aug 21, 2014 at 16:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.