1

This question already has an answer here:

I am working on a web project that has nearly 100 thousand instant users and there is a webpage that we are using for test cases. There are no links pointing to it from other pages. It shouldn't be indexed by Google or any other search engines.

"noindex" can be used in this situation, I know but I wonder if Google (or any others) indexes this page, if I don't do anything to prevent it.

marked as duplicate by John Conde Aug 21 '14 at 13:21

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2

Googlebot (the web crawler of Google) can discover your page by different ways:

  • if it finds a link to your page from a web page (inside or outside your domain name)
  • if you submit the URL to your Google Webmaster Tools account through a sitemap.xml

In that case, the page can be indexed. In most of cases, Googlebot decides to index it (for example, it's not the case when your page has a lot of duplicate content with another page).

If you don't want this page indexed, just put this <meta name="robots" content="noindex, nofollow"> tag in your page and block the page with your robots.txt like this:

User-agent: Googlebot (or * for all web crawlers)
Disallow: /the-page
1

If web crawlers get to this page one way or another, and if it is not marked as noindex and it is not blocked in your robots.txt, then yes, they may decide to index it.

You may also use noarchive to make sure they don't keep a cached version of it.

0

No, Google doesn't automatically index your pages, if no other site links to them. You should put a robots.txt file on the root directory of your server, which can prevent the search engine from indexing the pages of a specific folder.

It could look like this:

User-agent: *
Disallow: /
  • 3
    It's better to use the NOINDEX robots tag on the pages because it might index pages even if it's not "allowed to crawl". – Allan S. Hansen Aug 21 '14 at 12:06
  • @Christoph Bühler: Allan S. Hansen is right and no, if a page has no connection, Google can't find it. – Zistoloen Aug 21 '14 at 12:36
  • Google can find pages without inbound links when they are in accessible sitemaps. – Jérôme Verstrynge Aug 21 '14 at 12:59
  • @JVerstry: This is what I said in my answer :-). – Zistoloen Aug 21 '14 at 15:20
  • Sorry... too much coffee I guess... lol – Jérôme Verstrynge Aug 21 '14 at 16:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.