Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

From what I understand after reading Google's Controlling Crawling and Indexing:

  • The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in search results) if they are linked from crawlable pages
  • To prevent a page from being indexed I need to make it crawlable and add the noindex meta tag in its head

So, why would I setup a robots.txt file if, in the end, it has no impact on whether pages appear in search results or not?

share|improve this question

marked as duplicate by bybe Mar 11 at 9:27

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer 1

in the end, it has no impact on whether pages appear in search results or not?

You are wrong , if you prevent a page from being crawled , it means it will never show up in result search.

Let's suppose that you have a thread page that contains pretty good and rich content , then you have a comment section,
the comment section has 10 pages but not very interesting content. You will need to prevent google from crawling these pages to act faster and reference the threads that have good content. That's just one example , there are other security reasons.

share|improve this answer
You misunderstood my last sentence: robots.txt disallow crawling but not indexing, therefore a page can still appear in search results even if it's disallowed in robots.txt –  fiddler Mar 11 at 8:10
@fiddler - that is correct. Indexation can still occur even if you have a Disallow directive in robots.txt. To prevent indexation, you'd have to use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> in the <head> of the necessary pages. –  zigojacko Mar 11 at 8:30
@GeoffJackson-zigojacko Hence my question: why would I use a robots.txt then? –  fiddler Mar 11 at 8:30
@fiddler - To prevent the crawling of content on pages. Robots.txt may not prevent the indexing of URL's but it won't index the content - it'll just show a snippet along the lines of 'robots.txt prevented this page from being shown'. –  zigojacko Mar 11 at 8:31
@fiddler Indexing is a further step than crawling , you can not index a page if you aren't allowed to crawl it, indexing as mentioned is saving the content into a database (Google takes generally 250 characters of your content) , so if a robot is not allowed to fetch the page , it will never get the engine to index it. –  Aymane Shuichi Mar 11 at 8:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.