I have full text search engine installed on my site, accessible from URL like /search/<search-query>, and search landing in /search, containing large "Search" button and some popular tags. Do I need to hide these pages from search bots using robots.txt Disallow entries?

  • Absolutely no reason why you should want to hide a search page. Pages that are useful to users should be indexable by search engines unless you don't want people using the page. Indexing a search page is useful for example "twitter advance search". Nov 6 '16 at 11:20
  • 3
    @SimonHayter The question would seem to be addressing search results pages, not just the search landing page. (?)
    – MrWhite
    Nov 6 '16 at 18:30
  • @w3dk well, yes, my question addresses them both :) Nov 6 '16 at 18:54
  • 1
    Search, Yes. Search Results, No. It is that simple. Google hates search results pages. Don't do it.
    – closetnoc
    Nov 6 '16 at 22:10

Indexing search result pages isn't always the best idea - check this article from Matt Cutts quote: "Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index."

It's quite possible that your search result pages do offer added value for visitors - so difficult to judge whether or not you should block them in the robots.txt. If it's just a list with articles on the same topic (similar to /tag/ pages) - I would consider to put them on noindex / follow.

  • 1
    Since the OP does also mention the /search landing page itself - it's worth noting that this can/should be indexed (in contrast to the typical search results pages). Also, with respect to the search results pages and specific to the OP's question regarding the use of robots.txt, Google's Webmaster Guildines explicitly states: "Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages."
    – MrWhite
    Nov 7 '16 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.