4

There's a /search path on my web app that I don't want robots to crawl or index.

What would be the best approach for this scenario?

  • 1: Block them from accessing by adding a Disallow for /search on robots.txt
  • 2: Add a rel="noindex" to the links that lead to the /search URLs.

What is the best practice in this case?

1
  • A "URL" is what you can type into your browser to identify a document. A "route" is a mapping between URLs and code. While you might have a routing rule that maps /search URLs to code that does the searching, you should refer to /search as a URL and not as a route. Commented Nov 18, 2020 at 1:12

1 Answer 1

5

Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt.

However, that doesn't necessarily prevent the /search pages from appearing as link only results in the SERPS (if other pages link to them) - but this is unlikely. And they are very unlikely to rank higher than other search results anyway.

2: Add a rel="noindex" to the links that lead to the /search route.

noindex is not a recognised rel value for Google, maybe you mean rel="nofollow"?

You can set rel="nofollow" on the link to the search pages. However, this won't necessarily prevent the target pages from appearing in the SERPs (if there are other links to them) or even from being crawled, although it should. (This value appears to have been demoted to an advisory value by Google).

To prevent a page from being indexed then you need to include a <meta name="robots" content="noindex"> tag on those pages. Or serve with an X-Robots-Tag HTTP response header. But in this case, you can't block crawling in robots.txt, otherwise the bot won't see the meta tag (or response header).

Depending on your use case, it might be beneficial to allow Google to crawl your search pages (in order to find internal pages), but not to index them. However, you don't necessarily want to waste Google's crawl budget on your search pages either, so it can depend on the nature of your site.

2
  • Yes, I meant rel="nofollow". Thanks a lot for the detailed answer. Commented Nov 17, 2020 at 19:30
  • 2
    I'd add: Google doesn't want to index site search results pages and can penalize sites that try to index their search results. Using robots.txt to block crawling of the search results is good enough. When crawling is blocked, Google rarely indexes any of them and won't penalize your site. Commented Nov 18, 2020 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.