New answers tagged robots.txt
1
As discussed in the comments, you should disallow your entire site search results. However, there may be other pages for which you want to have search bots ignore some parameters. There are two ways to get crawlers to ignore specific URL parameters:
Robots.txt wildcards
Google and many other search engines support wild cards (*) in robots.txt rules. You ...
1
It's recommended best practice to disallow indexing of Site Search Pages.
Note that while this can be done in robots.txt, using:
User-agent: *
Disallow: /search/
where /search/ is the folder containing the Site Search.
Yoast recommends using meta robots instead:
<meta name="robots" content="noindex, follow">
See:
[...] blocking ...
Top 50 recent answers are included
