Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

Question

I've written my own SiteSearch Script in PHP.

The SiteSearch parses two GET parameters from the queryString:

• search // the search-phrase
• filters (optional) // indicating which parts of the site to include or ignore

I don't mind GoogleBot and other crawlers reading the search parameter.

But I would like to advise crawlers to ignore the filters parameter, because a very high number of configurations of that parameter would simply return the same results - and an identical, duplicate page as far as the crawlers are concerned.

---

As much as I would like to add to my robots.txt file something like:

User-agent: *
IgnoreParameter: filters

this isn't an option.

And a meta robots directive like:

<meta name="robots" content="ignoreparams[filters]">

isn't an option either.

Is there any creative way I can enable crawling of the page and have crawlers ignore the filters parameter in the queryString?

Or am I stuck with a directive as unrefined as:

<meta name="robots" content="noindex">

if I don't want to risk pages with identical search parameters (but different filters parameters) being crawled?

Answer 1

As discussed in the comments, you should disallow your entire site search results. However, there may be other pages for which you want to have search bots ignore some parameters. There are two ways to get crawlers to ignore specific URL parameters:

Robots.txt wildcards

Google and many other search engines support wild cards (*) in robots.txt rules. You can use this functionality to disallow URLs that contain specific URL parameters:

Disallow: /path?*filter=

While the big search engines know how to interpret wildcards, most crawlers don't. Rules with wildcards only work with some crawlers.

Webmaster tools and search console settings

Google has a tool in search console called the URL Parameters tool. Using this tool you could tell Google that the filter parameter "doesn't affect page content."

Answer 2

It's recommended best practice to disallow indexing of Site Search Pages.

Note that while this can be done in robots.txt, using:

User-agent: *
Disallow: /search/

where /search/ is the folder containing the Site Search.

Yoast recommends using meta robots instead:

<meta name="robots" content="noindex, follow">

See:

[...] blocking internal search pages via your robots.txt isn’t the best way to do this. It’s much better to allow search engines to crawl these kinds of pages, but to prevent them from indexing them by using a meta robots tag with a value of noindex, follow.

Source: https://yoast.com/blocking-your-sites-search-results/

---

Then and Now...

On March 10th, 2007, Matt Cutts, Head of Search Quality at Google, wrote:

In general, we’ve seen that users usually don’t want to see search results [...] in their search results. [...] search results that don’t add much value already fall under our quality guidelines [...] so Google does take action to reduce the impact of those pages in our index.

The new webmaster guideline that you’ll see on [... the quality guidelines page...] says “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

Source: https://www.mattcutts.com/blog/search-results-in-search-results/

On March 10th, 2020, Barry Schwartz of Search Engine Roundtable wrote:

In 2007, Google told webmasters to block internal search results from being indexed. The original guideline read "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines." Now it reads "Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages."

[...] Google's John Mueller explained why Google doesn't want your search result pages in its index. He said "they make infinite spaces (crawling), they're often low-quality pages, often lead to empty search results/soft-404s."

So it really wasn't always about spam but blocking pages that might not be as relevant to Google.

Source: https://www.seroundtable.com/google-indexing-search-result-pages-29131.html

---

Further Reading:

• Mar 2007: Search Results in Search Results (Matt Cutts at Google)
• Apr 2015: Always Block Google from Accessing your Site's Search Results (labnol.org)
• Aug 2018: Block Your Site’s Search Result Pages (yoast.com)
• Mar 2020: Google: Indexing Search Result Pages Waters Down Indexed Pages (SE Roundtable)
• Mar 2020: Should Internal Search Results Pages be Indexed? (Practical eCommerce)
• Nov 2020: Should Search Results Pages be Indexed? The Impact on SEO (GoInFlow)