2

I've written my own SiteSearch Script in PHP.

The SiteSearch parses two GET parameters from the queryString:

  • search // the search-phrase
  • filters (optional) // indicating which parts of the site to include or ignore

I don't mind GoogleBot and other crawlers reading the search parameter.

But I would like to advise crawlers to ignore the filters parameter, because a very high number of configurations of that parameter would simply return the same results - and an identical, duplicate page as far as the crawlers are concerned.


As much as I would like to add to my robots.txt file something like:

User-agent: *
IgnoreParameter: filters

this isn't an option.

And a meta robots directive like:

<meta name="robots" content="ignoreparams[filters]">

isn't an option either.

Is there any creative way I can enable crawling of the page and have crawlers ignore the filters parameter in the queryString?

Or am I stuck with a directive as unrefined as:

<meta name="robots" content="noindex">

if I don't want to risk pages with identical search parameters (but different filters parameters) being crawled?

5
  • 1
    You should disallow your entire site search in robots.txt. Google will penalize your site if they find they are indexing your site search results pages. They don't want to index your site search results because it is bad user experience for a user to leave the Google search results only to land on other search results. See Matt Cutts: Search results in search results. You shouldn't be worrying about having crawlers ignore just one parameter, you should have crawlers ignore the entire site search. – Stephen Ostermiller Apr 29 at 19:12
  • Aha. Well, that would make everything a lot easier! That Matt Cutts blog post is from March 2007, though... [1/2] – Rounin Apr 29 at 20:31
  • I wondered if anything might have changed in the last 14 years. I ran some searches and came across article from (only) just over a year ago (March 2020) by Barry Schwartz (aka rustybrick) at SE Roundtable: Google: Indexing Search Result Pages Is A Watering Down Indexed Pages Issue which references the original 2007 advice from Google, compares it with what was said in 2020 by John Mueller (in many ways Cutts' successor) and draws the same broad conclusion, albeit from a more nuanced perspective. [2/2] – Rounin Apr 29 at 20:31
  • What was bad user experience 14 years ago is still bad user experience today. Google still doesn't want user clicking from their search results to other search results. – Stephen Ostermiller Apr 29 at 20:32
  • Google has doled out manual penalties for indexed site search. They even sometimes put in a term like "viagra" and see if it has indexable site search results and then penalize for spam because of it. They may have multiple motives, but it isn't worth risking your SEO by having Google crawl your site search results. – Stephen Ostermiller Apr 29 at 21:22
1

As discussed in the comments, you should disallow your entire site search results. However, there may be other pages for which you want to have search bots ignore some parameters. There are two ways to get crawlers to ignore specific URL parameters:

Robots.txt wildcards

Google and many other search engines support wild cards (*) in robots.txt rules. You can use this functionality to disallow URLs that contain specific URL parameters:

Disallow: /path?*filter=

While the big search engines know how to interpret wildcards, most crawlers don't. Rules with wildcards only work with some crawlers.

Webmaster tools and search console settings

Google has a tool in search console called the URL Parameters tool. Using this tool you could tell Google that the filter parameter "doesn't affect page content."

1
  • Thanks for shining a light on Google's URL Parameters Tool. It came up in my searches (on this page: Taming Parameters by DeepCrawl.com ) alongside Bing's Ignore URL Parameters Tool (though I'm not sure the latter even exists anymore). – Rounin Apr 29 at 21:38
1

It's recommended best practice to disallow indexing of Site Search Pages.

Note that while this can be done in robots.txt, using:

User-agent: *
Disallow: /search/

where /search/ is the folder containing the Site Search.

Yoast recommends using meta robots instead:

<meta name="robots" content="noindex, follow">

See:

[...] blocking internal search pages via your robots.txt isn’t the best way to do this. It’s much better to allow search engines to crawl these kinds of pages, but to prevent them from indexing them by using a meta robots tag with a value of noindex, follow.

Source: https://yoast.com/blocking-your-sites-search-results/


Then and Now...

On March 10th, 2007, Matt Cutts, Head of Search Quality at Google, wrote:

In general, we’ve seen that users usually don’t want to see search results [...] in their search results. [...] search results that don’t add much value already fall under our quality guidelines [...] so Google does take action to reduce the impact of those pages in our index.

The new webmaster guideline that you’ll see on [... the quality guidelines page...] says “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

Source: https://www.mattcutts.com/blog/search-results-in-search-results/

On March 10th, 2020, Barry Schwartz of Search Engine Roundtable wrote:

In 2007, Google told webmasters to block internal search results from being indexed. The original guideline read "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines." Now it reads "Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages."

[...] Google's John Mueller explained why Google doesn't want your search result pages in its index. He said "they make infinite spaces (crawling), they're often low-quality pages, often lead to empty search results/soft-404s."

So it really wasn't always about spam but blocking pages that might not be as relevant to Google.

Source: https://www.seroundtable.com/google-indexing-search-result-pages-29131.html


Further Reading:

1
  • Thanks, @StephenOstermiller - your quick answer in the comments above saved me potentially several hours of work this evening. – Rounin Apr 29 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.