How to keep crawler robots from indexing my page when there is a query string present in the URL?

Question

I have a /products page.

In that page, users may use filters to select the products they want to see.

Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on.

I don't want good to index any of the filtered pages. I just need the root /products crawled and index.

Through my website, Google will not find those query string links, because my filter buttons are just regular HTML buttons that will call history.push() behind the scenes. So, there are no "visible" links with query strings on my page that Googlebot might find.

On my Sitemap.xml, only the root /products will be included of course.

But it's not impossible that one of my users could post my link somewhere on the web like this:

/products?some_query_string_with_filters

And Googlebot would eventually find out about it, even though I'm not showing it.

So I need a way to block Google from indexing that. What would be the best way of doing that?

OPTION #1

Try to add some Disallow on robots.txt. Which would not allow Google to crawl my page when there is a query string present?

Something like Disallow: /products?

OPTION #2

Let Google crawl it, but render a noindex meta tag when the page is rendered with active filters (which implies the query string would also be active).

OPTION #3

Try to detect (from user-agent) that a robot is fetching the page /products?something, and 301 redirect it to the root page /products without the query string.

OPTION #4

Let google crawl it and always keep a rel canonical pointing to the URL /products without the query string and trust that Google will most likely not index any links with query strings.

Is there a best practice / standard way of doing this?

Answer 1

To prevent Google to index a page, you should use the meta tag:

<meta name="robots" content="noindex">

Blocking the page in robots.txt won't prevent Google to index it, because if someone links your page Google can found it and will crawl the page.

Answer 2

I would recommend using robots.txt to disallow the URLs as you suggest. This will keep Googlebot from crawling all your products URLs with a query string.

Disallow: /products?

From the example query string you gave, it appears that there could be a very large number of possible query strings from various combinations:

• /products?brands=X,Y&size=1,2
• /products?brands=X,&size=1,2
• /products?brands=Y&size=1,2
• /products?brands=X,Y&size=1
• /products?brands=X,Y&size=2
• /products?brands=X&size=1
• /products?brands=Y&size=1
• /products?size=1,2&brands=X,Y
• ...

With so many possible URLs it is usually more important to limit crawling than to limit indexing. Googlebot could spend your site's entire crawl budget crawling query string permutations and not have the bandwidth to crawl your important pages which you would like to have indexed. If you allowed crawling but disallowed indexing with the meta tag, Googlebot would have to crawl each of those URLs before discovering that it can't index them. That is just a waste of resources.

When crawling is disallowed, Google may occasionally still decide to index one of those URLs. Google doesn't usually index blocked URLs, but it may in some rare circumstances such as when a URL gets a large number of external links. URLs that are disallowed but still index appear in the search results saying "A description for this result is not available because of this site's robots.txt". See How to resolve Google “Indexed, though blocked by robots.txt” for more information.

If you really don't want any of those pages indexed, then you should allow crawling and use the meta robots noindex tag. If you do that, you should try make sure that Googlebot doesn't find many of the links. You could:

• Use a POST form to set the filters and redirect to the URL. Googlebot won't crawl POSTed data.
• Build the URLs using JavaScript in an on-click event. Googlebot only evaluates JavaScript that runs when the page loads and not JavaScript that runs on user initiated events.
• Use nofollow on links to the filters: < href="/products?brands=X,Y&size=1,2" rel=nofollow>