In e-commerce sites it is common to have multiple parameters to filter, narrow, sort data. Hence Google provides the URL parameters section in Webmasters.

In our sample site we have the following 2 URLs generated which link to the same content:


and link with URL parameters


We have left the parameters options as "Let Google Decide" - however, it is noticed in the logs that Google is crawling both of the above links.

Why is Google crawling 2 similiar links? Is it because it finds it and hence crawls (seems logical)? But then what is the use of the "Let Google Decide"? Crawling 2 similiar links results in a waste of crawl budget and system resources.

To avoid the above we have 2 options:

  1. Include a Disallow the size, manufacturer, length in the robots.txt OR
  2. set each of the URL parameters to no crawl in the Google Webmasters.

Would there be any downside to each any of then options above? Is it a general practice for e-commerce stores to block all parameter related data (carefully ofcourse) since most of it is in general duplicate data.

  • Why are you using the parameters on the URLs at all? If you have the short URLs set up you should stop linking to the parameter version from any place. Then Google will never find them. Commented Sep 29, 2016 at 21:02

3 Answers 3


The Google Bot tries to craw everything mentioned or linked on your site / the whole www. I set up a test case and the bot even crawled urls like this:

// Even a url in a JS comment is crawled by google: stackoverflow.com

And i think it's more about "let google decide what url they serve the user" and not "let google decide what url they will crawl" in the WMT.

In case of a faceted navigation you have to be careful what you want to be indexed. In general it's best practice to set all options to "noindex, follow". "Follow" cause you want the Google Bot to crawl your detail pages.

www.samplesite.com/dresses/women.html = Index, Follow
www.samplesite.com/dresses/women.html?size=10 = NoIndex, Follow
www.samplesite.com/dresses/women.html?color=red = NoIndex, Follow
www.samplesite.com/dresses/women.html?page=2 = NoIndex, Follow 

If you have 5 categories and 50 products but 5k sites in the google index your site most likely will not perform well. On the other hand, if you think your site is strong enough, you can try to open one option to get some long tail keywords like "red women dresses" to rank.

  • Agree with you about "let google decided what url they serve the user" - makes sense. However, Why would I want Google to follow and crawl all the urls you mentioned since they don't have any additional data other than what www.samplesite.com/dresses/women.html already has. Isn't it just a waste of Crawl budget.
    – HopeKing
    Commented Aug 29, 2016 at 10:28
  • Also, can you suggest which of the 2 options (blocking through robots.txt or blocking through webmasters) is better. I get the sense that it is robots.txt from some of your explanation.
    – HopeKing
    Commented Aug 29, 2016 at 10:29

You can solve it with canonical tag in head of your pages: for example you set this canonical tag:

<link rel="canonical" href="www.example.com/dresses/women/prada-size32-kneelength.html" />

for two Urls above:

  • There are 1000's of url combinations possible. Doesn't work manually as you suggested.
    – HopeKing
    Commented Aug 30, 2016 at 3:38
  • If you have a short URL, you should set the canonical for the non-short version of the URL as suggested in this post. The idea is that you can tell Google exactly which version of URL you prefer to have indexed. The downside is that it won't prevent crawling. Googlebot still crawls the parameters before it sees the canonical. Commented Sep 29, 2016 at 21:01

I had that happen to me. So Google will try and crawl everything on your site, and I've even had Google bug out on me and ignore my robots.txt once. It took a month for Google to correct itself again!

Also, I've had Google moan at me under HTML Improvements about duplicate content where it has crawled random pages with URL Parameters. Once I had gone over each one of my URL parameters and manual configured each entry, the duplicate content warnings stopped appearing over a few weeks. The only downside here is if you pick the wrong URL parameter to be ignored.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.