In e-commerce sites it is common to have multiple parameters to filter, narrow, sort data. Hence Google provides the URL parameters section in Webmasters.
In our sample site we have the following 2 URLs generated which link to the same content:
and link with URL parameters
We have left the parameters options as "Let Google Decide" - however, it is noticed in the logs that Google is crawling both of the above links.
Why is Google crawling 2 similiar links? Is it because it finds it and hence crawls (seems logical)? But then what is the use of the "Let Google Decide"? Crawling 2 similiar links results in a waste of crawl budget and system resources.
To avoid the above we have 2 options:
- Include a Disallow the size, manufacturer, length in the robots.txt OR
- set each of the URL parameters to no crawl in the Google Webmasters.
Would there be any downside to each any of then options above? Is it a general practice for e-commerce stores to block all parameter related data (carefully ofcourse) since most of it is in general duplicate data.