recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These attribute filters are implemented as URL parts added to product category URL like so:
Simple product category URL:
Product category URL with one attribute filter:
Product category URL with two attribute filters:
And so on.....
So having around 30 attribute filters there can be A LOT of their combinations and therefore exponentialy increasing number of all available URLs to Google, for example:
www.domain.com/product-category/filter5/filter9/filter12 and so on ....
Google's traffic hit us very hard, because right now is continuous traffic rate around 30Mbits only to Google alone, indexing right now around 100 million pages from our domain (before this filter feature we had around 5000 indexed pages).
What I need to do is to find a way how to tell Google not to crawl all the 100 million pages over and over again, but only, let's say, once a week or so.... Because right now, Google is crawling all the pages over and over again.
I have tried to set the HTTP headers
cache-control: public, max-age:604800 and
expires: header one week in future. But I have no idea if Google will use them at all. I don't want to block Google from entirely indexing these pages with attribute filter in URL, but I just want Google not to crawl them too often.