How to limit Google crawling certain URL patterns?

Question

recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These attribute filters are implemented as URL parts added to product category URL like so:

Simple product category URL: www.domain.com/product-category

Product category URL with one attribute filter: www.domain.com/product-category/filter1

Product category URL with two attribute filters: www.domain.com/product-category/filter1/filter2

And so on.....

So having around 30 attribute filters there can be A LOT of their combinations and therefore exponentialy increasing number of all available URLs to Google, for example: www.domain.com/product-category/filter2/filter4 or www.domain.com/product-category/filter5/filter9/filter12 and so on ....

Google's traffic hit us very hard, because right now is continuous traffic rate around 30Mbits only to Google alone, indexing right now around 100 million pages from our domain (before this filter feature we had around 5000 indexed pages).

What I need to do is to find a way how to tell Google not to crawl all the 100 million pages over and over again, but only, let's say, once a week or so.... Because right now, Google is crawling all the pages over and over again.

I have tried to set the HTTP headers cache-control: public, max-age:604800 and expires: header one week in future. But I have no idea if Google will use them at all. I don't want to block Google from entirely indexing these pages with attribute filter in URL, but I just want Google not to crawl them too often.

Answer 1

It is not a good idea to let Googlebot crawl URLs that can be constructed with combinations of parameters. When you allow your categories to be filtered, my advice is to only allow Googlebot to crawl pages with a single filter applied, even if you allow users to view multiple filters at the same time.

To implement this, you need to change your URLs:

• /product-category/filter1 is fine
• /product-category/filter1/filter2 needs to have a non-search-engine friendly URL like /view-filters?category=product-category&filters=filter1/filter2

to change your URLs cleanly:

• Change all the links to filter pages to use the new URL structure
• Redirect your current URLs with multiple filters applied to the new URLs
• Add Disallow: /view-filters to robots.txt

The reason to change your URLs is because robots.txt rules are "starts with" rules. The best way to prevent bots from crawling some URLs but not others is to give the uncrawlable URLs a common prefix.

Unlike most bots, Googlebot allows wildcard rules. If you don't want to change your URLs, an option that could work for Googlebot would be to disallow any page with three slashes: Disallow: /*/*/*. That would disallow crawling of /product-category/filter1/filter2 but not /product-category/filter1. Of course it would also disallow crawling of any deep directory structure on your site that isn't related to product category filters as well.

There is no way to tell Googlebot how often to come back and crawl a URL. Your cache control headers are not effective for bots. Bots don't use them. Rather Googlebot determines how often it should crawl a URL based on how many other pages link to it. A popular page like the CNN home page will get crawled every few minutes. A page that is 6 hops away from your home page and only linked once may only get crawled every few months. The only way to control Googlebot's crawl frequency is to very carefully control the Pagerank passed to each URL, which isn't very feasible.

Your only real way of controlling Googlbot is the binary option given to you by robots.txt: "please crawl this", or "never crawl this".

Answer 2

Have you looked at limiting Google's bot threshold?

Per Google,

If Google is making too many requests per second to your site and slowing down your server, you can limit how fast Google crawls your site.

The one caveat is you cannot limit to a specific folder/directory. Per Google:

You cannot change the crawl rate for sites that are not at the root 
level—for example, www.example.com/folder.

and

The new crawl rate will be valid for 90 days.

Even though the limit is only for 90 days, I would imagine that after 90 days, your content wouldn't be perceived as new and needing to be crawled as much by Google.