New answers tagged

0

It is not a good idea to let Googlebot crawl URLs that can be constructed with combinations of parameters. When you allow your categories to be filtered, my advice it only allow Googlebot to crawl pages with a single filter applied, even if you allow users to view multiple filters at the same time. To implement this, you need to change your URLs. /...


1

Have you looked at limiting Google's bot threshold? Per Google, If Google is making too many requests per second to your site and slowing down your server, you can limit how fast Google crawls your site. The one caveat is you cannot limit to a specific folder/directory. Per Google: You cannot change the crawl rate for sites that are not at the ...


Top 50 recent answers are included