I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not much of content). Therefore, I want to optimize the way in which google uses its bandwidth for my website. Apart from few unnecessary resources that I can block, I want to limit the bots focus to crawling/recrawling high-value pages only. After discussing a lot I have 3 options
- 404 the low value pages. Not an option for me.
- Add no-index to the low-value pages. This should(although not confirmed) reduce the frequency with which those pages are requested for while crawling.
- Block the URLs via robots.txt. There is (no particular pattern + I have to block 150000+ URLs) to the low-value pages because of which I cannot use wildcards in the robots.txt. So, robots.txt is almost out of the picture.
Looking at these options 2nd one is the one most feasible. But my concern is that as per Google documentation crawling and indexing are independent.
- Robots.txt should be used to limit crawling.
- no-index should be used to prevent indexing.
So, perhaps adding no-index would not help my case. Any suggestions or alternatives?
robots.txtstill stands. But even if you did use
robots.txtand spell out each URL individually, you are likely to exceed Google's 500KB file size limit with 150,000+ URLs.