2

I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not much of content). Crawl Request Analysis Therefore, I want to optimize the way in which google uses its bandwidth for my website. Apart from few unnecessary resources that I can block, I want to limit the bots focus to crawling/recrawling high-value pages only. After discussing a lot I have 3 options

  1. 404 the low value pages. Not an option for me.
  2. Add no-index to the low-value pages. This should(although not confirmed) reduce the frequency with which those pages are requested for while crawling.
  3. Block the URLs via robots.txt. There is (no particular pattern + I have to block 150000+ URLs) to the low-value pages because of which I cannot use wildcards in the robots.txt. So, robots.txt is almost out of the picture.

Looking at these options 2nd one is the one most feasible. But my concern is that as per Google documentation crawling and indexing are independent.

  • Robots.txt should be used to limit crawling.
  • no-index should be used to prevent indexing.

So, perhaps adding no-index would not help my case. Any suggestions or alternatives?

| improve this question | |
  • 1
    Does it cost you resources you can't spare? If not, just leave it be. – Martijn Dec 9 '16 at 8:08
  • @Martijn I am talking about thousands of such request. I have added a graph showing the total and the duplicate request. Some weeks saw up to 45K duplicate requests. That is a criminal waste. It can be much better utilised with recrawling high value pages. – HBalyan Dec 9 '16 at 9:08
  • But blocking crawling of these "low value pages" is probably not going to do anything to benefit "recrawling high value pages", as you seem to suggest. It will simply reduce the bandwidth. So, Martijn's comment still stands... unless this crawling of low value pages is actually causing you a "problem" then you aren't really going to benefit from blocking it. – MrWhite Dec 9 '16 at 9:16
  • However, the technical challenge of blocking (in terms of crawling) a large number of URLs, with no discernable pattern, without using robots.txt still stands. But even if you did use robots.txt and spell out each URL individually, you are likely to exceed Google's 500KB file size limit with 150,000+ URLs. – MrWhite Dec 9 '16 at 9:48
  • Cross-post – unor Dec 9 '16 at 22:42
1

Google have too many cralwers based on backlinks, re-crawl same URL after few weeks/months, pagerank, sitemap, Google Webmaster Request etc.

By using noindex Google may crawl that URL less frequently, but it will not going to block it permanently, because noindex pages are crawlable and pass PageRank when it is linked from somewhere, so as per backlinks cralwer and pagerank crawler those pages will going to crawl.

So my first advice is try to links those pages rarely.

Second is remove those pages from sitemap or feed URL of your website.

Third is use Last Modified HTTP header, because when Google crawl some pages, then they will going to recrawl same URL after some time(May be after few weeks to check any changes).

I don't see any other solution for you. If it is possible then move your thin content to subdiretory and block that specific directoy in robots.txt.

| improve this answer | |
  • Thanks @Goyllo...I am not sure how to implement this header as the pages are product listing pages which have dynamic content and that can update anytime unlike a blog post where we can maintain the last modified status. And by the nature of the structure of my website those pages would be less linked to. I had already filtered my sitemap off these low content links. I will probably do a quality check once more. – HBalyan Dec 11 '16 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.