1

I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same. Crawl Request Analysis

Duplicate requests are when more than 1 request was made by google-bot for the same page within a week.
I am concerned about the amount of duplicate requests that the bot is making. This results in a significant waste of its bandwidth for crawling my website. On analysing the pages being requested by bot, I could see that a lot of requests were made for pages with low value(less content, lower SERP). Such pages are like list results for certain combination of filters in which majority of the combinations do not offer much value.
So, I am pondering whether it is wise to block the google-bot from accessing such low-value pages and concentrate on high value sitemap pages.
I had already removed such pages from my sitemap.

Improve this question
2
  • Google fetches pages with higher Pagerank more often. Pages like your home page are more likely to see repeat fetches. If you do have low-value pages that you don't want indexed, by all means you should block them. However, bandwidth for Googlebot is generally well worth any expense in terms of the amount of traffic that Google ends up sending.
    – Stephen Ostermiller ♦
    Commented Dec 6, 2016 at 16:28
  • Yes, even I confirmed by random sampling that the google does crawl pages with the higher value(PageRank) with more frequency. I do not have a problem with indexing those pages. It is just that the kind of bandwidth expended on them just outweighs the benefits of indexing them.
    – HBalyan
    Commented Dec 6, 2016 at 18:18

1 Answer 1

Reset to default
1

As I understand you have many pages that are less important that shouldn't be crawled (that often).

I don't know if it will help removing the pages from the sitemap. The sitemap is not a selection of pages that will be crawled. When a search engine finds links outside the sitemap, they will just as well be crawled.

Did you consider assigning a lower priority for these pages in your sitemap? This source from Google Webmasters Blog indicates that the 'priority' attribute helps in priorizising pages for search engines. Some articles about SEO claim that the pages with lower 'priority' will get crawled less often.

Improve this answer
3
  • Google has said that they now ignore the priority and change frequency fields in sitemap files.
    – Stephen Ostermiller ♦
    Commented Dec 6, 2016 at 16:29
  • Yes, I have seen Google representatives say that apart from the last mod attributes others are not used much by them because websites usually do not manage them properly. seroundtable.com/…
    – HBalyan
    Commented Dec 6, 2016 at 18:25
  • Thank you for your corrections. I guess I will leave my answer since the fact that it's not really an answer is also an answer. At this point I'm unfortunately out of ideas for the question asker.
    – Lucas Johnston
    Commented Dec 6, 2016 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.