2

Below is the graph for my website which has seen a significant decrease in the crawl rate(pages crawled per day) Crawl Per Day

Here is the graph for the time to download the page Download time

I wanted to know the reason for this continuous decrease in crawl rate since September. There were around 30 server errors(HTTP Code 500) which were fixed by September end. 404 errors were around 350 and were all valid 404s. The website has around 0.2 million pages out which we selectively submit around 20K URLs via sitemap. I checked that server connectivity time was 100% since August. I checked for deployed changes to the website but none is pointing to this issue. I did do a URL restructuring in August start for which I ensured proper 301s and canonicals.(No crawl errors were reported for URL restructuring)

Any advise in which direction should I look is more than welcome. I am also experiencing a fall in the organic traffic.

Improve this question
5
  • This is perfectly normal behavior. Google will index a site how it sees fit. Very large sites will see larger cycles of indexing with a consistent amount per day.
    – closetnoc
    Commented Nov 21, 2016 at 17:23
  • You have 200,000 pages with Google crawling 5,541 pages per day on average. At that rate Google could recrawl all the pages on your site every 36 days. Since Google crawls popular pages more often, your least popular pages may not get crawled nearly that often, but I'd still expect few months. That crawl rate sounds good enough for Google to index your site well and pick up on any changes you make.
    – Stephen Ostermiller ♦
    Commented Nov 21, 2016 at 18:16
  • @StephenOstermiller Yes, google says the average is 5541 but I am more concerned about the slump in the last month where it has gone as low as 659. I am also facing a gradual but continuous decline in the organic traffic since Sep 15. I do not know much about it but google released Penguin 4.0 update on 23rd Sep. Can it be related to that?
    – HBalyan
    Commented Nov 21, 2016 at 18:55
  • @closetnoc Though it appears to be a normal bot behaviour what can be some places which I can access to ensure things are alright?
    – HBalyan
    Commented Nov 21, 2016 at 18:57
  • Nothing on the indexing side. However, I would be working on why organic search is dropping. If you find that your site has a SERP penalty where page are being de-listed, then I would focus on that. Again, there are no clear clues here. Google is far from open these days!
    – closetnoc
    Commented Nov 21, 2016 at 19:05

1 Answer 1

Reset to default
2

Google uses an algorithm to decide how frequently to recrawl pages. While the exact mechanics are kept confidential the basics are that it is based on how large the site is, how high quality the site is, the number of other sites that link back to it, and how frequently the content on the pages changes. The figures you have posted seem quite reasonable crawl rates for a site your size.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.