We noticed huge difference in the metrics. For example, last saturday :

  • According to Webmaster Tools :

    • 6,082 pages crawled
    • 76,119 kilobytes downloaded
    • 1,550 ms in average to download a page
  • Now according to our apache logs, grepping "Googlebot" :

    • 1,444 pages crawled (2,997 including all our subdomains : blog, images, tracking, ...)
    • 201 ms on average (we use apache logs' "%D" to monitor this, which is the time to last byte)

We are not on the same timezone so the results may vary, but the trend is the same everyday : 2 to 3 times more pages crawled on Webmaster Tools, and 5x to 10x more time to download the pages. And it's similar on other domains, too.

I can't find any explaination about this. Are there other useragents we need to monitor ? What else ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.