We noticed huge difference in the metrics. For example, last saturday :
According to Webmaster Tools :
- 6,082 pages crawled
- 76,119 kilobytes downloaded
- 1,550 ms in average to download a page
Now according to our apache logs, grepping "Googlebot" :
- 1,444 pages crawled (2,997 including all our subdomains : blog, images, tracking, ...)
- 201 ms on average (we use apache logs' "%D" to monitor this, which is the time to last byte)
We are not on the same timezone so the results may vary, but the trend is the same everyday : 2 to 3 times more pages crawled on Webmaster Tools, and 5x to 10x more time to download the pages. And it's similar on other domains, too.
I can't find any explaination about this. Are there other useragents we need to monitor ? What else ?