Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

It seems that in Google Webmaster Tools I can only see about a month and a half of crawler history for my website.

My site has been verified for several months now. Analytics has a broader history than this. Is there something I'm doing wrong is it that just how Webmaster Tools is?

share|improve this question

migrated from webapps.stackexchange.com Nov 17 '11 at 20:49

This question came from our site for power users of web applications.

    
Why do you need more crawl history? The only important thimg is how Googlebot is crawling your site NOW. –  DisgruntledGoat Nov 18 '11 at 0:49
    
@DisgruntledGoat if you make multiple changes to your site over time - especially with regards to file structures and so forth - sometimes it would be helpful to compare the changes you have made with changes in the crawl rate to see if there any obvious relationships –  joshuahedlund Nov 18 '11 at 17:18

2 Answers 2

up vote 3 down vote accepted

The crawler report provided by Google Webmaster Tools does not provide data points beyond the past 90 days and it would be fairly difficult to maintain accurate records given the crawl rate data that is made available.

The data is actually embedded in the chart image URL's chd parameter using the simple encoding scheme described in the Data Format document for the Google Charts API.

It is possible to extract the data (included JS functions culled from Google's documentation) but the data is normalized and, without the context of the high and low crawl rate figures, it wouldn't be particularly useful for tracking crawl rate over time.

/**
 * number encoding map
 */
var SIMPLE_MAP='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
/**
 * simpleDecoding function from
 * URL: http://code.google.com/apis/chart/image/docs/data_formats.html
 */
function simpleDecoding(val) {
    if (val.length != 1 || val == "_") {
        return "";
    }
    var decodedVal = SIMPLE_MAP.indexOf(val.charAt(0));
    if (decodedVal < 0) {
        alert("Invalid decoding character.");
        return "";
    }
    return decodedVal;
}
/**
 * ... and decode a whole string
 */
function simpleDecode(val) {
    var output = '';
    for ( i = 0; i < val.length; i++ ) {
        output += " "+simpleDecoding(val.charAt(i));
    }
    return output;
}
share|improve this answer
    
Clever. Like you said, not useful without some very heavy finagling, although if I absolutely had to have this data I can think of ways to finagle it. Hopefully they will just add a download link. –  joshuahedlund Nov 18 '11 at 17:15
    
Apparently Google considers crawl rate to be proprietary information - only reason I can imagine that they'd display a normalized version but withhold the source data. –  danlefree Nov 18 '11 at 17:18

Unfortunately this is just how Webmaster Tools is. The only way I know to preserve historical crawl stats information is to take screenshots every now and then and compare your images.

share|improve this answer
1  
"The only way I know to preserve historical crawl stats information is to take screenshots every now and then and compare your images." - the "Download this table" links at the bottom of report pages are a better option for comparing reports over time. –  danlefree Nov 18 '11 at 2:40
    
This is a very good point. However some of the pages, including the Crawl Stats pages, do not at this time have a "Download this table" link. –  joshuahedlund Nov 18 '11 at 13:48
    
Point taken - curious that they don't just provide a download link. –  danlefree Nov 18 '11 at 15:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.