2

It seems that in Google Webmaster Tools I can only see about a month and a half of crawler history for my website.

My site has been verified for several months now. Analytics has a broader history than this. Is there something I'm doing wrong is it that just how Webmaster Tools is?

Improve this question
2
  • Why do you need more crawl history? The only important thimg is how Googlebot is crawling your site NOW.
    – DisgruntledGoat
    Nov 18, 2011 at 0:49
  • @DisgruntledGoat if you make multiple changes to your site over time - especially with regards to file structures and so forth - sometimes it would be helpful to compare the changes you have made with changes in the crawl rate to see if there any obvious relationships
    – joshuahedlund
    Nov 18, 2011 at 17:18

2 Answers 2

Reset to default
3

The crawler report provided by Google Webmaster Tools does not provide data points beyond the past 90 days and it would be fairly difficult to maintain accurate records given the crawl rate data that is made available.

The data is actually embedded in the chart image URL's chd parameter using the simple encoding scheme described in the Data Format document for the Google Charts API.

It is possible to extract the data (included JS functions culled from Google's documentation) but the data is normalized and, without the context of the high and low crawl rate figures, it wouldn't be particularly useful for tracking crawl rate over time.

/**
 * number encoding map
 */
var SIMPLE_MAP='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
/**
 * simpleDecoding function from
 * URL: http://code.google.com/apis/chart/image/docs/data_formats.html
 */
function simpleDecoding(val) {
    if (val.length != 1 || val == "_") {
        return "";
    }
    var decodedVal = SIMPLE_MAP.indexOf(val.charAt(0));
    if (decodedVal < 0) {
        alert("Invalid decoding character.");
        return "";
    }
    return decodedVal;
}
/**
 * ... and decode a whole string
 */
function simpleDecode(val) {
    var output = '';
    for ( i = 0; i < val.length; i++ ) {
        output += " "+simpleDecoding(val.charAt(i));
    }
    return output;
}
Improve this answer
2
  • Clever. Like you said, not useful without some very heavy finagling, although if I absolutely had to have this data I can think of ways to finagle it. Hopefully they will just add a download link.
    – joshuahedlund
    Nov 18, 2011 at 17:15
  • Apparently Google considers crawl rate to be proprietary information - only reason I can imagine that they'd display a normalized version but withhold the source data.
    – danlefree
    Nov 18, 2011 at 17:18
3

Unfortunately this is just how Webmaster Tools is. The only way I know to preserve historical crawl stats information is to take screenshots every now and then and compare your images.

Improve this answer
3
  • 1
    "The only way I know to preserve historical crawl stats information is to take screenshots every now and then and compare your images." - the "Download this table" links at the bottom of report pages are a better option for comparing reports over time.
    – danlefree
    Nov 18, 2011 at 2:40
  • This is a very good point. However some of the pages, including the Crawl Stats pages, do not at this time have a "Download this table" link.
    – joshuahedlund
    Nov 18, 2011 at 13:48
  • Point taken - curious that they don't just provide a download link.
    – danlefree
    Nov 18, 2011 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.