I launched my website in Jan '11 and it is indexed in Google which is great - around 300 pages of content so far. I'm trying to understand what happened the Crawl Stats around the end of June and start of July. What has caused the massive climb? What does it mean going forward? Is there anything else I should be doing?

enter image description here


2 Answers 2


If you feel that they are over-crawling your site (perhaps even missing deeper content) you should ensure that your HTTP headers are returning good values for things like 'lastmodified' time etc. It may be that Googlebot overestimates how much your site is changing. As a bonus your site will behave better with regards to caching (be it proxy or browser based) and thus feel a little faster.

You'd do well to study what URLs are being crawled (by reviewing your server logs). If they are recrawling the same URL again and again, you definitely have a problem. A common variant on this is if you have a page that can be displayed in many different ways using request variables. Googlbot may try to crawl every possible combination of those variables.

An example I've encountered as a crawl operator was a page that had a list of twenty headings, any combination of which could be expanded. Basically, that page had 2^20 different URLs!

Make sure that Googlebot isn't stuck crawling basically the same page over and over again with trivially different parameters (I've seen it get caught up in this)

  • Thanks for this, I noticed today there was another 'spike' in activity so am becoming more concerned as Google's Crawl stats crawled 1,000 pages of content - I only have 300 pages! How do I check to see what pages Google is re-crawling again and again? And how do I check that my HTTP headers are returning good values - it there a test App?
    – Ubique
    Aug 18, 2011 at 11:47
  • Your server logs should tell you what pages Google is crawling. As for the HTTP headers, there are a number of Firefox plugins. Personally I use Firebug.
    – Kris
    Aug 18, 2011 at 12:17

I guess Google changes crawl rate with site age, popularity (links to your site), markup and header, proper site maps, etc. They also changed their crawler a while ago so content now can appear in search results much faster than it used to (at least 2 weeks before the change).

So when I released my blog 2 years ago it took Google months to index all the content and weeks to index new posts. Now I see the any post in search results the same day I publish it.

That simple, google doesn't like new sites but respects the elders.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.