Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I launched my website in Jan '11 and it is indexed in Google which is great - around 300 pages of content so far. I'm trying to understand what happened the Crawl Stats around the end of June and start of July. What has caused the massive climb? What does it mean going forward? Is there anything else I should be doing?


enter image description here

share|improve this question

migrated from stackoverflow.com Aug 10 '11 at 13:24

This question came from our site for professional and enthusiast programmers.

2 Answers 2

up vote 2 down vote accepted

If you feel that they are over-crawling your site (perhaps even missing deeper content) you should ensure that your HTTP headers are returning good values for things like 'lastmodified' time etc. It may be that Googlebot overestimates how much your site is changing. As a bonus your site will behave better with regards to caching (be it proxy or browser based) and thus feel a little faster.

You'd do well to study what URLs are being crawled (by reviewing your server logs). If they are recrawling the same URL again and again, you definitely have a problem. A common variant on this is if you have a page that can be displayed in many different ways using request variables. Googlbot may try to crawl every possible combination of those variables.

An example I've encountered as a crawl operator was a page that had a list of twenty headings, any combination of which could be expanded. Basically, that page had 2^20 different URLs!

Make sure that Googlebot isn't stuck crawling basically the same page over and over again with trivially different parameters (I've seen it get caught up in this)

share|improve this answer
    
Thanks for this, I noticed today there was another 'spike' in activity so am becoming more concerned as Google's Crawl stats crawled 1,000 pages of content - I only have 300 pages! How do I check to see what pages Google is re-crawling again and again? And how do I check that my HTTP headers are returning good values - it there a test App? –  ubique Aug 18 '11 at 11:47
    
Your server logs should tell you what pages Google is crawling. As for the HTTP headers, there are a number of Firefox plugins. Personally I use Firebug. –  Kris Aug 18 '11 at 12:17

I guess Google changes crawl rate with site age, popularity (links to your site), markup and header, proper site maps, etc. They also changed their crawler a while ago so content now can appear in search results much faster than it used to (at least 2 weeks before the change).

So when I released my blog 2 years ago it took Google months to index all the content and weeks to index new posts. Now I see the any post in search results the same day I publish it.

That simple, google doesn't like new sites but respects the elders.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.