Google Webmaster Tools Crawl Stats

I saw a drop in traffic of my website in early September 2014.

But, if I check the "Crawl Stats" in Google Webmaster Tools, I see an increase in "Pages Crawled per day" and "Kilobytes downloaded per day".

This is just reverse to traffic drop. Should I be concerned?

I am also seeing daily 20-30 new 500 errors in Google Webmaster Tools despite the fast loading of the website for users.

  • Did you add a lot of new content to your website? – Wexford Sep 18 '14 at 11:10
  • No, I didn't add new content. Instead, I stopped adding new content for around 4 days. Generally, my frequency of adding new content is 2-3 articles a day. – Prince Jindal Sep 18 '14 at 11:16
  • fastloading isn't the only error GWT gives. 500 is more than needed, might want to fix (some of) those. – Martijn Sep 18 '14 at 13:15
  • I daily fix all the 500 errors, but couple of errors appear daily, when I check my account. I am – Prince Jindal Sep 18 '14 at 14:19
  • 1
    There is no correlation between a drop in traffic and Google crawling your site and incidentally site response times. It is just a coincidence. – closetnoc Sep 18 '14 at 14:27

This happens when Google is re-fetching your website pages and trying to do in-dept analysis of your website. If you are not uploading the new content on your website then the problem is related to the code {PHP /HTML etc} of your website. Ask your developer to fix all the errors that Google has rectified. Google webmaster tool is now upgraded to advanced features where they check all your website code to see whether your website is complying with the slandered guidelines of website coding {W3C Validation}. If it is not then they will show you error pages in the webmaster tool. You have to resolve these errors as early as possible to get your website ranking back.

  • Thanks for your insights Prashant. I am now opting for a complete website optimization so that all the issues gets resolved with the improved page speed. I hope that will surely help. Also, I checked the sitemap and that was also showing some errors in Google Webmaster Tools. I have corrected that. My sitemap was also getting generated in HTML format (besides the usual XML) and that was creating the problem too. I am hoping to see the better results soon. – Prince Jindal Sep 23 '14 at 20:04

The third graph (green - Time spent downloading a page) is worrying to me. It appears that around the date that you lost traffic, your website went from serving the page in a third of a second to almost 4 seconds. If just the page request took that long, than the all the images, JavaScript, and CSS could have easily pushed the page load time above 10 seconds.

When your website takes more than 7 seconds to render a page, Google starts penalizing your site in the search results. I suspect that your traffic drop is related to the performance problems with your website starting around this time.

  • Yep. Thanks Stephen for adding your thoughts in this thread. Better performance is what I am trying to achieve now. – Prince Jindal Sep 23 '14 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.