I am experiencing high traffic on my website at certain times of the data, at regular intervals. I am trying to determine whether GoogleBot or BingBot access of my website correlates with the high load we experience on our server.

Is there a log analyzer or any other way by which I can trace googlebot's access of my website - and the time it occurred and visually show that to me, so I can connect the two metrics (googlebot access vs high load on server)?

Details of what happens during the load is described here: https://dba.stackexchange.com/questions/123946/high-data-traffic-from-mysql-server-to-web-server

This is a Magento website with a large number of products.

  • Piwik (piwik.org) is free and one of the very best log file analyzers available. It can also use a bug feature for something more real-time like GA. – closetnoc Dec 23 '15 at 3:23
  • Great - thanks for your recommendation. in your experience, does it do something like what I've described? – siliconpi Dec 23 '15 at 3:25
  • Piwik will tell you as much as any analytics package can. If you are experiencing high traffic, you will be able to see where it is coming from and the user agent string associated with it as well as other information. Please understand that there are plenty of bad bots out there that could be pinging the snot out of your site. As well. search engines can come in a fury sometimes all of them at once. For a while a few months ago, it seemed like all of the search engines would visit my site at the same time day after day for weeks. It happens like that sometimes. – closetnoc Dec 23 '15 at 3:30
  • Yeah - we're experiencing something like that as well - bad-time-bots! Or at least that's what it looks like based on some basic manual analysis... How did you tackle the problem? – siliconpi Dec 23 '15 at 3:33
  • 1
    I am not a big fan of GA. Some people rely on GA only which is a mistake in my book. There is nothing like a good ole fashioned log file analyzer. If someone wants to use GA too, then that is alright of course. I barely look at GA these days but will from time to time. – closetnoc Dec 23 '15 at 4:08

These both free services for log file realtime visualization will help you:

  • Logstalgia is amazing! It plays back a video of the data in the log file - so you can see multiple requests coming in (color coded) - where they are coming from and what are they requesting! – siliconpi Jan 12 '16 at 9:34
  • Did you also experience that Logstalgia shows different results every time you view it? I saw our server being bombarded at a certain time and when I check it again, it doesn't show up again at that time. – siliconpi Jan 12 '16 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.