3

Have had a very strange set of httpd access logs today from before a pagespeed mod and apache crash. It seemed that the following bots appeared to be indexing the site, like crazy, all at once.

  • googlebot
  • googlebot-image
  • bingbot
  • yahoo! slurp
  • yandexBot
  • Baiduspider
  • Sogu Web spider
  • AhrefsBot
  • SeznamBot
  • msnbot

Plus a couple more I can't locate at once.

This looks highly suspicious to me. In amongst the bot requests there are some odd URL requests that are not in the site, so are not being spidered (things like word press admin URLs and it not even a wp site).

The bot IPs look genuine. Anyone know how someone might be able to trigger a bot-storm? Never seen that before - seems very odd indeed, can't believe this is mere coincidence.

  • How much difference between today and yesterday/past hits? Do you continuously check your site logs to see what's going on? Do you think they are wasting more bandwidth in your site? They are accessing some non-existent page because someone may link to you. Most of all above bot crawl webpages from one link to another. You can block some of unwanted bots(Like ahref, seznambot, baidubot if you're website is not chinese, yandex if you'r site is not russian) in your robots.txt – Goyllo Jan 24 '17 at 11:37
  • Thanks for the response. Massive difference between normal, had 18 times the usual network packets during a 4-5 hour period compared to over the last month or so. Usually we get one or two bots at a time. Unfortunately I had the pagespeed mod stats log running and it got itself into a knot and conked out. All seems to have settled down now. – Jon Holland Jan 24 '17 at 12:24
  • Correction it is still going on... – Jon Holland Jan 24 '17 at 13:06
  • 1
    @JonHolland Mail.RU_Bot is also search engine. Look's like some ping tool tell search engine to crawl your site again and again. I can't confirm, but that may be reason. – Goyllo Jan 24 '17 at 13:55
  • 1
    Have you tried a reverse DNS check on the IP's, I have sometimes found that bots come in in waves - all together in my logs and sometimes they are not Google bots (IP's do not check out) for example that you do not want to block. I've also implemented the (perishablepress.com/6g) Firewall checklist in htaccess and am successfully blocking some bots with increased numbers of 403's returned – zod Jan 24 '17 at 22:11
2

There are occasions where a large number of search engines will try to index your site at once. As has been mentioned in the comments this can be from one or more of the large number of ping tools on the web which tell all search engines out there that there is new content to be indexed.

You have done the right thing in verifying the IP's as resolving to Bing and Google as that is the most important thing to ensure that the crawling is coming from a legitimate location.

It isn't necessarily something to worry about, sometimes by pure chance a large number of search engines may try and crawl your site at the same time, especially if someone adds a new link to your site for a page on your site that the search engines don't know about yet.

  • It does indeed seem a coincidental perfect bot-storm. I have added some robots.txt blocks to non-relevant bots (we have other TLD sites for various International traffic). – Jon Holland Jan 26 '17 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.