3

Have had a very strange set of httpd access logs today from before a pagespeed mod and apache crash. It seemed that the following bots appeared to be indexing the site, like crazy, all at once.

  • googlebot
  • googlebot-image
  • bingbot
  • yahoo! slurp
  • yandexBot
  • Baiduspider
  • Sogu Web spider
  • AhrefsBot
  • SeznamBot
  • msnbot

Plus a couple more I can't locate at once.

This looks highly suspicious to me. In amongst the bot requests there are some odd URL requests that are not in the site, so are not being spidered (things like word press admin URLs and it not even a wp site).

The bot IPs look genuine. Anyone know how someone might be able to trigger a bot-storm? Never seen that before - seems very odd indeed, can't believe this is mere coincidence.

Improve this question
10
  • How much difference between today and yesterday/past hits? Do you continuously check your site logs to see what's going on? Do you think they are wasting more bandwidth in your site? They are accessing some non-existent page because someone may link to you. Most of all above bot crawl webpages from one link to another. You can block some of unwanted bots(Like ahref, seznambot, baidubot if you're website is not chinese, yandex if you'r site is not russian) in your robots.txt
    – Goyllo
    Jan 24, 2017 at 11:37
  • Thanks for the response. Massive difference between normal, had 18 times the usual network packets during a 4-5 hour period compared to over the last month or so. Usually we get one or two bots at a time. Unfortunately I had the pagespeed mod stats log running and it got itself into a knot and conked out. All seems to have settled down now.
    – Jon Holland
    Jan 24, 2017 at 12:24
  • Correction it is still going on...
    – Jon Holland
    Jan 24, 2017 at 13:06
  • 1
    @JonHolland Mail.RU_Bot is also search engine. Look's like some ping tool tell search engine to crawl your site again and again. I can't confirm, but that may be reason.
    – Goyllo
    Jan 24, 2017 at 13:55
  • 1
    Have you tried a reverse DNS check on the IP's, I have sometimes found that bots come in in waves - all together in my logs and sometimes they are not Google bots (IP's do not check out) for example that you do not want to block. I've also implemented the (perishablepress.com/6g) Firewall checklist in htaccess and am successfully blocking some bots with increased numbers of 403's returned
    – zod
    Jan 24, 2017 at 22:11

1 Answer 1

Reset to default
1

There are occasions where a large number of search engines will try to index your site at once. As has been mentioned in the comments this can be from one or more of the large number of ping tools on the web which tell all search engines out there that there is new content to be indexed.

You have done the right thing in verifying the IP's as resolving to Bing and Google as that is the most important thing to ensure that the crawling is coming from a legitimate location.

It isn't necessarily something to worry about, sometimes by pure chance a large number of search engines may try and crawl your site at the same time, especially if someone adds a new link to your site for a page on your site that the search engines don't know about yet.

Improve this answer
1
  • It does indeed seem a coincidental perfect bot-storm. I have added some robots.txt blocks to non-relevant bots (we have other TLD sites for various International traffic).
    – Jon Holland
    Jan 26, 2017 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.