1

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something have prompted all of these bots to scan my sites?

These sites have been up for a long time, and only one has had any recent changes. Even those changes were minor.

2
  • Those are all search engines, and they scan as much of the Internet as they can as frequently as they can.
    – ceejayoz
    Oct 20, 2014 at 19:03
  • @ceejayoz - Right, and we've been scanned by all of them multiple times over the years. What's odd is that all of them scanned us this morning. Usually only one or two (at most) are scanning us at a time. It was a strange enough coincidence for me to wonder if something triggered it.
    – Fatbird3
    Oct 20, 2014 at 20:49

1 Answer 1

1

Usually all it takes is a single link from one site to another. Once your site is discoverable via links, it will eventually be crawled as the spiders move throughout the web.

As for why you are just now seeing it, no idea. There could be a number of factors in play with this, including a simple frequency increase as the search engines decide that you have stuff that should be indexed more often.

If you are checking the IP addresses of the bots to be sure they are real and not malware masquerading as a bot you should be fine and there isn't anything to worry about.

1
  • They look to be behaving normally and coming from the correct addresses. I don't think they're malware. I think it was just an odd coincidence.
    – Fatbird3
    Oct 20, 2014 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.