What is the time interval that reputable search engines can hit pages of your website during a crawl?

Question

I'm trying to implement some protection against automated attacks on my website (mostly via scripts, probing for SQL injection, etc.) So I'm checking if two consecutive visits from the same IP fall within a certain time range. And if so, I then show a blank page with a short sentence that the frequency of visits is too high.

But now I'm wondering, say, if Google, Bing, Duck-Duck-Go, or other reputable search engines visit a website, do we know time interval between page visits when they crawl a site?

For instance, they won't start loading all pages of the website all at once, or within a short span of time (say, measured in seconds.)

---

PS. By saying reputable I mean search engines that play by the rules that they clear post on their websites.

Answer 1

Depends from website to website. If you have a news website with lots of fresh content getting posted hourly then you can expect more visits from search bots. If your pages are just static then you can expect much less visits from search bots.

Showing blank page idea is risky one.

Moreover if you really want to make a guesstimate then head onto your Google Webmaster Tools and check crawl rate. It will give you some idea on crawl rate.

Now coming to the point. If you really want to stop this malicious bots and stuff then maybe code some kind of bot trapper. For example you can put a small 0x0 pixel link in the footer of your website that leads to a script. This script will check the IP of the visitor. Only bots will be going there as humans won't be even seeing it. Make a log in database for all the hits you get there. Now to differentiate good and bad bots you can put Disallow in robots.txt and if still bots are hitting it then they are malicious ones and could be added to blacklist dynamically. The blacklist could be anything a .htaccess will do.

Answer 2

Rate-limiting bots is not a good way to protect your site from XSS and SQL injection. If your site is vulnerable, a bot might only need to try a handful of tactics to get to it, which it can spread out over days or weeks or multiple IPs if it's being rate limited or IP banned. The correct way to protect against injecton is to (1) use high-quality server software and keep it properly patched, (2) consider using Content Security Policy, and (3) consider using a Web Application Firewall.

As Keral mentioned in the other answer, the frequency of a legitimate crawler varies dramatically based on how popular your website is and how frequently it is updated. For example with a website like Stack Exchange, it seems like Googlebot visits the front page multiple times per second. Some websites can even see 150 requests per second from Googlebot if they are misconfigured. So there's no hard and fast rule for how often a crawler can visit before it's for sure "malicious".

That being said, if a single bot is hitting your site so much that it's causing your server to buckle under the pressure (or just "obviously too much", like 10 times a second), you can use the "blank page" tactic as you mentioned, but make sure that it returns the 429 Too Many Requests status code so that your intent is machine-readable as well. This tells the bot to chill and stop sending so many requests so quickly. If the bot ignores this signal egregiously, then it's fair to ban it but there's probably not much reason to bother unless the 429s themselves are causing too much server load.