0

I'm trying to implement some protection against automated attacks on my website (mostly via scripts, probing for SQL injection, etc.) So I'm checking if two consecutive visits from the same IP fall within a certain time range. And if so, I then show a blank page with a short sentence that the frequency of visits is too high.

But now I'm wondering, say, if Google, Bing, Duck-Duck-Go, or other reputable search engines visit a website, do we know time interval between page visits when they crawl a site?

For instance, they won't start loading all pages of the website all at once, or within a short span of time (say, measured in seconds.)


PS. By saying reputable I mean search engines that play by the rules that they clear post on their websites.

2 Answers 2

1

Depends from website to website. If you have a news website with lots of fresh content getting posted hourly then you can expect more visits from search bots. If your pages are just static then you can expect much less visits from search bots.

Showing blank page idea is risky one.

Moreover if you really want to make a guesstimate then head onto your Google Webmaster Tools and check crawl rate. It will give you some idea on crawl rate.

Now coming to the point. If you really want to stop this malicious bots and stuff then maybe code some kind of bot trapper. For example you can put a small 0x0 pixel link in the footer of your website that leads to a script. This script will check the IP of the visitor. Only bots will be going there as humans won't be even seeing it. Make a log in database for all the hits you get there. Now to differentiate good and bad bots you can put Disallow in robots.txt and if still bots are hitting it then they are malicious ones and could be added to blacklist dynamically. The blacklist could be anything a .htaccess will do.

1
  • What I'm asking is not how often the site will be crawled (the Google Console reference) but how many pages at once will it crawl on my site. In other words, say, I have page1.html, page2.html, page3.html. During the crawling session, how soon will page1.html be crawled vs page2.html vs page3.html. I tried to make it clear in my OP, but I guess I didn't ...
    – c00000fd
    Jun 30, 2020 at 18:35
0

Rate-limiting bots is not a good way to protect your site from XSS and SQL injection. If your site is vulnerable, a bot might only need to try a handful of tactics to get to it, which it can spread out over days or weeks or multiple IPs if it's being rate limited or IP banned. The correct way to protect against injecton is to (1) use high-quality server software and keep it properly patched, (2) consider using Content Security Policy, and (3) consider using a Web Application Firewall.

As Keral mentioned in the other answer, the frequency of a legitimate crawler varies dramatically based on how popular your website is and how frequently it is updated. For example with a website like Stack Exchange, it seems like Googlebot visits the front page multiple times per second. Some websites can even see 150 requests per second from Googlebot if they are misconfigured. So there's no hard and fast rule for how often a crawler can visit before it's for sure "malicious".

That being said, if a single bot is hitting your site so much that it's causing your server to buckle under the pressure (or just "obviously too much", like 10 times a second), you can use the "blank page" tactic as you mentioned, but make sure that it returns the 429 Too Many Requests status code so that your intent is machine-readable as well. This tells the bot to chill and stop sending so many requests so quickly. If the bot ignores this signal egregiously, then it's fair to ban it but there's probably not much reason to bother unless the 429s themselves are causing too much server load.

3
  • Well, IDK, almost 47k hits just for yesterday from this one person. Is that enough? He was using several different IPs. All VPNs, except for a few in between. Those were from Russia. (Probably forgot to turn on his VPN.) I wouldn't call it a crawler though. Just some automated script. Basically re-iterating different versions of SQL Injection parameters for the URLs.
    – c00000fd
    Jun 30, 2020 at 20:33
  • @c00000fd I'd recommend posting a new question with this info, since "trying to implement some protection against injection" is quite a different story than "help, my website is currently being hit 47k times per day by a determined attacker who keeps hopping IPs". When you post that question, it would also be helpful to post what type of server software you're running. Jun 30, 2020 at 20:49
  • At the end of the day, though, either your website is vulnerable or it isn't. An attacker can't afford to hit your site 47k times per day, day after day without any payoff, unless they are targeting you specifically for some reason. After you make sure your server is fully patched, you might consider using something like CloudFlare to help filter requests until they give up and go away. Jun 30, 2020 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.