4

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot protection (article). I do not have the resources to spend much time or money for a self-built solution (e.g. custom made honeypot), so I discovered the Apache Ultimate Bad Bot Blocker. So my questions is whether it is actually useful/helpful and not too performance impacting (adding 8000 lines in htaccess or virtual host file and 5000 lines in robot.txt)?

My skepticism both comes from the fact that this is obviously public information, so any "bad bot" writer would of course not use any of the User Agents or Referrers present in that list ?? I am not claiming that the information is wrong, but I assume this is collected ex-post, i.e. through honeypots or other means, but any "bad bot" would just use these parameters for a few days, and then move on with other parameters, which initially are not part of that list?

This recent (2018) article also explains why trying to prevent bad bots is a waste of time and resources.

Any real-world advice on what to do?

  • 2
    Unless you are running a site that has hundreds of thousands of users (or more) interacting with it every day trying to prevent things like "bad bots" is going to waste your limited resources. I've only been running small web sites for 15 years and not once have any of them run into problems with bots. IMO I think that "bot blocking" hype is designed to help sell solutions looking for a problem and/or consulting services. Again, IMO, very similar to the SEO hype that has now pretty much come and gone. – Dave Nov 6 '18 at 12:27
  • @Dave Thanks for your "experience-based" feedback. My question was triggered by the fact that I added our teams' email addresses on the website. They are protected by recaptcha v3, but as I read about smart bots being capable to validate the recaptcha, I was just thinking about an additional layer of protection against bad bots / web crawlers (giving in to my OCD tendencies ;). But per my question, I was wondering whether it really was (i) worth the hassle (time and effort to set it up and maintain it) and (ii) the potential performance degradation. – Peter K. Nov 6 '18 at 12:53
  • 1
    The only question is the "why". Why do you want/need to do so? Are you running out of ressources? Are your sites going down by bot attack? Not? Let it be and don't spent your time for such bullshi*t – Evgeniy Nov 6 '18 at 14:50
  • @Evgeniy Main objective is to avoid that bots/crawlers that would manage to evade reCaptcha v3, would be able to harvest email addresses from the website. – Peter K. Nov 6 '18 at 15:29
  • I believe that using such an Apache module comes with a hit to performance. I run a large site that has throuh periods been scrapped to pieces as there a lot of people who want the data for their own use or to reproduce on their own sites. So I have spent considerable time blocking bots. In terms of protecting email addresses, injecting them into the page with JS and obsfucated should keep them safe. For contact forms I inject the form with JS also. As well as using the clean talk block list (availale from IPHol). – Kohjah Breese Dec 28 '18 at 22:40
2

To answer your question, no, I don't think adding that many lines in htaccess etc. is a good solution. As you suspected you only block already detected bots.

Its more efficient to use a list like spamcop https://www.spamcop.net/fom-serve/cache/291.html and Apache Spamassasin https://spamassassin.apache.org/ directly on mailserver level to protect you from incoming spam.

In the end the brain is your best protection, in combination with the tools available. Unfortunately you need to spend resources to stay safe long-term.

1. Limit the amount of emails displayed on site(s).

If possible change to a submit form where visitor enters his or hers email and you contact them rather then showing an email in clear-text. If not possible, then make sure to have only 1 email displayed on the website, and then monitor that inbox like a real webmaster should. Anytime you receive spam or infected emails make sure to add the specific unique thing in that email to a spam filter so that any future email with same characteristics gets completely ignored in the future. This takes some practice but works great. You will be amazed how predictable most spam is. In time you get less and less "bad" emails. Be careful not to block legitimate emails creating too sensible filters though.

2. Educate the email users.

You can never stop "bad bots" completely (they are not bad at all, they are great, but the people that use them are bad) and you can spend endless resources trying to do it. It is on the other hand easier to educate the users of the emails connected to your servers. Make sure they understand to never open an attachment or click a file or even download one unless they have previously confirmed that they know the sender and are expecting such file, and even then be suspicious. If an email is unexpected, don't open it, but if possible forward it to you as webmaster etc. Educating people about how to use email safely is easier then stopping hackers from sending infected emails.

3. Don't worry, but be serious about security.

Don't go around paranoid worrying about bad bots and such. It's nothing to worry about unless you have something to steal. If you do have something valuable, then learn how to protect it. Make backups. Store backups on external servers in other data centers. Be serious about using latest version of scripts and keep an eye on security patches etc etc. Stay up to date!

  • 1
    Thank you for the answer, so I will not further worry ;) For my main website, the email addresses are only displayed after validation through reCaptcha v3, and we use Office365 - Exchange for email and I think the anti-spam is sufficient (not perfect, but acceptable). Regarding content, there is not much to steal, so I do not really worry about that. – Peter K. Jan 13 at 16:40
  • Sounds like a good setup :) Glad to be of help! – Don King Jan 14 at 21:25

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.