4

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and bad traffic coming to the site.

The guy on the helpline recommended adding the following few lines into the .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} (bot|crawl|robot)
RewriteCond %{HTTP_USER_AGENT} !(Bing|Google|msn|MSR|Twitter|Yandex) [NC]
RewriteRule ^/?.*$ "http\:\/\/127\.0\.0\.1" [R,L]

This code will basically block all bots except "Bing|Google|msn|MSR|Twitter|Yandex".

As I started looking into this more, I've also came across to a few informative videos and tutorials on how to use the .htaccess file to block the bots. For example this video here. But in this case, the guy is adding about 30 lines with the bots that he wants to block instead of bots he want to allow.

Should I be adding a list of bots to ALLOW or should I be adding a list of bots to BLOCK instead?

Improve this question
3
  • Or you can use Fail2Ban or mod_security. You will find there are a ton of bad bots and using an agent name will not be effective. Why? Because it is trivial to fake an agent name. Most bad bots will use browser agent names. The two apps above keep the task simple instead of having to keep up with all the IP addresses. They are the two most effective apps out there. Cheers!!
    – closetnoc
    May 16, 2018 at 4:21
  • Hey @closetnoc, thanks .. I'm looking into both suggested apps now, but what are they? Are they plugins? How do I use them on a WP site?
    – Joe Bloggs
    May 16, 2018 at 22:46
  • Both are applications that work at the web server level. Not plugins as WP does. fail2ban.org/wiki/index.php/Main_Page and modsecurity.org Many here swear by Fail2Ban so I would start there. Cheers!!
    – closetnoc
    May 16, 2018 at 23:01

3 Answers 3

Reset to default
2

The types of bots

One way of thinking about bots is to treat them as following the Lawful-Chaotic and Good-Evil spectrum.

(Although bots cannot be evil, they can be instructed to act on behalf of people doing evil.)

An example for Chaotic good is Shodan, which helps improve security by helping research and checking your own systems, even if other people can use it too.

Is blocking bots going to help?

Not really, unless your quota is really low, good bots should hit your site very rarely. If your quota is this low it would be trivial to DoS your site by just crawling it normally.

Bad bots will just spoof good bots, so they will not be blocked by this. User agent limits will only stop Lawful bots (and possibly borderline neutral ones).

Will this cause problems?

Possibly, all sorts of tools use scraping, including web site checkers (some of which block users if they cannot tell if the site is infected or not), content filters (may block by default) and tools like Alexa (depending on what your site provides).

How can I stop bad bots?

You can use a service like cloudflare, but even that will not stop all bots, if you could detect the difference between a human and a bot without inconveniencing the user you could replace reCAPTCHA and have a great service.

Improve this answer
1
  • @JoeBloggs No problem, I'd also keep in mind that anything that makes it hard for computers to parse the page may hurt accessibility (screen-readers in particular)
    – jrtapsell
    Jun 16, 2018 at 12:50
1

Allowing certain bots will cause some troubles for you.

For instance, you will be blocking Alexa, Archive, etc. robots from accessing your website.

I would recommend blocking bad robots by user-agent.

Important note

Bad bots can change their user-agents in request header. So this won't completely safeguard your website against bad crawlers.

Improve this answer
4
  • 1
    I block Alexa and Archive! These are two of the most abusive and useless bots out there. These are blocked before anyone else!
    – closetnoc
    May 16, 2018 at 8:07
  • 1
    Because bad bots can easily spoof browser user agents it is impossible to block bad bots either way using an agent name. The only way to block bad bots is to block by IP address blocks. Since users and bots are not using the same address blocks, this works but requires a lot of expertise and time. One reason why I suggested using installed security apps.
    – closetnoc
    May 16, 2018 at 8:19
  • I had been using Siteground for a while but I didn't face such issue.
    – Shayan Davoodi
    May 16, 2018 at 8:19
  • I will check out that app! For Fail2Ban, they update their system frequently. I wrote my own app since this was a part of my business. I created self defending networks and analyzed abuses including hacking, spam, bad bots, etc. for a long time. I maintained a trust analysis database for access control in highly secure environments.
    – closetnoc
    May 16, 2018 at 8:49
0

Update: following my question and looking more into this, I came across to a plugin called Blackhole for bots.

The good and legit bots should naturaly follow the robot.txt file, the bad bots don't

What this plugin does is that it adds a trap disallow command line to robots.txt file.

If the bot ignores the Disallow command line and follows is regardless, it ends up in a trap 'blackhole' and basically lock it self out of the site.

The plugin also has a whitelist that by default whitelists the known good boots.

So far, this has been working great.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.