2

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and PHP/5.2.81, versions which don't exist. However, most UA strings consist of this raw glob pattern, which indicates what the spambot is trying to pretend to be:

PHP/5.{3|2}.{1|2|3|4|5|6|7|8|9|0}{1|2|3|4|5|6|7|8|9|0}

As such, blocking all user agents starting with PHP/5. using .htaccess would bring a halt to this spambot activity. Is there a significant risk of blocking legitimate crawlers?

  • As a side note, a CMS like Drupal has a project named Links Checker which will check whether a link from the CMS content is broken. If you return an error, it will look like the link is dead and the CMS author is not unlikely to remove it. (which for SEO is bad...) – Alexis Wilke Oct 27 '15 at 3:18
3

Yes you can. In fact, I recommend it and others too.

I would not work so hard on my regular expression to include version numbers- just ^PHP.*$

You will find that some agent names are consistently up to no good even though some will argue that they can be used for good and should not be blocked. That is a negative argument. I study these things and I cannot see in my database where an agent name of PHP.* has been used for anything good at all. So block away my friend! Have at it.

As far as blocking whole countries, that is not always something I recommend, however, as for China, it is not necessarily a bad thing. You would be cutting out legitimate users from Baidu searches and so on, but that may not be a concern. If you do this and it is possible, I suggest blocking larger blocks of IP addresses in a firewall if you have one.

  • I've gone ahead and blocked ^PHP/.*$ user agents as well as blank user agents. Now, server load has plummeted and more than one third of accesses return HTTP 403 Forbidden. Those pesky spambots are gone. Thanks! – bwDraco Dec 6 '14 at 23:35
  • My pleasure. You will see plenty of these kinds of accesses over time, not just using PHP. Mostly, if it is a good bot, it will identify itself. Otherwise, my attitude is- they get a limit chance to prove any value or they are blocked. – closetnoc Dec 7 '14 at 0:39
2

PHP user agents are known as libraries.

It should be no problem if you block them since legitimate crawlers use other string types, however, can't you check and block only the IP's?

In this site, for example, you have ways to block only Chinese traffic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.