Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
In your robots.txt add
You can block by IP address using the
You can use the following directive in
However, crawlers may decide to ignore the content of your robots.txt. Moreover, the file can be cached by search engines and it takes time before changes are reflected.
The most effective approach is to use your server capabilities. Add the following rule to your
Remember to restart or reload Nginx in order to apply the changes.
Wordpress solution (not the best but helps)
Same problem with Biadu spider, that aggressive that my box ranked over 35 in my console using top. Obviously that even a fast computer cannot handle effectively outside requests running at 35....
I traced the number of IP's (from that University building ????) to be several hundreds, with mainly two useragents)
Direct consequence ? As I have a cloud server I had to upgrade the same to higher memory in order to allow a decend response.
Previous answer :
User-agent: Baiduspider Disallow: /
Baidu seems totally unable to respect the robot.txt indication.
What I did :
I installed WP-Ban plugin for Wordpress (free) and banned the following :
USER AGENTS :
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Furthermore using Wp Super Cache I re-address the relative error page to a static page, thus the whole wordpress installation does not / or at least only for the banned useragents check the Mysql datatable.
(This is standard Wordpress blablabla, so everybody being able to install a Wordpress Plugin can do it, as no coding or ftp access is required for this procedure)
I agree with everyone : Internet is free, banning whoever or whatever is absolutely the last thing anyone should do, but Baidoo today costs me USD 40 more/month , just to spider a webside written in Portughese, and I have some doubts if there are lots of Chinese people and vistors able to read and understand this language.