Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
In your robots.txt add
You can use the following directive in
However, crawlers may decide to ignore the content of your robots.txt. Moreover, the file can be cached by search engines and it takes time before changes are reflected.
The most effective approach is to use your server capabilities. Add the following rule to your
Remember to restart or reload Nginx in order to apply the changes.
You can block by IP address using the
Wordpress solution (not the best but helps)
Same problem with Biadu spider, that aggressive that my box ranked over 35 in my console using top. Obviously that even a fast computer cannot handle effectively outside requests running at 35....
I traced the number of IP's (from that University building ????) to be several hundreds, with mainly two useragents)
Direct consequence ? As I have a cloud server I had to upgrade the same to higher memory in order to allow a decend response.
Previous answer :
Baidu seems totally unable to respect the robot.txt indication.
What I did:
I installed WP-Ban plugin for Wordpress (free) and banned the following :
USER AGENTS :
Furthermore using Wp Super Cache I re-address the relative error page to a static page, thus the whole wordpress installation does not / or at least only for the banned useragents check the Mysql datatable.
(This is standard Wordpress blablabla, so everybody being able to install a Wordpress Plugin can do it, as no coding or ftp access is required for this procedure)
I agree with everyone : Internet is free, banning whoever or whatever is absolutely the last thing anyone should do, but Baidoo today costs me USD 40 more/month , just to spider a webside written in Portughese, and I have some doubts if there are lots of Chinese people and visitors able to read and understand this language.
I have just successfully blocked the Chinese searchbot
Reasons for deciding to block
So I hope Su' and others concerned about Xenophobia will understand this decision is a cool-headed response to a unpolite number of requests.
Baiduspider accesses my server using many different IP addresses but these addresses do fall inside certain ranges. So my .htaccess file now contains the following lines:
The bottom line basically describes 4 IP ranges in which I know Baiduspider and ONLY Baiduspider accesses my server. Each of the 4 ranges is 256 consecutive addresses (total 1024). Please note, the syntax for the IP ranges on the
The internet should be free, open and fair. But that means organisations like Baidu learning to obey Robots.txt and being less greedy with the regularity of it's crawls. My solution involves tinkering with very powerful settings so before you mess around with the .htaccess file be sure to back up your original, ready to rollback if you take down your server in a blaze of glory. Proceed at your own risk.
Use .htaccess with
The "RewriteEngine On" allows you that the following lines are parsed correctly. The HTTP_USER_AGENT is the line where the spiders are identifying themselves. The condition is true if the line contains "MJ12bot" or "Baidu". NC means "not case-sensitive" and you can chain conditions with OR. The last line must not contain "OR" or the rule does not work.
Baidu is particularly nasty because it tries to read Wordpress entries ("fckeditor","wp-content") for which is has absolutely no reason. MJ12bot is also one of the bad critters.
The Rewrite rule means block the spider with a 403 Forbidden ([F]) to access all files (.* is a regular expression for any file) and stop further evaluation ([L]) of htaccess.