Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.

share|improve this question
    
What exactly are you trying to accomplish here? (Blocking these spiders is a conclusion; what's the problem that led you to asking how to do it?) Unless they're spidering your site too aggressively and slowing things down, for example, there would seem to be very little reason to do this. –  Su' Jul 9 '12 at 4:28
    
why shouldn't I? I don't want chinese traffic. –  Zhianc Jul 9 '12 at 4:44
4  
I'm not saying you shouldn't, I'm as asking why bother at all. Are they hurting you in some way? Is it not easier to just ignore? "Not wanting Chinese traffic" isn't really an answer here, just borderline xenophobia. And blocking one search engine isn't nearly the same thing as blocking the Chinese, so you're not even really making sense. It seemed to me that you might've been seeking a solution based on a faulty problem. But apparently you do just arbitrarily want to block some spider, so go ahead, I guess. –  Su' Jul 9 '12 at 5:12
    
Sisyphus and the Baidu boulder, it's an interesting endeavor... –  Fiasco Labs Jul 9 '12 at 5:19
add comment

4 Answers

In your robots.txt add

#Baiduspider
User-agent: Baiduspider
Disallow: /

#Yandex
User-agent: Yandex
Disallow: /
share|improve this answer
    
What if they don't respect robots.txt? –  Zhianc Jul 9 '12 at 4:45
    
I happen to know that "Baiduspider/2.0" accesses from Chinese IP addresses do not obey robots.txt on my site. Also, they often use coordinated accesses (some minutes apart) from the same IP address block that do not use the "Baiduspider" user agent ID. –  mgkrebbs Jul 9 '12 at 5:00
    
Yandex obeys robots.txt I had them locked out till we realized a good chunk of our business comes from Russia and had to remove the block. Baidu, you have to check the netblocks. Not all Baidu announced UA are actually Baidu so you need to do a robots.txt block for the legitimate Baidu traffic and then do a UA string block in your .htaccess for what remains and then waste your life doing block by block identification and denial for the rest that gets around the UA string block. It's just as much work as just beefing your server up to take the traffic. –  Fiasco Labs Jul 9 '12 at 5:12
add comment

You can block by IP address using the ngx_http_access_module of nginx. To block a single IP you can add a line to the conf file like

deny 12.34.567.1;

To block a range, use CIDR notation, like 12.34.567.1/24 for the 24-bit subnet block (of 256 IP addresses) which includes the 12.34.567.1 IP address. For more details see, for instance, this page.

share|improve this answer
add comment

You can use the following directive in robots.txt to disallow the crawling of your site.

# robots.txt
User-agent: Baiduspider
Disallow: /

However, crawlers may decide to ignore the content of your robots.txt. Moreover, the file can be cached by search engines and it takes time before changes are reflected.

The most effective approach is to use your server capabilities. Add the following rule to your nginx.conf file to block Baidu at server level.

if ($http_user_agent ~* ^Baiduspider) {
  return 403;
}

Remember to restart or reload Nginx in order to apply the changes.

share|improve this answer
add comment

Wordpress solution (not the best but helps)

Same problem with Biadu spider, that aggressive that my box ranked over 35 in my console using top. Obviously that even a fast computer cannot handle effectively outside requests running at 35....

I traced the number of IP's (from that University building ????) to be several hundreds, with mainly two useragents)

Direct consequence ? As I have a cloud server I had to upgrade the same to higher memory in order to allow a decend response.

Previous answer :

#Baiduspider
User-agent: Baiduspider
Disallow: /

Baidu seems totally unable to respect the robot.txt indication.

What I did:

I installed WP-Ban plugin for Wordpress (free) and banned the following :

USER AGENTS :

  • Baiduspider+(+http://www.baidu.com/search/spider.htm)

  • Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Furthermore using Wp Super Cache I re-address the relative error page to a static page, thus the whole wordpress installation does not / or at least only for the banned useragents check the Mysql datatable.

(This is standard Wordpress blablabla, so everybody being able to install a Wordpress Plugin can do it, as no coding or ftp access is required for this procedure)

I agree with everyone : Internet is free, banning whoever or whatever is absolutely the last thing anyone should do, but Baidoo today costs me USD 40 more/month , just to spider a webside written in Portughese, and I have some doubts if there are lots of Chinese people and visitors able to read and understand this language.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.