0

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall.

Is that a good way that won't block legit users/traffic as well?

What are the downsides for such action?

  • 2
    Are you sure it is Yandex and are you sure that they have not obeyed robots.txt?? Yandex has a good reputation for obeying robots.txt. I study these things and have not seen Yandex behave badly in 8 years. If you want to block whole blocks of IP addresses, that is fine. As long as it is not a subscriber block (users) that you want accessing your site. – closetnoc Jun 1 '15 at 17:53
  • 4
    "it can bypass .htaccess rules." - only if you have the wrong rules. – MrWhite Jun 1 '15 at 18:44
  • 3
    The problem is not with Yandex, the problem is with your site. – nathangiesbrecht Jun 1 '15 at 18:55
  • If you had an extra 5 visitors a day that wouldn't be a problem right?... so why would it be with bots? nowadays hosting is cheap as nuts and much faster than it was 10 years ago... optimise your site and focus on your customers/visitors. The only ever time you should really need to block IPs is malicious attacks, or if you prefer you could throttle the bots. – Simon Hayter Aug 1 '15 at 10:24
1

I'm getting too many hits on my website from Yandex.

Take a look at http://en.wikipedia.org/wiki/Yandex. Yandex is a popular russian search engine. Never knock down search engines as they make your site more available to the public which means higher chance of revenue for you if you run ads and/or an online business.

It doesn't obey robots.txt...

As closetnoc states, search engines are good at obeying robots.txt

...and by some way it can bypass .htaccess rules.

I agree with w3d here. Your rules are likely wrong. Perhaps they are being applied to the wrong user agents or IP addresses.

So I'm thinking to block all Yandex IP ranges in my system firewall. Is that a good way that won't block legit users/traffic as well?

This is definitely not a good idea unless you plan on blocking a portion of the world from discovering your website.

What are the downsides for such action?

If you decide to block a fixed set of IP addresses, then unless they are well known and you know for a fact those IP addresses belong to the same individual or business every time, you could be blocking legit users.

Try using other methods to block spam by adding captcha features to any of your web forms on your site or limit the connection rate for the bots.

Also, make excellent use of whois. There's an online version at whois.com and in the top right box, enter any IP addresses in question to see who actually has that IP address you're trying to block. Chances are its one from china.

Also, look into Honeypot. It's like a method where a bad bot can be discovered by making an invisible link that only robots click on, not real users. Then block further access from those IP addresses. Such idea can all be done in PHP and I think in ASP as well.

See: http://en.wikipedia.org/wiki/Honeypot_(computing)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.