3

After blocking all traffic from China and Russia ( using Incapsula ) on an eCommerce application which only ship into USA, I noticed a significant performance improvement. Is there any drawback of country wide crawlers blocking consider SEO? Am I losing anything?

Look at AWS CloudWatch. CPU, network in and network out significantly decreased. Fig 1: Look at AWS cloud watch. CPU usages, web traffic significantly reduced.

Server response time incredibly improved - Newrelic. Fig 2: Server response time incredibly improved - Newrelic.

App index jump up - Newrelic.

Fig 3: App index jumped up - Newrelic.

Improve this question
0

1 Answer 1

Reset to default
3

Unless you get meaningful business from Yandex or Baidu then there's no disadvantage to blocking them.

However, they're known to ignore robots.txt. I'm unsure if this has changed in the last 4 years but you'll be looking at a server level block. Their user agents are easy to spot, a .htaccess example would be:

SetEnvIfNoCase User-agent “Baidu” spammer=yes
SetEnvIfNoCase User-agent “Yandex” spammer=yes

<Limit GET PUT POST>
order deny,allow
deny from env=spammer
</Limit>
Improve this answer
1
  • Can you also put an nginx equivalent? (I understand that using the if conditional there is not a good idea, though most configs on the net use it)
    – Ethan Collins
    Mar 17, 2017 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.