How do you determine when to block an IP address or range of addresses?

Question

When analyzing traffic to some of my sites, I almost always notice a large amount of traffic from obscure sources like Japan, Russia, and other countries that really have no business visiting my site(s).

Most of the time, these "visitors" are viewing the site(s) considerably more than other visitors that appear to be legitimate.

Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter?

Also, is there a better way to handle these visitors so that I can both determine their intention(s) and "herd" them away from my site(s)?

---

EDIT:

If I can't determine the intention(s) of these "visitors" by using all available resources, would it be a bad idea (in terms of SEO) to create a page/section of my site(s) that, if visited, would tell me without a doubt that the "visitors" are bots (not robots, but bots that pry the site to expose flaws)?

Answer 1

Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter?

You tell us. If they're interacting with your site as any normal user would there is no need to block them. If they're using the site more than visitors from other regions then your site may appeal to them then users from other regions.

Also, is there a better way to handle these visitors so that I can both determine their intention(s) and "herd" them away from my site(s)?

Before you go blocking IP addresses, or entire ranges of IP addresses, you need to be sure these users are malicious. Check your logs. Do you see malicious requests? Do you see unusual activity beyond longer visit durations and page requests? What user-agents are they using? Do they identify themselves as bots? If so, have you researched them? Do they honor robots.txt?

Before you can determine a course of action you need to determine if there is even an issue here. As of right now there isn;t anything suspicious or unusual about what you've posted here.

If I can't determine the intention(s) of these "visitors" by using all available resources, would it be a bad idea (in terms of SEO) to create a page/section of my site(s) that, if visited, would tell me without a doubt that the "visitors" are bots (not robots, but bots that pry the site to expose flaws)?

Sure. Just put a link to a page on your website that humans can't find and is blocked by robots.txt. This will keep humans and good bots out. Then when this page is hit by a bad bot you can log it. Because it did not obey your robots.txt file you can lean towards it being a bad bot and blocking it. You can also compare it to your logs to see what it is up to to make a more informed decision as to your course of action.

Answer 2

You could try something like Google analytics (http://www.google.com/analytics/) that will let you see what they are doing. You also might want to consider why they have no business on your site, and consider expanding to cover what could be an interested community. also, not everyone it countries like china are malicious and wan't to steal stuff.