You will find abusive bots in your log files if you have a website of any age. For some, it is a huge problem, while some sites experience less abuse.
You have an excellent idea about being able to edit the log file. Perhaps there should be a tool. Unless you are able to write code, it is not really practical to edit the log file to remove entries and there is no tool that I am aware of that does this for you. Most people seek to block these accesses as a way to keep their log files clean.
It is not exactly easy to determine who to block when. I am in the security research realm and this is a topic area for me and I tell you it is always a judgement call. But I will give you some clues.
When you look at your log file or log file analysis, you want to look for a few things:
- Accesses that do not request images.
- Accesses that do not request robots.txt.
- Accesses that do not obey robots.txt.
- Accesses that occur rapidly within a time interval that is not likely
a human.
- If accesses change browser or operating system at any point.
There are more clues of course, but it does get complicated.
- A bad bot may or may not request images. The fact that a page view is
followed by image requests is not necessarily an indication of a
human. However if accesses do not include image requests, then it is
a bot.
- A bad bot may or may not request robots.txt. Just because a bot
requests robots.txt does not mean it is a well behaved bot.
- If a bad bot requests robots.txt and it attempts to access areas
restricted by robots.txt then it should be blocked. You can create a
small image link to restricted area. It can be a page, directory
without index enabled, another image- it does not matter. Just make
sure it is something that a human would not likely follow. Just do
not a 1 pixel link. Make is a small image. If any access to this area
occurs, you should block access.
- Bad bots often access sites at a pace that is impossible to be a
human. A human can click links at a pace just less than one second.
If you have at least three accesses within 2 seconds, it is likely a
bot.
- Some bad bots can change browsers and operating systems over time,
but not always. If this occurs, then it is safe to block.
This is an area where you need to use your best judgement. You can Google domain names and IP addresses to see what experience other people have and whether anyone else is blocking access to what you have found. Use the list above to make a judgement for yourself. You will begin to see some patterns.
- Bad spiders come from similar bad neighborhoods.
- Bad spiders use a block of similar IP addresses.
- Bad spiders use subscriber sub-domains from telecos.
It depends which web server you have of course. I have not worked with IIS in a long time nor have I used any of the newer web servers. I know Apache so I will give some examples that you can use in your .htaccess file if you have Apache.
RewriteCond %{REMOTE_HOST} example.com [NC]
RewriteRule .* - [F,L]
-and-
RewriteCond %{REMOTE_ADDR} 10.0.1.101 [NC]
RewriteRule .* - [F,L]