CPanel Visitors filled with bandwidth thieves

Question

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots.

Background: I have never had any website or IT training or instruction. I created several websites about transportation hobby subjects, which should be very calm and uneventful. The kind of stuff that might be interesting to a dozen or so people in the USA, Canada, Japanese, UK, and Australia. These websites were hosted with a web hosting company run by an absentee manager. They shut down and I transferred the domains to a new company, which turned all the controls over to me. About a month into the new service, and the largest of the websites exceeded the 5gb basic setting and had to be bumped up into the 15gb service and increased charge. I started watching the bandwidth usage climb, and figured out how to use the IP Blocker console. I have not managed to figure out HTAccess or Robots.txt, but will leave those for another time. I have managed to block most of the worth of the SEO Profiler and Chinese/Russian bots which don't follow Robots.txt rules anyway. And I have managed to block the rotating bunch of Russian/Ukranian URLs that cluster requests for one specific file a dozen times a day. But the bandwidth draw is not decreasing.

I started noticing IPs requesting image files without any Referring URL, and not identified as a Bot, but identified as a browser or a cel phone. I know this is not normal surfing, because there are no Html files being pulled to call up the image files, and the image files are in clusters that do not correlate to any group used on any of the web pages. I am running the IPs through the Whois Lookups, and coming up with a couple groupings: Soviet Block countries - Russia, Ukraine, Bulgaria, Estonia... Latin Speaking Countries - Brazil, Spain, Costa Rica... China. South Africa. All places with nothing to do with the website content as the topic matter was not available in those countries.

And there are some IPs coming up as Canada, Washington State, New Jersey, etc., but I am hesitant to block those.

What am I looking at here? Is there some kind of money making deal of aiming remote and overseas IPs at my websites and hitting a couple pictures in rapid succession a dozen times a day from 100+ different addresses?

Answer 1

There is a way to directly block the viewing of the images using .htaccess

RewriteEngine on 
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC] 
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC] 
RewriteRule \.(gif|jpg)$ - [F]

This is going to return a 403 Forbidden error if you access the image directly, but it does allow them to be displayed on a web page. This should help with them not being able to visit them by using the direct URL to the image.

Answer 2

These are not Googlebot IP addresses. It is not uncommon for a bad bot operator to use Googlebot as an agent name to make you think they are okay. I found that a lot of bad bot operators come from Amazon IP addresses though I would not considering blocking Amazon IP addresses except one at a time.

The Apache documentation can be found here:

http://httpd.apache.org/docs/

It takes a while to sift through and understand though it is not really too difficult.

Apache configurations can be found in one of several places. You can find the base Apache directory in /etc/apache2/ or /etc/local/apache2/.

If httpd.conf file is empty, then the base configuration is apache2.conf and ports.conf. You likely do not need to edit these files, I am just mentioning them.

You should also see a sites-available directory. In this sites-available directory, you will likely see a file in the format of example.com.conf. This is the file you will want to first look at. You should see the directive AllowOverride. If this is set to None, then the use of .htaccess is not available. Change the None to All or other value.

Then, within the web space, likely /home/example.com/www, look for a hidden file .htaccess. You can use ls -al. Edit this file. You want to make sure that RewriteEngine On is the first line or near the first. If there are any ErrorDocument directives, you want to put the following before the directives so that ErrorDocument is the last directive processed.

RewriteCond %{REMOTE_ADDR} 23.20.165.34 [NC]
RewriteRule .* - [F,L]

RewriteCond %{REMOTE_ADDR} 54.198.86.102 [NC]
RewriteRule .* - [F,L]

I do not think you have to escape the . [dots]. If this does not work then try changing the IP address to something like 23\.20\.165\.34. This should work. If not, then change the IP addresses to something like ^23\.20\.165\.34$.

If you have these already blocked then you do not have to do anything. I just wanted to give you some alternatives.