Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots.

Background: I have never had any website or IT training or instruction. I created several websites about transportation hobby subjects, which should be very calm and uneventful. The kind of stuff that might be interesting to a dozen or so people in the USA, Canada, Japanese, UK, and Australia. These websites were hosted with a web hosting company run by an absentee manager. They shut down and I transferred the domains to a new company, which turned all the controls over to me. About a month into the new service, and the largest of the websites exceeded the 5gb basic setting and had to be bumped up into the 15gb service and increased charge. I started watching the bandwidth usage climb, and figured out how to use the IP Blocker console. I have not managed to figure out HTAccess or Robots.txt, but will leave those for another time. I have managed to block most of the worth of the SEO Profiler and Chinese/Russian bots which don't follow Robots.txt rules anyway. And I have managed to block the rotating bunch of Russian/Ukranian URLs that cluster requests for one specific file a dozen times a day. But the bandwidth draw is not decreasing.

I started noticing IPs requesting image files without any Referring URL, and not identified as a Bot, but identified as a browser or a cel phone. I know this is not normal surfing, because there are no Html files being pulled to call up the image files, and the image files are in clusters that do not correlate to any group used on any of the web pages. I am running the IPs through the Whois Lookups, and coming up with a couple groupings: Soviet Block countries - Russia, Ukraine, Bulgaria, Estonia... Latin Speaking Countries - Brazil, Spain, Costa Rica... China. South Africa. All places with nothing to do with the website content as the topic matter was not available in those countries.

And there are some IPs coming up as Canada, Washington State, New Jersey, etc., but I am hesitant to block those.

What am I looking at here? Is there some kind of money making deal of aiming remote and overseas IPs at my websites and hitting a couple pictures in rapid succession a dozen times a day from 100+ different addresses?

share|improve this question
    
There are a lot of bots from Russia, China, and Poland (just to start) that steal content at the very least. You seem to be doing the right thing by blocking them. I am not sure how you are blocking them. Firewalls are the best. Next I personally would use the .htaccess file. It maybe that the method you are using is just fine. Sometimes you will never know why certain sites are targeted. Often, it is because you score well for certain searches. I will think on this more. Is there a specific question that you have? –  closetnoc Mar 20 at 21:45
    
Thanks for the words of encouragement. –  Untrained Apache Operator Mar 26 at 14:38
    
I would very much like to use .htaccess, but all the references say "to activate it just type in ______" and "paste these lines in to block/stop hot linking/etc." The only problem being _where to type the commands to turn it on, and where to paste the text to block things. At the moment, my only tool is the IP block. Go down the list of recent visitors, pick out the ones that do not show normal web page viewing, run them through the whois lookup, block all bots other than Google/MSN/Yahoo and everything not from US/Canada/Japan/Western Europe. –  Untrained Apache Operator Mar 26 at 14:48
    
In the last day or so, I have found a bunch of Googlebot entries coming from 23.20.165.34 and 54.198.86.102, both show Amazon servers and were blocked because Shopwiki comes through five times a day and downloads every single page and image file. That is perplexing. –  Untrained Apache Operator Mar 26 at 14:48
add comment

2 Answers 2

There is a way to directly block the viewing of the images using .htaccess

RewriteEngine on 
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost [NC] 
RewriteCond %{HTTP_REFERER} !^http://(www\.)?localhost.*$ [NC] 
RewriteRule \.(gif|jpg)$ - [F]

This is going to return a 403 Forbidden error if you access the image directly, but it does allow them to be displayed on a web page. This should help with them not being able to visit them by using the direct URL to the image.

share|improve this answer
add comment

These are not Googlebot IP addresses. It is not uncommon for a bad bot operator to use Googlebot as an agent name to make you think they are okay. I found that a lot of bad bot operators come from Amazon IP addresses though I would not considering blocking Amazon IP addresses except one at a time.

The Apache documentation can be found here:

http://httpd.apache.org/docs/

It takes a while to sift through and understand though it is not really too difficult.

Apache configurations can be found in one of several places. You can find the base Apache directory in /etc/apache2/ or /etc/local/apache2/.

If httpd.conf file is empty, then the base configuration is apache2.conf and ports.conf. You likely do not need to edit these files, I am just mentioning them.

You should also see a sites-available directory. In this sites-available directory, you will likely see a file in the format of mysite.tld.conf. This is the file you will want to first look at. You should see the directive AllowOverride. If this is set to None, then the use of .htaccess is not available. Change the None to All or other value.

Then, within the web space, likely /home/mysite.tld/www, look for a hidden file .htaccess. You can use ls -al. Edit this file. You want to make sure that RewriteEngine On is the first line or near the first. If there are any ErrorDocument directives, you want to put the following before the directives so that ErrorDocument is the last directive processed.

RewriteCond %{REMOTE_ADDR} 23.20.165.34 [NC]
RewriteRule .* - [F,L]

RewriteCond %{REMOTE_ADDR} 54.198.86.102 [NC]
RewriteRule .* - [F,L]

I do not think you have to escape the . [dots]. If this does not work then try changing the IP address to something like 23\.20\.165\.34. This should work. If not, then change the IP addresses to something like ^23\.20\.165\.34$.

If you have these already blocked then you do not have to do anything. I just wanted to give you some alternatives.

share|improve this answer
1  
Wouldn't adding a few lines into .htaccess also block direct access to the images, but still allowing them to be displayed within the web pages itself? –  user37204 Mar 26 at 16:34
    
Yes. Though I have not done this in a long while. I have a commented out example that I cannot swear works. I know I had a working example at one point. I am not sure this one is it. –  closetnoc Mar 26 at 16:39
    
I'm trying to post my working example that I know that works, but links are not allowed... trying to find a work around for it as we speak. –  user37204 Mar 26 at 16:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.