Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a www.example.com site where I serve all static content (e.g. http.//www.example.com/image.jpg) off a CDN with URLs like http://static.example.com/image.jpg

An Apache 2 .htaccess rule prohibits non-empty, non-example.com referrals:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?example.com [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://example.cloudfront.net [NC]
RewriteRule \.(jpeg|jpg|gif|bmp|png|mp3)$ - [NC,F,L]
</IfModule>

I still receive huge amounts of traffic from a very popular website that uses some sort of "news roundup" feature and hotlinks to www.example.com/image.jpg (instead of static.example.com): this resource exists (for origin pull reasons), but the URL is not published anywhere.

In fact, serving so many 403s is bringing my server down.

How can this kind of hotlink traffic be blocked, or at least, how can it be redirected to the CDN before it reaches and cripples apache2?

share|improve this question
    
If that URL isn't published anywhere then how did they know to look there in the first place? A lucky guess? –  w3d Mar 18 '13 at 19:07
1  
I suppose they assume the resource exists anyway and strip the domain part out of the URL –  blau Mar 18 '13 at 19:24
    
Indeed, I have seen SO many links not published anywhere that eventually get linked and crawled! Seems like some people also have fun guessing the next news too by entering related URLs. –  Itai Mar 18 '13 at 19:42
    
Maybe you could monetize that traffic by serving big advertisement picture (small B/W or SVG)? There is no way to get link in IMG but maybe they will get annoyed and finally remove your links. –  gertas Mar 19 '13 at 23:07

2 Answers 2

up vote 2 down vote accepted

There is no way to forward the www > cdn without reaching the request on the main server since well they are linking to you.

My advice is not to 301 redirect but rather than just flat out 404, hopefully by doing so they will stop hot linking you.

While this has some traffic on your server it shouldn't be too much and you should consider upgrading your hosting if its a problem.

My Drill Down Advice

  • Consider upgrading your hosting (IMG Hot-linking actually helps your SEO) A load balancer could serve you well here by doing the 403s, 404s or 301s.
  • Consider using a 404 if you do not want them hot-linking.

Another option is that you can use robots.txt on your CDN but again, hotlinking these days shouldn't be looked down on since well as I said its impossible to use rel=nofollow on images and it does help your SEO a little bit. I personally would appericate the traffic / links and upgrade my server. But other than that... serve them with a good old 404 if they are linking to something that doesn't exist - they will soon stop.

share|improve this answer
1  
404 vs 403 may have an impact on the end user, but as I understand it takes the same load to generate on the server. –  blau Mar 18 '13 at 19:44
1  
Yep same time. There's no other way of blocking it since its not the website linking to you making the request so either way your going to get these requests on the server.. Rather than preventing them which obviously is using resources up you could allow them which would gain SEO value and still use the same resources. –  bybe Mar 18 '13 at 19:52

Unless you have your images on a sub-domain, there is no way to separate the traffic before it hits your web server.

One performance tip that might help you is to use both nginx and Apache on the same server. Have apache run on a non-standard port. Have nginx run on port 80 and have it reverse proxy requests to apache. The rewrite rule for the images can be put on the nginx server. You might choose this setup because nginx is faster and lighter than apache and can handle many more of these simple types of requests. Here is a guide for implementing this: http://tumblr.intranation.com/post/766288369/using-nginx-reverse-proxy

share|improve this answer
    
Yes, I am also considering this setup in the future. –  blau Mar 19 '13 at 5:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.