1

Blank HTTP_REFERER user agents are downloading and uploading a couple specific files to and from our server! This is causing HUGE bandwidth and CPU usage.

We blocked blank referers on our CDN and Server.

if ($http_referer = "") {  return 403; }

Now sending 403 for those requests and Saving bandwidth:

enter image description here enter image description here

1M GB of Bandwidth

We blocked ALL Blank Referers Now see the change: enter image description here

Sample log

38.134.6.58 - - [24/Sep/2018:19:46:03 +0000] "POST /upload HTTP/1.1" 200 0 "-" "-684696252"
81.0.36.148 - - [24/Sep/2018:19:46:03 +0000] "POST /upload HTTP/1.1" 200 0 "-" "-732114871"
71.76.48.173 - - [24/Sep/2018:19:46:03 +0000] "POST /upload HTTP/1.1" 200 0 "-" "-380991683"
206.72.213.76 - - [24/Sep/2018:19:54:58 +0000] "OPTIONS /upload HTTP/1.1" 403 189 "-" "Mozilla
185.154.211.49 - - [24/Sep/2018:19:56:01 +0000] "OPTIONS /upload HTTP/1.1" 403 134 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:62.0) Gecko/20100101 Firefox/62.0"
160.20.166.18 - - [24/Sep/2018:20:06:36 +0000] "OPTIONS /upload HTTP/1.1" 403 189 "-" "Mozilla/5.0 (Linux; Android 5.0.2; SM-A300H Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36"
160.20.166.18 - - [24/Sep/2018:20:06:36 +0000] "OPTIONS /upload HTTP/1.1" 403 189 "-" "Mozilla/5.0 (Linux; Android 5.0.2; SM-A300H Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36"
66.249.88.40 - - [24/Sep/2018:20:06:36 +0000] "OPTIONS /upload HTTP/1.1" 403 189 "-" "Mozilla/5.0 (Linux; Android 5.0.2; SM-A300H Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36"
66.249.88.44 - - [24/Sep/2018:20:06:36 +0000] "OPTIONS /upload HTTP/1.1" 403 189 "-" "Mozilla/5.0 (Linux; Android 5.0.2; SM-A300H Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36"

Looks like some Android App is making this request.

Questions are

Any way to Trace and find the Application name & Developer?

How an Android app can make a request like this?

I know a normal web browser cannot send BLANK HTTP_REFERER unless the user is using some fancy extensions.

They are targeting Only 2 files.

More logs AFTER BLOCKING

  • 1
    Banning those without $http_referer is silly, you will be blocking legit users, HTTPS > HTTP will have a blank referer. If you want to prevent aggressive users/bots, then use a proper setup... e.g fail2ban, cloudflare etc. – Simon Hayter Sep 24 '18 at 23:48
  • @SimonHayter I understand, Now i cannot view my logo if i typed the address manually. if i click from google keep it will work fine. This is a Dedicated CDN and And API Server. ONLY Request from our main Domain need a valid response. When Main domain loading on user's browser it will send request to CDN and API server. So a valid user Must have an HTTP_REFERER right? I think Normal users will not affect with this change. Yes maybe some geeks who using some fancy extension will affect badly. – Vishnu Sep 25 '18 at 8:26
  • 1
    Most bots including Googlebot don't send referrers. Your work around will cause your website to get completely de-indexed from Google because Googlebot will no longer be able to crawl it. – Stephen Ostermiller Sep 25 '18 at 14:49
  • @StephenOstermiller This is a valid concern and i never thinked this before. I didn't blocked BLANK HTTP_REFERER to my Main Webpage. The Bandwidth leecher interested in our CDN file and API server. Now i went to Google Search console and used FETCH as Google bot feature. GoogleBot can index main web page no issues. But i do found Errors for STATIC files hosted on the CDN server (BLANK HTTP_REFERER Blocked). That is a bad sign because all images are deindexed from search! that will be disaster. But after few minutes google bot indexed everything without a problem. Hoping for the best! – Vishnu Sep 25 '18 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.