2
  1. I put this rule into htaccess file to deny empty referers which is returning 403

    SetEnvIfNoCase Referer "^$" bad_user  
    Deny from env=bad_user
    

    I can see the log, it's denying also Googlebot which is acting as empty referrer also. Is there a way to modify the rule to allow access for Googlebot and deny all the rest empty referrers?

  2. I have blocked one referrer www.example.com using

    RewriteCond %{HTTP_REFERER} example\.com [NC]  
    RewriteRule .* - [F]
    

    It's working well, returning 403, but what if he is referring from example.com/another_page?

    So I did this:

    RewriteCond %{HTTP_REFERER} example\.com [NC,OR]  
    RewriteCond %{HTTP_REFERER} example/another_page\.com/ 
    RewriteRule .* - [F]
    

    Is it correct?

  3. How can I block this user agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com? What should the rule look like? Will this one work?

    RewriteCond %{HTTP_USER_AGENT} ^nbertaupete95(at)gmail.com [NC]  
    RewriteRule .* - [F,L]
    
  • It is not the referrer which is causing the 403, rather it is the server. A 403 Indicates that the server understands the request, but refuses to authorize the request. I suggest referencing the question asked in Stack overflow here: stackoverflow.com/questions/3297048/… – Josh Salganik Nov 9 '17 at 21:52
  • yes..but the server is telling this way that the referrer has no permission to access the website, right? – Michael Kowalski Nov 9 '17 at 22:54
  • Once thing is for certain (or at least in most cases), which is that we certainly would not want to deny the GoogleBot. Does your site have a robots.txt file? – Josh Salganik Nov 9 '17 at 22:57
  • yes it does..thats why Im asking how to refuse an empty referrer without blocking the googlebot. – Michael Kowalski Nov 9 '17 at 22:59
  • 1
    Why are you trying to deny access to his with no referrer? In addition to search engine spiders not using referrers, visitors that type in your URL or visit from an email won't have them. Some users install browser plugins to disable referrers. The referrer is so often empty that you could be causing problems for 10% or more of your legitimate visitors. – Stephen Ostermiller Nov 9 '17 at 23:59
1

As you have found (and warned against in comments), you should not try to block requests that contain an empty Referer header. Google (and most bots) and many legitimate users will not send the HTTP Referer header (at least at some point), so this will just cause you problems.

To answer the two remaining queries in your question...

  1. I have blocked one referrer www.example.com using

    RewriteCond %{HTTP_REFERER} example\.com [NC]  
    RewriteRule .* - [F]
    

    It's working well, returning 403, but what if he is referring from example.com/another_page?

You don't need to change anything to your existing directive(s). The first RewriteCond directive already blocks any referer that simply contains example.com. Note that example\.com is a regex (regular expression) (which is why the dot is backslash escaped). Without any anchors, the pattern is naturally matched anywhere inside the HTTP_REFERER.

(However, the pattern in your additional directive example/another_page\.com/ is a bit mashed anyway, so would never match as intended. That wouldn't have mattered, since it will have matched the first condition anyway.)

  1. How can I block this user agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com? What should the rule look like? Will this one work?

    RewriteCond %{HTTP_USER_AGENT} ^nbertaupete95(at)gmail.com [NC]
    RewriteRule .* - [F,L]
    

No, this would not work, the regex is not correct. You have introduced a start of string anchor (^) on the regex, so it will only match user-agents that start "nbertaupete95...". In the stated user-agent string, this substring is contained within it.

Also, parentheses (( and )) are special meta characters in regex - they are used for alternation and to capture subpatterns. To match a literal parenthesis, these must be backslash-escaped, eg. \(.

So, your example should read something like the following instead:

RewriteCond %{HTTP_USER_AGENT} nbertaupete95\(at\)gmail\.com
RewriteRule .* - [F]

The L flag is not required when using F (it is implied). The NC flag on the _CondPattern_ is also unnecessary, unless this user-agent really does have case variation? Also, dots must be escaped to match a literal dot, as you did in your earlier directives.

In the stated user-agent, the "email address" occurs at the end of the user-agent. If this is the case, then you can add an end-of-string anchor ($) to the regex, for example: nbertaupete95\(at\)gmail\.com$.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.