How do I realize a UA string block by regular expression in the config files of my Apache webserver?

For example: if I would like to block out all bots from Apache on my debian server, that have the regular expression /\b\w+[Bb]ot\b/ or /Spider/ in their user-agent.

Those bots should not be able to see any page on my server and they should not appear neither in the accesslogs nor in the errorlogs.

http://global-security.blogspot.de/2009/06/how-to-block-robots-before-they-hit.html supposes to uses mod_security for that, but isn't there a simple directive for http.conf?

  • Are you sure you don't want to just have them not crawl your site using a robots.txt? (seems more efficient then doing an Apache regex for every request) – dan Jun 26 '13 at 5:53
  • I have many websites on the server, so I would have to add a robots.txt to each folders at once and only if there is none yet. the simplest solution would be such a directive in the main config. And I want to unclutter the accesslogs and errorlogs too – rubo77 Jun 26 '13 at 6:00
  • I see. I'm not sure specific regexes are going to catch them all, particularly those with differing or missing UA's. See this for more: Crawler Identification and Examples – dan Jun 26 '13 at 6:12
  • I know I can't get them all, but I still would like to add this easy solution to reduce some traffic and logging. I dot't need a complete solution, just an easy question: What line would I have to add in the apache-config? I can't find it ;) – rubo77 Jun 26 '13 at 6:29
  • 1
    I'll take a look at the regex for what you have. Here's list of UA's for spiders, robots, and crawlers though: List of User-Agents The regex will be very incomplete indeed. – dan Jun 26 '13 at 6:31

I enabled rewrite engine in Apache:

a2enmod rewrite

and added this block to my /etc/apache2/httpd.conf

<Directory /var/www/>
       <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} sosospider [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} BaiduSpider [NC]
                # Allow access to robots.txt and forbidden message
                # at least 403 or else it will loop
                RewriteCond %{REQUEST_URI} !^/robots\.txt$
                RewriteCond %{REQUEST_URI} !^/403\.shtml$
                RewriteRule ^.* - [F,L]

and restarted Apache:

apache2ctl graceful

now these calls from those spiders all cause 403 Errors:

grep -E 'spider|bot' /var/log/apache2/*.log
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.