2

How do I realize a UA string block by regular expression in the config files of my Apache webserver?

For example: if I would like to block out all bots from Apache on my debian server, that have the regular expression /\b\w+[Bb]ot\b/ or /Spider/ in their user-agent.

Those bots should not be able to see any page on my server and they should not appear neither in the accesslogs nor in the errorlogs.

http://global-security.blogspot.de/2009/06/how-to-block-robots-before-they-hit.html supposes to uses mod_security for that, but isn't there a simple directive for http.conf?

7
  • Are you sure you don't want to just have them not crawl your site using a robots.txt? (seems more efficient then doing an Apache regex for every request)
    – dan
    Commented Jun 26, 2013 at 5:53
  • I have many websites on the server, so I would have to add a robots.txt to each folders at once and only if there is none yet. the simplest solution would be such a directive in the main config. And I want to unclutter the accesslogs and errorlogs too
    – rubo77
    Commented Jun 26, 2013 at 6:00
  • I see. I'm not sure specific regexes are going to catch them all, particularly those with differing or missing UA's. See this for more: Crawler Identification and Examples
    – dan
    Commented Jun 26, 2013 at 6:12
  • I know I can't get them all, but I still would like to add this easy solution to reduce some traffic and logging. I dot't need a complete solution, just an easy question: What line would I have to add in the apache-config? I can't find it ;)
    – rubo77
    Commented Jun 26, 2013 at 6:29
  • 1
    I'll take a look at the regex for what you have. Here's list of UA's for spiders, robots, and crawlers though: List of User-Agents The regex will be very incomplete indeed.
    – dan
    Commented Jun 26, 2013 at 6:31

1 Answer 1

1

I enabled rewrite engine in Apache:

a2enmod rewrite

and added this block to my /etc/apache2/httpd.conf

<Directory /var/www/>
       <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} sosospider [NC,OR]
                RewriteCond %{HTTP_USER_AGENT} BaiduSpider [NC]
                # Allow access to robots.txt and forbidden message
                # at least 403 or else it will loop
                RewriteCond %{REQUEST_URI} !^/robots\.txt$
                RewriteCond %{REQUEST_URI} !^/403\.shtml$
                RewriteRule ^.* - [F,L]
       </IfModule>
</Directory>

and restarted Apache:

apache2ctl graceful

now these calls from those spiders all cause 403 Errors:

grep -E 'spider|bot' /var/log/apache2/*.log

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.