Tag Info

New answers tagged

1

I may be somewhat overboard with my regex part, but these lines in your apache configuration will block them. Basically it scans the entire user agent identification string to see if it contains "googlebot", "bingbot" or "alexa" and if it does then direct users to the default error 403 page RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^(.*)googlebot(.*)$ ...


0

This line: Checks to see if mod_rewrite is installed in apache and if it isn't then anything between the above line and... </IfModule> is ignored. This line: RewriteCond %{QUERY_STRING} http://www.google.com/humans.txt\? [NC] checks to see if the URL contains http://www.google.com/humans.txt right after the question mark and the casing ...


2

It looks like that code was developed to stop bots probing humans.txt via query strings. Not Blocked: http://example.com/humans.txt Blocked: http://example.com/?some_path=http://www.google.com/humans.txt? There is plenty of online guides about blocking humans.txt additionally lots of websites explaining what does what, a lot of those rules in that block ...



Top 50 recent answers are included