5

Some bots have been crawling my site for every link that ends with:

?utm_source=dlvr.it&utm_medium=twitter

I haven't checked out its IP.

Then other bots (10+) follow the link rules ?utm_source=dlvr.it&utm_medium=twitter and also crawl my site. This results in a huge amount of traffic which then causes my site to shut down. I have added 10+ other bots into my blacklist with an HTTP 403 status code when they access my site.

But I think the best way is to find out the first bot which crawled my site for every link that ends with:

?utm_source=dlvr.it&utm_medium=twitter

Or, use a an HTTP 403 redirect status code when the URL contains:

?utm_source=dlvr.it&utm_medium=twitter

I know of some ways to add .htaccess code that would prevent someone from crawling my xmlrpc.php page, such as:

<Files xmlrpc.php>
Order Deny,Allow
Deny from all
</Files>

But what about a query in URL?

  • Are you sure that utm_source and utm_medium are not required? They are typically used by Google Analytics to track campaigns. – MrWhite Feb 14 '15 at 11:45
4

If you have the mod_rewrite module installed, then you can put this in your .htaccess file in the root folder of your website (which usually is the public_html folder):

RewriteEngine On
RewriteRule ^\?utm_source\=dlvr.it\&utm_medium\=twitter$ - [R=403,NC,L]

You might have to remove the \ from the =, I can't remember if equals needs escaping.

Another way would be this if you're searching for the string anywhere in the URL:

RewriteEngine On
RewriteRule ^(.*)\?utm_source\=dlvr.it\&utm_medium\=twitter(.*)$ - [R=403,NC,L]

The NC at the end means not case-sensitive, so if the text is all upper-case, then the bots would be directed to an error 403.

|improve this answer|||||
  • 2
    Did the OP actually try this? This cannot work, unless the query string is not actually a query string and the ? is %-encoded in the request? The query string is removed from the URL-path before pattern matching. – MrWhite Feb 13 '15 at 9:59
  • I think rules in mod_rewrite take precedence over just standard files. That's why I posted that answer. One nice thing about that module is that you can make non-existing folders appear to the end user as real folders on the system. For example, To make the end user think folder abc exists in the root of the website and make your own file manager that displays contents of it inside PHP, put this line in .htaccess RewriteRule ^abc$ /somefile.php?folder=abc [L] where somefile.php is the script that does the processing. – Mike Feb 14 '15 at 0:18
  • Hi @w3d, so how about use SetEnvIf Request_URI "(?utm_source=dlvr.it&utm_medium=twitter)" banned Deny from env=banned in htaccess? – space and time Feb 14 '15 at 9:10
  • 1
    @spaceandtime That wouldn't work either, the query string (I assume this is an ordinary query string on the request?) is omitted from Request_URI - the same reason why the above mod_rewrite directives won't work (although there are also other issues with the regex). (Although you've marked the answer as "accepted" and not stated anything to the contrary, which implies that this does work for you?) – MrWhite Feb 14 '15 at 11:53
4

?utm_source=dlvr.it&utm_medium=twitter

utm_source and utm_medium are used by Google Analytics (and possibly other trackers) to monitor campaigns, so blocking access purely on this query string does not "feel right", however, if this is correct in your situation then ok...

An important point to realise with query strings is that they cannot be matched using mod_rewrite's RewriteRule alone (or using the Request_URI variable in mod_setenvif - as suggested in comments). The query string is removed from the URL-path before it is matched against the RewriteRule pattern.

Enable the rewrite engine (mod_rewrite) if not already:

RewriteEngine On

You need to use the RewriteCond directive. So, in order to serve a "403 Forbidden" for all requests that match the above query string then you can use something like:

RewriteCond %{QUERY_STRING} =utm_source=dlvr.it&utm_medium=twitter
RewriteRule .* - [F]

This is an exact match for the specified query string (it's not a regex, so the dot does not need to be escaped). The L flag is not required when specifying the F flag (it is implied). (F is shorthand for R=403.)

If you need it to be less restrictive and match all query strings that just starts with the above query string then use a regex:

RewriteCond %{QUERY_STRING} ^utm_source=dlvr\.it&utm_medium=twitter
RewriteRule .* - [F]

You can also make the regex case-insensitive by using the NC (NOCASE) flag. However, only use this if you specifically want to ignore case in the match. People tend to append this flag out of habit, however, it's often unnecessary (or sometimes even incorrect) and just makes the regex engine work that bit harder.

This is not particularly efficient since every request will be processed. If, for instance, only the URLs within the /path/to/files path is targeted then you could make the RewriteRule pattern more restrictive:

RewriteRule ^path/to/files/ - [F]
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.