2

My website was hit by the pharma spam attack which created hundreds of URLs, all of which contain the last 5 characters -1234.pdf.

So that Google and other search engines will remove these quicker, I'd like to return these pages with a 410/GONE status code, is there an easier way of doing this in .htaccess other than listing all of the URLs and setting RewriteRules for every single one (a process which would take considerable hours!)

1

To serve a 410 Gone for all URLs that end in the literal string -1234.pdf then you can use the following mod_rewrite directive at the top of your .htaccess file:

RewriteEngine On

RewriteRule -1234\.pdf$ - [G]

Or to reject any URL that ends in -NNNN.pdf where NNNN is a number of at least 4 digits then you can change the RewriteRule pattern to something like: -\d{4,}\.pdf$.

but the URLs are still returning a 404.

In the case where the server has defined a custom 410 ErrorDocument and is overriding the HTTP response code then set the following at the top of your .htaccess file (make sure you've not defined a custom 410 ErrorDocument yourself later in the .htaccess file or server config):

ErrorDocument 410 default

UPDATE: we also have some other spammy URLs which have the string "stmap_9874100.html" (where the last three digits are always different), can a similar query be written to set these URLs to 410's too?

Yes, you can add another directive similar to before:

RewriteRule stmap_9874\d{3}\.html$ - [G]

Explanation of the RewriteRule pattern (regex / Regular Expression) and directive:

  • stmap_9874 - These characters are matched literally (anywhere in the URL-path).

  • \d is a shorthand character class representing any digit (0-9) and \d{3} is exactly 3 of them.

  • \. matches a literal dot, otherwise, a dot in regex matches any character (except newlines).

  • html - These characters are matched literally.

  • $ is an "anchor" that represents the end of the string. So this string of characters must occur at the end of the URL-path.

  • - (hyphen) - not part of the regex, this is the second argument to the RewriteRule directive. This is just a "placeholder" indicating no substitution. The URL is not rewritten/redirected to anything else.

  • The G flag (short for R=410) results in a 410 Gone response. (The L flag is not required here, as it is implied when specifying a response code out of the 3xx (and 2xx) range.)

  • Thanks @MrWhite, I just have tried this out and inserted the above mod_rewrite directive as suggested at the top of the .htaccess file but the URLs are still returning a 404. I have cleared my cache and double checked I have the directive correct! Can you help? – Sam Jan 29 at 17:05
  • Can you post an actual URL you are trying to catch? Do you have other directives in .htaccess? Also make sure that your server is not configured to send 404 instead of 410 (for instance, if there is a custom 410 ErrorDocument defined somewhere). – MrWhite Jan 29 at 17:08
  • The URL is usually a string of text, eg domain.com/wwwmedridnet-1234.pdf but the string of text is random, it is only the preceding 5 characters/digits that are identical In terms of other directives, I have Wordpress: # BEGIN WordPress <IfModule mod_rewrite.c> RewriteEngine On RewriteBase / RewriteRule ^index\.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L] </IfModule> Alongside that, rewrite rules to block fake referral traffic; 301 redirects for older pages & IP blocking – Sam Jan 30 at 9:01
  • Please edit your question to include your entire .htaccess file. The only reasons why this directive wouldn't work are... either: #1 The request doesn't actually end -1234.pdf. #2 There is a conflict with existing directives (do you have any mod_alias Redirect and/or RedirectMatch directives)? #3 The server is configured to return a 404 instead of a 410 (in the case of a custom ErrorDocument being defined). – MrWhite Jan 30 at 10:45
  • I've updated my answer with regards to #3 - if a custom error document is overriding the HTTP response code. – MrWhite Jan 30 at 10:51

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.