2

Our site was hacked and links to random content where added to the site. We completely removed the hacked site and put a new one in its place. Everything new including images and content, no other part of the old site was used.

The problem we have now is that the hacker has submitted 100,000's of links to the search bots and the server is continuously visited every 1 second by the bots trying to index the links that don't exist and have never existed on the old and the new site.

We have tried to combat this using the htaccess file of the site with several instances of various conditions and rewrite rules that tell the bots the content is gone.

Example

RewriteCond %{REQUEST_URI} .*/product/.*
RewriteRule ^ - [R=410,L]

The trouble with this is that some requests are getting through and producing 301 and 404 errors.

This is causing the bots to retest the request again and report our site as having 100,000s of bad links.

I am looking for a solution that returns 410 code to the bots for all requests excluding all source requests that our actually part of the site.

The site only has approx 10 pages but is a Joomla CMS so there are a mass of resources that get loaded in the background to deliver the page.

My idea was to visit each page in the site and use the browsers inspect to gather a list of all resource requests that a page makes.

The question is how do I formulate this into conditions and rules for the htaccess so that all page requests including route / are delivered but the hackers links requested by the bot aren't?

Also we are working on sending emails to the bot's to say their requests are being instigated by the hacker.

  • A 410 status is not significantly different than a 404 status as far as search engine bots go. The biggest difference is that they will give a 24 grace period before de-indexing 404 pages. There is no need to ensure that every request for the spam gets a 410. – Stephen Ostermiller Nov 16 '18 at 15:37
  • An alternate solution: Optimize your 404 page so that it puts much less strain on your server. Then your server will be able to deal with all the 404 requests just fine, until the bots back off. – Maximillian Laumeister Nov 17 '18 at 17:31
  • If some URLs are giving 404 then the search engine bot has to a second call to my server to verify a 410 status should be given. I'm aiming for a one hit and done as there are 100,000's of spam urls. If I could get all spam urls returning 410 and get the search engine bot to speed up there requests then I can clear the problem quicker. Can I use anything in the robot.txt file to request the speed of a bots indexing? – 744 Nov 18 '18 at 16:20
2

You basically want to send a 410 instead of a 404.

Possibly the easiest method (if you are familiar with Joomla) is to override the error.php file in your template and manually set a 410 status when a 404 has already been determined. For example, at the top of your error.php file, something like this:

if ($this->error->getCode() == '404'){
    header("HTTP/1.0 410 Gone");
}

Alternatively, since you only have about 10 pages, you could do this in .htaccess by creating a white-list of your page URLs and trigger a 410 if the requested URL does not match your white-list and does not exist as a physical file. This also avoids the request being slowed down by Joomla (pros and cons - see end).

For example:

# Whitelist of page URLs 
# NB: Last condition must not have an OR flag
RewriteCond %{REQUEST_URI} =/second-page [OR]
RewriteCond %{REQUEST_URI} =/third-page [OR]
RewriteCond %{REQUEST_URI} =/fourth-page [OR]
RewriteCond %{REQUEST_URI} =/fifth-page [OR]
RewriteCond %{REQUEST_URI} ^/admin/
RewriteRule . - [E=PAGE_URL:yes]

# Send 410 Gone
RewriteCond %{REQUEST_URI} !\.(css|js|png|jpg|gif|svg|pdf)$ [NC]
RewriteCond %{ENV:PAGE_URL} !yes
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule !^(index\.php)?$ - [G]

This assumes the home page (ie. /) is a valid page URL. The G (gone) flag is shorthand for R=410 and the L flag is not required (it is implied).

The first RewriteCond directive excludes URLs that look like static resources (denoted by file extension) from being served a 410 when they don't exist - they should get the default 404. This assumes the spam URLs are not for these types of static resources. If you are getting spam-like requests for nonsense .css files for instance then remove that extension from the RewriteCond directive or remove the condition entirely to serve a 410 for all these missing resources.

The = prefix on the CondPattern eg. =/second-page, makes it an exact match string comparison, not a regular expression, so no need to escape regex meta chars etc.

However, this loses your nice user-friendly "Joomla 404 page", so you may need to create one.


UPDATE:

Some of the spam urls have this type of structure /?anything=

If you want to block any URL that contains a query string then you need a condition that checks against the QUERY_STRING server variable. Try adding the following:

# Block any URL that contains a query string on unknown pages
RewriteCond %{QUERY_STRING} .
RewriteCond %{ENV:PAGE_URL} !yes
RewriteRule ^ - [G]

The single dot regex (that matches a single character) simply checks that there is a query string present.

  • 1
    In their question they say they have about 10 pages but additional css, js, and images. They were asking how they could get a list of all the pages and resources to include into .htaccess. – Stephen Ostermiller Nov 16 '18 at 16:19
  • 1
    You don't need to list all static resources that map to physical files as you can simply do a file system check. But I'm also assuming that these "spam" URLs are not for .css, .js files etc. so these URLs are simply ignored (ie. processed normally). – MrWhite Nov 16 '18 at 16:55
  • Yes the spam urls are also targeting a range of file extensions. When I tried a 410 for the whole site with a single page exception as suggested by MrWhite, the file resources for the page had a 410 reported in the logs and the resource was missing from the page. The homepage / is a valid page URL. I can get all the source paths and build the exceptions using spreadsheet joins based on your example. The thing I'm unsure of is the homepage / exception. Anything other than / or exceptions should give a 410. Example: example.com/anythingbutexceptions should give a 410. – 744 Nov 18 '18 at 15:05
  • While I'm building the resources urls how do I deal with these types of links: data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMzIiIGhlaWdodD0iMzIiIHZpZXdCb3g9IjAgMCAzMiAzMiIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48cGF0aCBkPSJNMTEuNDMzIDE1Ljk5MkwyMi42OSA1LjcxMmMuMzkzLS4zOS4zOTMtMS4wMyAwLTEuNDItLjM5My0uMzktMS4wMy0uMzktMS40MjMgMGwtMTEuOTggMTAuOTRjLS4yMS4yMS0uMy40OS0uMjg1Ljc2LS4wMTUuMjguMDc1LjU2LjI4NC43N2wxMS45OCAxMC45NGMuMzkzLjM5IDEuMDMuMzkgMS40MjQgMCAuMzkzLS40LjM5My0xLjAzIDAtMS40MmwtMTEuMjU3LTEwLjI5IiBmaWxsPSIjZmZmZmZmIiBvcGFjaXR5PSIwLjgiIGZpbGwtcnVsZT0iZXZlbm9kZCIvPjwvc3ZnPg== – 744 Nov 18 '18 at 15:29
  • I've updated my answer... the previous code would have possibly blocked the home page! You really shouldn't have to list all your static resources, if that is what you are suggesting? That would be impossible to manage on anything but the smallest of sites. Static resources are physical files, so you only need to check that the file exists. (Unless you also want to serve 410 for select resources that do exist?!) A data URI is embedded in the document itself - it's not a separate resource. – MrWhite Nov 18 '18 at 22:03

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.