Ensure that all pharma URLs are removed from Google after cleaning up from a hacked site

Question

My client's website was hacked by a "pharma hack" which was a WORDPRESS website. Since then we reworked the website(design and logic) and it's a completely new website that is no longer using WORDPRESS or any other CMS, it's just plain PHP, JS and CSS files with a few forms.

The website is hosted on https://www.digitalocean.com/ and I've rebuilt the droplet that it was using, added a firewall, redirected all http traffic to https and the only thing that is the same is IP Address with the domain name.

After cleaning the server and website I've begun to clean the search results by using Google search console. In the tools google provides I tried using "URL Inspection tool" and requested indexing of the website, submitted a "sitemap.xml" and used "Removals tool" to remove cached content. But sadly the search results stayed the same.

Next thing I tried was returning a 410 status code for every page that doesn't exist by using .htaccess file to redirect non-existing pages in my website to 404.php page with the code below.

<?php
   header($_SERVER["SERVER_PROTOCOL"]." 410 Gone"); 
   exit;
?>

After these changes I can see in apache logs that crawler bots (e.g SemrushBot, Dotbot, Googlebot, Petalbot and etc.) or some unknown user agents (e.g The Knowledge AI, ANTIPIDERSIA) are requesting infected pages(that no longer exist) and receives mostly 410 status code or sometimes 301 and right after a duplicate request with 410 status code.

For example

66.33.212.13 - - [23/Jul/2021:18:24:51 +0000] "GET / HTTP/1.1" 301 557 "-" "ANTIPIDERSIA"
66.33.212.13 - - [23/Jul/2021:18:24:51 +0000] "GET / HTTP/1.1" 200 43030 "-" "ANTIPIDERSIA"

64.62.252.176 - - [23/Jul/2021:20:22:35 +0000] "GET /cialis-long-term-effects/ HTTP/1.1" 301 675 "-" "The Knowledge AI"

66.249.70.92 - - [23/Jul/2021:18:38:41 +0000] "GET /viagra-discounts/ HTTP/1.1" 301 659 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.69 - - [23/Jul/2021:18:38:42 +0000] "GET /viagra-discounts/ HTTP/1.1" 410 5502 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I was hoping that if bots receive 410 status code they will stop indexing them and remove them, but like before my search results were not starting to clear up.

Next I found out I can disavow links to my website by using this tool. I copied all unwanted search results and put them in a .txt file and submitted it to the tool.

All this is the span of a few days and I understand that it could take a while to clear up, but what's bothering me is that search results length are changing in number. Sometimes they go down and sometimes they go up. One day there are 60+ other day it's 100+ the other day its 80+ and so on...

To inspect search results I use keyword site:sitename and this gives different result lengths depending whether I include www, non-www, http or https. There are only 6 pages in my website so only 6 results should be present.

Currently I have no robots.txt to allow bots to crawl.

Am I missing something? Can someone point me to a right direction?

---

Added additional info 2021-07-28

.htaccess as requested

Options +MultiViews
Options All -Indexes

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.php [NC,L]

ErrorDocument 404 /404.php

---

Added additional info 2021-07-30

This can be improved if we know more precisely the format of your URLs and of the "spam" URLs you are trying to rid your system of. For example, do your URLs include a trailing slash?

No my urls do not have an ending trailing slash nor do links inside.

URL examples:

domain.com
domain.com/contacts

Links examples

/
/contacts

And since you asked I've noticed in apache logs that infected URLS mostly do end up with a trailing slash with some rare exceptions where it doesn't or ends with .html extension.

I would categorize these URLS in 3 ways.

1. URLS that end with .html extension.

114.119.158.142 - - [29/Jul/2021:06:50:57 +0000] "GET /gambling/it/slots/gonzo's-quest.html HTTP/1.1" 410 4785 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"

2. URLS that starts with /maint/index.php/ but not always end with a trailing slash.

216.244.66.194 - - [29/Jul/2021:20:07:17 +0000] "GET /maint/index.php/pc-Asst-2-x-28-sanding-59292/ HTTP/1.1" 410 4762 "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])"

114.119.154.161 - - [29/Jul/2021:20:12:19 +0000] "GET /maint/index.php/Leder-Schulbedarf-Sonnenleder-25771 HTTP/1.1" 410 4785 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"

3. URLS that end with a trailing slash

78.46.61.245 - - [29/Jul/2021:06:51:01 +0000] "GET /best-online-tadalafil/feed/ HTTP/1.1" 410 4928 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"

---

Added additional info 2021-07-30

After I updated my .htaccess the way MrWhite mentioned in his comment, my spammed results were removed 90% in less than 12 hours. Amazing! This is the way.

Answer 1

Ideally, you should be serving an immediate 410 Gone (no "redirect") for these "spam" URLs that you want to be completely removed and a 404 otherwise. Serving a 410 for every non-existent URL is a short term workaround.

.htaccess as requested

Options +MultiViews
Options All -Indexes

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.php [NC,L]

ErrorDocument 404 /404.php

There are no (301) redirects in the code you've posted, so where are these coming from? This (unnecessary) redirect should be resolved. (Although the updated .htaccess code I've posted below might workaround this.)

You also have a conflict between MultiViews and mod_rewrite. For some reason^*1, you have explicitly enabled MultiViews, but this does the same (and more) as what your mod_rewrite directives are trying to do... ie. append the .php extension. MultiViews will "win" here, so your mod_rewrite directives aren't actually doing anything.

^*1 You might have enabled MultiViews (in desparation) if your URLs end in a trailing slash (as per the example "spam" URLs in your log). In this case the mod_rewrite directives as written will not work, so you would be dependent on MultiViews for this to work at all.

It would be preferable to disable MultiViews and improve your mod_rewrite directives, so that instead of rewriting every request (that does not map to a directory or file) to append .php, only append the .php extension if the corresponding .php file exists. This avoids a second file system check and the 410 can then be triggered on the initial request, rather than the rewritten request. eg. Currently /viagra-discounts/ would first get rewritten to /viagra-discounts/.php before triggering a 404 (later converted to 410 by 404.php).

You also don't need the 404.php file to convert the 404 to 410 - this can all be done in .htaccess.

For example:

ErrorDocument 404 /404.php

# Disable directory listings AND MultiViews
Options All -Indexes -MultiViews

RewriteEngine On

# Abort early if the request maps directly to a file or directory
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]

# Serve a 410 for URLs that don't contain a dot (ie. that don't look like static files)
# and don't map to a corresponding ".php" file
RewriteCond %{DOCUMENT_ROOT}/$1.php !-f
RewriteRule ^([^.]+?)/?$ - [G]

# Append ".php" to remaining (non-static file) requests
# Handles URLs both with and without a trailing slash
RewriteRule ^([^.]+?)/?$ $1.php [L]

You should change 404.php so that it doesn't force a 410.

Now, not every 404 is changed to a 410. Only URLs without a dot (ie. don't look like a static resource) that don't map to a .php file are served a 410.

eg. A request for /viagra-discounts/ will immediately trigger a 410 Gone, because /viagra-discounts.php does not exist.

The last rule does not need a preceding condition (RewriteCond directive), since if the preceding rule was not successful then the corresponding .php file must exist.

NB: Literal dots do not need to be backslash-escaped when used inside a regex character class, as you had initially (eg. [^\.]+). The dot carries no special meaning when used inside a character class. So, [^\.]+ and [^.]+ are the same.

This can be improved if we know more precisely the format of your URLs and of the "spam" URLs you are trying to rid your system of. For example, do your URLs include a trailing slash?

Also consider implementing a redirect from the .php file request to the non-PHP URL. Although strictly unnecessary if the .php URLs have never been exposed.

To inspect search results I use keyword site:sitename

Bear in mind that site: searches can return results that are not ordinarily returned in organic search. So this may not be a true reflection on what users might see.

---

UPDATE: Following your update (on 2021-07-30) that describes the URL format more precisely, the above rules are not strictly correct. In that some of the spammy URLs (namely those that end in .html or contain index.php will result in a 404, not a 410 - unless you have kept the 404.php document that converted 404s to 410?).

If you are never using .html files or passing path-info to /maint/index.php then you can resolve this by adding a couple of additional rules immediately after the RewriteEngine On directive. For example:

# Reject all requests that end in ".html"
# ie. Serve a 410 Gone
RewriteRule .\.html$ - [G]

# Reject all requests that start "/maint/index.php/"
RewriteRule ^maint/index\.php/ - [G]

These additional "spam" URLs will now be served a 410 instead of a 404.

Answer 2

I wouldn't use the 410 error code. It's better to stick with 404 error codes on all those pages you need to remove. That way, you are telling the search engines that they don't indeed exist anymore.

They shouldn't redirect either. In the part of the log you posted, it sends the Googlebot through a 301 redirect. Try to send just the 404.

Once you are sure they all show 404 error codes, use the Google Removal Tool and request to remove each of the offending pages: https://search.google.com/search-console/removals

You can also use the same approach with Bing's Removal tool: https://www.bing.com/webmasters/tools/contentremoval

The removal takes a while to process and everything, but it's the fastest approach I know to clean up the mess. I hope it helps!