No-index tag that affects all pages that have a specific structure

Question

I wanted to ask if there is any possibility and how a no-index tag could be added that affects all the pages of a domain that follow the following structure example.com/nnnn97nnnn.html with "n" being any random number and 97 being in the middle They are always repeated in all URLs. Example: example.com/6512978912.html

This is a website that was affected by the Japanese Keyword Spam malware that generated thousands of URLs pointing to a Japanese e-commerce. The malware has already been cleaned, all these URLs return a 404 error and Google is gradually deindexing these URLs also with the help of the Google Search Console URL Removal tool.

However, I want to do everything possible so that these URLs disappear and are not reindexed despite giving a 404 error since, according to what other users tell me, this malware can generate links from external sites pointing to the affected site that can cause them to be reindexed despite giving error 404 and they are not completely deleted so quickly when searching with the site: command. Sometimes pages deleted with the Google Search Console URL removal tool reappear, because the request of removals expires in a couple of weeks, supposedly because there are other external links that are pointing to those URLs.

(Note: the way I see these pages supposedly being re-indexed is by doing a Google search with the site: command)

What can I do so that no URL is indexed with this structure that I mentioned and speed up the deindexing process as much as possible?

Regards, and thank you very much.

Answer 1

Noindex can only be set for URLs using one of the following two methods:

• Robots meta-tag within the HTML
• X-Robots HTTP header

The robots meta-tag would require you to actually change the HTML of those pages. The X-Robots HTTP header only requires you to change the HTTP header.

Most webservers allow you to add headers based on rules. E.g. with Apache 2.4+ mod_headers can be used to add an X-Robots HTTP header for all URLs that match your pattern. The pattern could be defined as a regular expression in your case.

Note: Setting noindex directives only makes sense for URLs with a status code 200, not for pages with an error status code 404, as the 404 HTTP status code already indicates that the ressource was not found.

To speed up the deindexation you can do the following:

• send a 410 status code instead of the 404 for the affected URLs to get the URLs removed right after the next crawl. 404 errors can be treated as soft errors and might require multiple recrawls to get the URLs deindexed.
• try adding adding a temporary XML sitemap that list all known URLs your want to deindex. Make sure to specify a modification date for all the URLs that indicate that the URL changed since the last crawling.

Note: Having 404 or 410 pages in XML sitemaps will trigger notifications in Google Search Console but it will help to speed up the deindexation anyway.

Once all URLs are gone from the index you can consider adding a "Disallow" directive to your robots.txt file to prevent crawling.

Answer 2

One thought I had for this, while risky (dependent on the current/future URLs of the site), is to: add a wildcard entry in your website's robots.txt file. Something like this could work: "Disallow: /*97*" - this will tell the bots to ignore all URLs that include "97" somewhere in the string.

Like I said, risky, if you have any URLs with 97 in them, or WILL have URLs with 97 in them. But if that is not a risk for you, then... maybe this could do something?!