Shall I submit sitemap containing all the bad (now 404) URLs after the spam URL hack?

Question

Earlier this month, one of my sites got hacked and an attacker created near 1 million spam URLs. The site has been fixed relatively fast, however, after one month Google search results still shows near 500,000 results (number of real pages is ~100).

My site has slowly backed up in rankings but I want to speed things a little. Since I have a list of all the bad URLs, I thought of creating a sitemap with it and submitting it to the Google Search Console. Google will look for them, will see that they are now 404 error pages and hopefully they will get removed from the index.

I saw that some people did this tactic with 100-1000 URLs, but will submitting near 1 million 404 URLs actually hurt my site even more than the actual hack? Shall I leave it as is and hope for the best?

Answer 1

You can submit a temporary sitemap with all the hacked URLs to Google as a method for getting Googlebot to come crawl all of them quickly. When you submit a sitemap of URLs, it will trigger Googlebot to crawl the URLs with greater priority.

You want Googlebot to come crawl the URLs so that it sees that they are "404 Not Found." If you don't submit a sitemap with these URLs, it could take Googlebot months to get around to crawling all of the URLs.

Since you have more than 50,000 URLs, you would actually need to submit multiple sitemaps each containing 50,000 URLs. 50,000 is the most URLs allowed in a sitemap. I would not recommend creating a sitemap index file because Google Search Console has had problems with them for the last few years. It typically doesn't show how many URLs it has found in them and it isn't clear that they work properly.

Once Google has seen a sitemap and crawled the URLs in it, you should remove it. I would expect that you should leave each sitemap up for at most a few months.

Reference: Use 404s & Temporary Sitemaps To Speed Up Page Removal From Google Search which quotes Google's John Mueller:

One way to speed this up could be to submit a temporary sitemap file listing these URLs with the last modification date (eg, when you changed them to 404 or added a noindex), so that we know to recrawl & reprocess them. This is something you'd just want to do for a limited time (maybe a few months), and then remove, so that you don't end up in the long run with a sitemap file that's not needed by your site

Submitting a sitemap with tons of 404 errors from hacked URLs won't hurt your site. Again quoting John Mueller

404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. https://developers.google.com/search/blog/2011/05/do-404s-hurt-my-site

I wouldn't try this technique with other search engines like Bing, Yandex and Baidu. They haven't said that this would be appropriate and I haven't seen any indication that it would work with them.

Answer 2

Leave it as is and hope for the best.

Why you shouldn't list 404s in your sitemap.xml

Your sitemap should only contain URLs you wan't indexed. URLs in your sitemaps should be clean -they should return 200, be indexable, be canonical, and also unique.

If search engines come across 404 pages in your sitemaps, they may stop trusting the sitemaps for crawling and indexing signals.

---

Duane Forrester, former Bing employee

Your Sitemaps need to be clean. We have a 1% allowance for dirt in a Sitemap. Examples of dirt are if we click on a URL and we see a redirect, a 404 or a 500 code. If we see more than a 1% level of dirt, we begin losing trust in the Sitemap.

Full Interview

Answer 3

In addition to @StephenOstermiller's answer, to further expedite the removal of these URLs you should:

• ensure they return a "410 Gone" status (rather than simply a 404 Not Found - which is perceived as more "temporary").
• Submit these URLs to the Google Removal tool - although this will be a bit tough if these URLs do not have a common prefix.
• Make sure they are not blocked by robots.txt!

Reference:

• https://support.google.com/webmasters/answer/9689846?hl=en

Google SERP still shows near 500000 results

Presumably this is via a site: search? If it's any consolation, Google will return results via a site: search that are not ordinarily returned in organic search (including URLs that might redirect or 404).