Ask Google to remove thousands of pages from its index after cleaning up from hacked site

Question

I've had my website hacked. I have cleaned it and searched google site:example.com and made a list of all the results. Removed the actual links that are OK and made a list of the ones in the format of example.com/ad0-b1fermarte54eb17chb-1244425. I've submitted these links to Google to remove and Google removed these links.

The problem here is that after my initial search, that resulted in 200 URLs which we've removed, other new links appeared out of nowhere. These links don't actually work anymore and return a 404 page, but Google still has these in its results.

We've tried deleting our sitemap and re-submitting it. Google has crawled it, but again, it has not removed the dummy search results.

Looking in our google webmaster console, we saw that in the coverage tab the number of indexed links spiked from 230 (normal usage) to 10.900 links - I'm guessing these are all dummy links that were infected and now cleaned.

Any magic way of having Google remove dead links automatically? Or a way to force it to recrawl the entire website?

Answer 1

Google will automatically remove pages that now return a 404 status. They will get removed 24 hours after Googlebot next tries to crawl them. If you want to speed up the process slightly, return "410 Gone" status for those URLs instead. Then they will be removed without the day grace period after they are next crawled.

The only problem is that it may take Googlebot months to get around to crawling all those dead pages. If you want to speed the crawling up, you have two options:

1. Submit each URL individually to the Google Search Console URL removal tool.
2. Create a temporary sitemap of all the dead URLs and add that sitemap to Google Search Console. (reference)

To get a list of all the URLs, I would suggest using your server logs. They will have a more complete record of the URLs than a site: search or Google Search Console. I would use grep on the command line. If the all the URLs are similar to the URL you posted, you could come up with a regular expression pattern for them. That URL is 31 characters long with letters, dashes, and digits. It ends in a numbers. Maybe something like this. It will look for 15 to 30 of those characters followed by a dash and 4 to 10 digits.

grep -oE '/[0-9a-z\-]{15,30}-[0-9]{4,10}' /var/log/apache2/example.com.log

Answer 2

This problem won't be solved by pinging Google to recrawl your site or resubmitting the sitemap because it would index the new URLs and not delete the old/dummy ones.

The Webmaster tool used for URL removal is the only way to ask Google to remove links from its index, however, it only allows one link at a time, to be submitted for removal.

In order to overcome this, you can use a chrome extension to automate this process. It is a paid tool(about $9) on chrome extensions store but you can get it for free on GitHub.

1. Go to this link.
2. Download the .zip file.
3. Extract and import into the chrome extensions.

Now reload your URL removal tab and you will see an option to upload a .csv or.xls file.

Download the list of URLs that you need to delete from Search console and upload the file here. (These links will be excluded from your sitemap so you would find the list of these URLs easily)

Let the tool do its job because it sure will take time depending upon the number of links you have.