Google Search Console Removal Tools Consumes 20 GB of RAM

Question

I work as a customer support specialist for a large site (millions of pages). Every day, we receive approximately 1000 removal requests from users and businesses. We remove listings from our site, then we submit removal requests to Google to remove information from Google's cache.

Now, we are facing two issues with the Google Search Console.

1. Google has a daily quota on the number of requests you're allowed to submit. We (and our users) are allowed to submit 1000 or fewer requests (combined) per day. Sometimes, we'd like to submit 1100-1200 requests.
2. I think the Google Search Console (GSC) has a memory leak. We've submitted 30k+ requests so far this year. GSC loads the history of ALL requests into a browser's DOM. As a result, the GSC quickly consumes 1.5-2 Gigs of RAM after 5-10 new requests. Every day I see a situation when a browser consumes 20-30 Gigs of RAM. It quickly becomes unresponsive (I work on MacBook Pro 16-inch, 2019).

What we have tried:

1. We tried to use different browsers: Chrome, Firefox, Safari, Vivaldi, Brave. This did not help us.
2. We tried to give the administrator's role to a new Google account. No luck. All 30,000 requests are listed under the new account as well.
3. We tried to use sitemaps. Removed pages return the 410 error code. We started to add removed links into a special sitemap. When we add a new page, we ping Google to notify them — the sitemap has been updated. Unfortunately, it does not help. Google keeps the removed pages in cache for several weeks.
4. We tried to remove 30,000 DOM elements. It did not help us. Google loads those elements through JS script.

What would you do if you were in my place? What is the best way to remove 100s of pages from Google's Cache daily? Is it possible to 'reset' the history?

Answer 1

...to remove information from Google's cache.

Not a solution to your immediate problem, but if the issue is with Google's "cache" (rather than the index per se) then maybe you should be telling Google not to "cache" these pages in the first place (where Google provides an additional menu option to view a cached version of the page).

If the indexed pages are not available in Google's cache then the urgency to remove these URLs from the index is reduced since the content is no longer accessible. The URLs will drop from the index in good time (providing you are serving a 410 and maybe a temporary "404" sitemap etc.).

For example, to prevent Google offering a cached version of the page in the SERPS then you can either...

Add the following in the head section of the HTML source:

<meta name="robots" content="noarchive">

Or, set an HTTP response header on the relevant (or "all") pages:

X-Robots-Tag: noarchive

Answer 2

during the day, generate a list of urls that need to be removed from the Google cache in a regular text document, and at the end of the day left it in the root directory of the site and sent this sitemap for re-indexing

inform users that the page on the website has already been deleted, and when requesting url, the server response with 404 Not Found

according to the google procedure, it will disappear from the search within 2 weeks. if user need to speed up the removal of a page from search, the user can contact Google on their own

Answer 3

We remove listings from our site, then we submit removal requests to Google to remove information from Google's cache.

Why bother? When a third party asks you to take down content (typically via a DMCA), your responsibility ends at your domain name. What happens on other sites is the responsibility of managers at those other sites.

And the complainer will be filing with Google anyway. When people "gear up" (scripts etc.) to do large-scale takedowns, Google is the first site they target. Why? Typically their #1 concern is stopping the offending content from being found, and Google is an easy-to-work-with, one-stop shop that covers all sites. Far easier than the non-scalable skilled labor of tracing a site to a particular ISP, figuring out how to reach and serve that ISP, how to send them a URL list in electronic form, etc. etc. So, since they are already working with Google, why bother?

Also you could not satisfy that requirement in any case. Google never imagined the Search Console would be used for mandatory takedowns at such a scale... that's why it's breaking down! And Google does not vouch for its suitability for that purpose. So it is not a valid way to satisfy any obligations even if you did have any (which you don't).

If you have large volumes of mandatory takedowns, you need to be talking to Google Legal about that, and if appropriate, they will provide you an alternate way to do that reliably. But I think they will tell you what I just said: "Tell your complainer to contact Google about takedowns on search results".

And this situation is only going to get worse. I imagine it didn't start out pushing 1000 URLs a day... it snowballed over time. It will continue to do so, and you're only looking at it now because it's not scaling. It won't scale; it'll only get worse.

If you think you have an obligation here, you may have simply fallen into the common fallacy that Google is the only search engine. The complainers don't expect you to tidy up on Bing, Yahoo, DuckDuckGo, Yandex, Baidu, Bojeek, Qwant, Unbubble, Mail.ru, Referat, Ecosia, etc. etc. Obviously, the idea that you need to go to all of them is laughable. What, search your web logs to determine which spiders have seen the offending page? It's just not workable, and any complainers who expect that are out of line to demand it. They won't get far in a lawsuit, because their own lawyer will tell them that.

The bonus service of adding the URL into Search Console is a limited commodity with value to your company, that you don't owe them. And it robs from your company's ability to manage your own URLs as the tool intended. As such, you could charge this as a premium, and even set up a UX for it: when they submit their list of URLs, they could tick a box and provide their credit card details, and charge them $5 a URL to load them into Search Console (limit 100 per day). I don't think you'd get many takers, since as I say, they're already hitting Google themselves. However even this is not a scalable product since if you get 1001 requests in a day, you can't make good on it.