13

I work as a customer support specialist for a large site (millions of pages). Every day, we receive approximately 1000 removal requests from users and businesses. We remove listings from our site, then we submit removal requests to Google to remove information from Google's cache.

Now, we are facing two issues with the Google Search Console.

  1. Google has a daily quota on the number of requests you're allowed to submit. We (and our users) are allowed to submit 1000 or fewer requests (combined) per day. Sometimes, we'd like to submit 1100-1200 requests.
  2. I think the Google Search Console (GSC) has a memory leak. We've submitted 30k+ requests so far this year. GSC loads the history of ALL requests into a browser's DOM. As a result, the GSC quickly consumes 1.5-2 Gigs of RAM after 5-10 new requests. Every day I see a situation when a browser consumes 20-30 Gigs of RAM. It quickly becomes unresponsive (I work on MacBook Pro 16-inch, 2019).

vivaldi tab consumes 20 GB memory

What we have tried:

  1. We tried to use different browsers: Chrome, Firefox, Safari, Vivaldi, Brave. This did not help us.
  2. We tried to give the administrator's role to a new Google account. No luck. All 30,000 requests are listed under the new account as well.
  3. We tried to use sitemaps. Removed pages return the 410 error code. We started to add removed links into a special sitemap. When we add a new page, we ping Google to notify them — the sitemap has been updated. Unfortunately, it does not help. Google keeps the removed pages in cache for several weeks.
  4. We tried to remove 30,000 DOM elements. It did not help us. Google loads those elements through JS script.

What would you do if you were in my place? What is the best way to remove 100s of pages from Google's Cache daily? Is it possible to 'reset' the history?

1
  • 6
    If I were in your place I would stop using the tool and tell users that removals from Google take a while. Sep 26 at 19:20
17

...to remove information from Google's cache.

Not a solution to your immediate problem, but if the issue is with Google's "cache" (rather than the index per se) then maybe you should be telling Google not to "cache" these pages in the first place (where Google provides an additional menu option to view a cached version of the page).

If the indexed pages are not available in Google's cache then the urgency to remove these URLs from the index is reduced since the content is no longer accessible. The URLs will drop from the index in good time (providing you are serving a 410 and maybe a temporary "404" sitemap etc.).

For example, to prevent Google offering a cached version of the page in the SERPS then you can either...

Add the following in the head section of the HTML source:

<meta name="robots" content="noarchive">

Or, set an HTTP response header on the relevant (or "all") pages:

X-Robots-Tag: noarchive
2
  • 2
    this is a great idea, but Google will only execute it after re-indexing the page
    – skill
    Sep 27 at 1:25
  • The header route is probably going to be the most scalable in this situation...in my opinion - I wrote my answer essentially to supplement your answer. Sep 29 at 16:31
3

during the day, generate a list of urls that need to be removed from the Google cache in a regular text document, and at the end of the day left it in the root directory of the site and sent this sitemap for re-indexing

inform users that the page on the website has already been deleted, and when requesting url, the server response with 404 Not Found

according to the google procedure, it will disappear from the search within 2 weeks. if user need to speed up the removal of a page from search, the user can contact Google on their own

3

We remove listings from our site, then we submit removal requests to Google to remove information from Google's cache.

Why bother? When a third party asks you to take down content (typically via a DMCA), your responsibility ends at your domain name. What happens on other sites is the responsibility of managers at those other sites.

And the complainer will be filing with Google anyway. When people "gear up" (scripts etc.) to do large-scale takedowns, Google is the first site they target. Why? Typically their #1 concern is stopping the offending content from being found, and Google is an easy-to-work-with, one-stop shop that covers all sites. Far easier than the non-scalable skilled labor of tracing a site to a particular ISP, figuring out how to reach and serve that ISP, how to send them a URL list in electronic form, etc. etc. So, since they are already working with Google, why bother?

Also you could not satisfy that requirement in any case. Google never imagined the Search Console would be used for mandatory takedowns at such a scale... that's why it's breaking down! And Google does not vouch for its suitability for that purpose. So it is not a valid way to satisfy any obligations even if you did have any (which you don't).

If you have large volumes of mandatory takedowns, you need to be talking to Google Legal about that, and if appropriate, they will provide you an alternate way to do that reliably. But I think they will tell you what I just said: "Tell your complainer to contact Google about takedowns on search results".

And this situation is only going to get worse. I imagine it didn't start out pushing 1000 URLs a day... it snowballed over time. It will continue to do so, and you're only looking at it now because it's not scaling. It won't scale; it'll only get worse.

If you think you have an obligation here, you may have simply fallen into the common fallacy that Google is the only search engine. The complainers don't expect you to tidy up on Bing, Yahoo, DuckDuckGo, Yandex, Baidu, Bojeek, Qwant, Unbubble, Mail.ru, Referat, Ecosia, etc. etc. Obviously, the idea that you need to go to all of them is laughable. What, search your web logs to determine which spiders have seen the offending page? It's just not workable, and any complainers who expect that are out of line to demand it. They won't get far in a lawsuit, because their own lawyer will tell them that.

The bonus service of adding the URL into Search Console is a limited commodity with value to your company, that you don't owe them. And it robs from your company's ability to manage your own URLs as the tool intended. As such, you could charge this as a premium, and even set up a UX for it: when they submit their list of URLs, they could tick a box and provide their credit card details, and charge them $5 a URL to load them into Search Console (limit 100 per day). I don't think you'd get many takers, since as I say, they're already hitting Google themselves. However even this is not a scalable product since if you get 1001 requests in a day, you can't make good on it.

2
  • 2
    They don't say there the removals are DMCA related. DMCA would be one use case that could lead to a large number of removal requests, but there are other possible reasons as well. Sep 29 at 17:15
  • @StephenOstermiller sorry, I wore an ISP hat for too long. We use DMCA as a slang to describe all of them, since most are DMCAs, and it makes no difference to us LOL. Sep 29 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.