Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've asked a website to remove content about me and they did it, removing the whole page (404). In their sitemap, they tell bots that page changes weekly.

A week ago they removed the page, but the result still shows up in Google SERP. How is it possible?

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Google's indexing behavior is optimized for quickly finding new pages, not so much for noticing the disappearance of old pages. Generally, this makes sense — when you search Google, you want to find something, so not finding a new page that has just appeared is normally a much bigger failure than occasionally being sent to a page that used to be there, but isn't any more. Besides, most popular web pages don't usually disappear, and if they do, that's often just a temporary issue caused by a server hiccup.

In particular, Google will normally retain removed pages in its index until:

  1. it happens to re-crawl the page and notices that it returns a 404 response, and

  2. this happens sufficiently many times in a row that Google's algorithm decides that the page probably isn't coming back.

Thus, it's not uncommon for removed pages to linger for weeks or even months in Google's index, especially if the site they were on is relatively low-ranking and therefore infrequently re-crawled by Google.

In those rare cases where you do want to have a page quickly removed from Google's index, there are a couple of things you can do:

  • You can ask the owner of the site to submit a removal request for the page via Google's Webmaster Tools.

  • If the page has already been removed from the site, you can use the Google public URL removal tool yourself to tell Google to recrawl the page and remove it from their index.

Both of the above methods should result in the page being removed from Google's index within a day or so. Note that, if the page has been indexed under multiple URLs, you'll need to file removal requests for all of them.

For completeness, as John Conde suggests in his answer, you can ask the webmaster of the site the page was on to configure their web server to return an 410 Gone response for the page instead of the usual 404. This will not make Google re-crawl the page any faster, but when it does, the 410 reponse does tell Google that the page probably isn't coming back.

Also, don't forget that Google isn't the only search engine out there. To get the content completely off the web, you'll also need to file removal requests with Bing, Yandex and any other search engines that may have indexed it. (In some cases, you may be out of luck; for example, the only URL removal instructions I could find for Baidu involve sending e-mail to a helpdesk address.) And don't forget archives like the Wayback Machine, either; while such archives are not searchable, someone who knew where the page used to be might still be able to find an archived copy. It's up to you to decide whether or not you consider that an issue.

share|improve this answer
    
Great answer. Every time I ask a question on this network, I'm astonished by the great contents users provide. Keep up the good work! –  Surfer on the fall Mar 9 at 16:29

It takes a while for Google's index to catch up. They have to re-crawl the page (which may not have happened yet) and get the 404 response. But they won't act on that immediately because that may be a temporary error. Only after seeing the page is gone for good will they remove it from their index.

They may get faster results if they send a 410 GONE response which says the page removal is on purpose and permanent.

share|improve this answer
    
Thanks for the tip, even if it isn't my website. –  Surfer on the fall Mar 5 at 19:17
1  
Also, the site owner can in Google webmaster tools have Google fetch the specific page. This may speed things up. Keep in mind that under normal circumstances it can take 30-60 days for a page to be removed from the index. Having Google fetch the page may reduce it to just weeks. Of course, you cannot speed up Google so patience is required. Sorry. –  closetnoc Mar 5 at 19:29

In their sitemap, they tell bots that page changes weekly.

Also, the change frequency in the sitemap is merely a hint to the search engine. There is no requirement for them to respect that - they can still come along every day, or they can skip a week.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.