Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've added 404 error codes and no-index meta tag to pages quite some time ago. However, when I check in Google Webmaster Tools, looking at the graph of "removed" pages from their index, I see very low numbers that don't correspond with the amount of error codes and no-index tags I have on my pages. I can also reach those pages through Google Search.

I've double checked the implementation and it seems fine. Can there be another reason?

share|improve this question
    
Is it possible that google did not reindex the site yet? –  edvinas.me Mar 26 at 13:07
    
Presumably the pages with NOINDEX meta tags are still crawlable and not blocked by robots.txt etc? –  w3d Mar 26 at 23:22
    
@w3d not blocked by robots.txt only 404 or NOINDEX meta tag –  Noam Mar 27 at 10:39

4 Answers 4

Days, Weeks and Months...

It can take Google days, weeks and even months for Google to remove pages marked noindex, robots and 404's. Generally it takes Google several crawls before Google acts on the new information of a page.

User Errors

It should be also noted that more than often users make human errors and create 404's, noindexs and so forth by error, simply removing the pages straight away would cause harm in rankings to such little mistakes, so It's my belief Google does this on purpose to ensure that these are intended changes.

Web Master Tools

Over the years many webmasters have said exactly the same as you over and over, and more times than I can count. Google made changes to Webmaster Tools several years ago allowing webmasters to quickly remove a URL by making a request. If you have a few pages that you want removed straight away then you should make a request, this is by far the quickest method and you should ensure that the pages return noindex if the page exists or a 410 gone to prevent it being re-indexed or denied removal.

410 vs 404

While a 404 is perfectly normal on a small scale it should be noted that 410 is actually preferred, this tells Google the page is GONE FOREVER! while a 404 could be used temporary, or if you have just a few pages that your not fussed about.

share|improve this answer
    
So 404 won't be removed or just not as certain as 410? (404 best describes the current states of the page, but I would want him to be removed from the index for now) –  Noam Mar 27 at 10:41
    
Both do the same, just 410 means its never ever coming back while a 404 can mean its gone, or will return at some point. For quick removal you should use Webmaster Tools as well as noindex. –  bybe Mar 27 at 14:57

If you marked those pages with a noindex meta tag, there are two possible reasons I see:

  • Googlebot didn't reindex the site yet and you don't have wait enough (it can take a lot of time like several weeks, it depends on the site crawl rate)
  • You use 404 instead of 410 HTTP status (Gone)
share|improve this answer
    
Can you elaborate on why should I use 410 and not 404? (I'm using 404) –  Noam Mar 27 at 10:40
    
Unlike 404, 410 HTTP status specifies to search engines your page is gone and won't never come back. Therefore, it's just logical for search engines to remove this page from their index. –  Zistoloen Mar 27 at 11:39
    
And there's logic in keeping 404 in the index? –  Noam Mar 27 at 14:22

The graph you are looking at in Webmaster Tools showing 'removed' I presume that is the index status graph and if so, that shows URLs removed by manual request and not through normal 404 site maintenance.

Look under Crawl > Crawl Errors and you should see recent 404 pages.

share|improve this answer
    
Are you sure about this? Got any reference? –  Noam Mar 27 at 10:42
    
Yes I am sure, removed = pages removed, not 404 pages. You will probably find reference/more information on google's webmaster forum. –  user29671 Mar 27 at 12:06

Sadly, you are taking all of the steps with the removal process. Google Webmasters also explains this issue with their YouTube video. It takes a good while for the actual removal process from all of Google and the search engine. Try to be patient now, and it will happen.

Also, you have to remember, Google is indexing, re-indexing, and deleting records from their Search Engine all of the time at a massive rate already, but there is trillions upon trillions of sites that this needs to be done to and more links for indexing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.