3

I've added 404 error codes and no-index meta tag to pages quite some time ago. However, when I check in Google Webmaster Tools, looking at the graph of "removed" pages from their index, I see very low numbers that don't correspond with the amount of error codes and no-index tags I have on my pages. I can also reach those pages through Google Search.

I've double checked the implementation and it seems fine. Can there be another reason?

3
  • Is it possible that google did not reindex the site yet?
    – ek9
    Commented Mar 26, 2014 at 13:07
  • Presumably the pages with NOINDEX meta tags are still crawlable and not blocked by robots.txt etc?
    – MrWhite
    Commented Mar 26, 2014 at 23:22
  • @w3d not blocked by robots.txt only 404 or NOINDEX meta tag
    – Noam
    Commented Mar 27, 2014 at 10:39

4 Answers 4

4

Days, Weeks and Months...

It can take Google days, weeks and even months for Google to remove pages marked noindex, robots and 404's. Generally it takes Google several crawls before Google acts on the new information of a page.

User Errors

It should be also noted that more than often users make human errors and create 404's, noindexs and so forth by error, simply removing the pages straight away would cause harm in rankings to such little mistakes, so It's my belief Google does this on purpose to ensure that these are intended changes.

Web Master Tools

Over the years many webmasters have said exactly the same as you over and over, and more times than I can count. Google made changes to Webmaster Tools several years ago allowing webmasters to quickly remove a URL by making a request. If you have a few pages that you want removed straight away then you should make a request, this is by far the quickest method and you should ensure that the pages return noindex if the page exists or a 410 gone to prevent it being re-indexed or denied removal.

410 vs 404

While a 404 is perfectly normal on a small scale it should be noted that 410 is actually preferred, this tells Google the page is GONE FOREVER! while a 404 could be used temporary, or if you have just a few pages that your not fussed about.

3
  • So 404 won't be removed or just not as certain as 410? (404 best describes the current states of the page, but I would want him to be removed from the index for now)
    – Noam
    Commented Mar 27, 2014 at 10:41
  • Both do the same, just 410 means its never ever coming back while a 404 can mean its gone, or will return at some point. For quick removal you should use Webmaster Tools as well as noindex. Commented Mar 27, 2014 at 14:57
  • Frustrating,i tried with 410 in .htaccess,tried with "noindex" meta,process is slow as snail. Commented Sep 17 at 7:59
2

If you marked those pages with a noindex meta tag, there are two possible reasons I see:

  • Googlebot didn't reindex the site yet and you don't have wait enough (it can take a lot of time like several weeks, it depends on the site crawl rate)
  • You use 404 instead of 410 HTTP status (Gone)
3
  • Can you elaborate on why should I use 410 and not 404? (I'm using 404)
    – Noam
    Commented Mar 27, 2014 at 10:40
  • Unlike 404, 410 HTTP status specifies to search engines your page is gone and won't never come back. Therefore, it's just logical for search engines to remove this page from their index.
    – Zistoloen
    Commented Mar 27, 2014 at 11:39
  • And there's logic in keeping 404 in the index?
    – Noam
    Commented Mar 27, 2014 at 14:22
1

The graph you are looking at in Webmaster Tools showing 'removed' I presume that is the index status graph and if so, that shows URLs removed by manual request and not through normal 404 site maintenance.

Look under Crawl > Crawl Errors and you should see recent 404 pages.

2
  • Are you sure about this? Got any reference?
    – Noam
    Commented Mar 27, 2014 at 10:42
  • Yes I am sure, removed = pages removed, not 404 pages. You will probably find reference/more information on google's webmaster forum.
    – user29671
    Commented Mar 27, 2014 at 12:06
0

Sadly, you are taking all of the steps with the removal process. Google Webmasters also explains this issue with their YouTube video. It takes a good while for the actual removal process from all of Google and the search engine. Try to be patient now, and it will happen.

Also, you have to remember, Google is indexing, re-indexing, and deleting records from their Search Engine all of the time at a massive rate already, but there is trillions upon trillions of sites that this needs to be done to and more links for indexing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.