Pages that return a 404 not found or 410 Gone code are not indexed by search engines.
From Google webmasters center:
This HTTP response code clearly tells both browsers and search engines
that the page doesn’t exist. As a result, the content of the page (if
any) won’t be crawled or indexed by search engines.
We recommend that you always return a 404 (Not found) or a 410 (Gone)
response code in response to a request for a non-existing page.
More info can be found here
Now if you want to remove pages from Google there are 2 techniques that you can use.
- Use error code 404 or 410
- Use noindex meta tag
Both the techniques error codes or noindex tag, are useful but for other purpopses.
If you have pages that exist, but you dont want them to be
indexed eg privacy.html, you should add the noindex meta tag
<meta name="robots" content="noindex">
The next time that Google will crawl these pages, it will remove
them from the search results. More info about how Google treats
noindex meta tag can be found here
If you have pages that existed in the past and are no longer existing (eg. you deleted them), you
should return a 404 or 410 error code and not use the noindex meta
tag. No index meta tag, will tell Google that the page exists, but I
dont want it to be indexed
The html that you return along with the error code does not play a
role, what it is important is the error code that you return.
Google will then recheck the page and will see that the code is not
200 but it is 404 or 410, and will drop the page from the indexes.
Note that Google must have to recrawl a page that returns 404 or 410
several times before dropping the page completely from the index.
This is done to prevent a webmaster from doing an error and return
wrong codes accidentantly, eg database malfunction.
To sum up:
- On deleted pages or urls that never existed, you should return a 404 or
410 error code and not using the meta noindex tag, and wait until Google drops them from the search results.
- On existing pages
that you do not want them to be indexed, you should use the meta