Google Webmaster Tools is reporting 404 errors on URLs that are not present in the submitted sitemap anymore. These URLs were present but since then I have resubmitted my sitemap and GWT shows that it was, indeed, crawled again after the change.

Is there anything I can do to stop Google from indexing these URLs without manually excluding them? There are quite a lot of them so I would love to be able to avoid doing it.

  • 1
    possible duplicate of How to effectively close a page? – Zistoloen Jul 14 '13 at 12:57
  • 1
    Look at my answer on this topic, it answers to your need. – Zistoloen Jul 14 '13 at 12:58
  • @Zistoloen, thank you for the answer. It does help a bit, I have been thinking about using 301s instead of 410s (the pages are not gone, their URLs changed). However, I hoped for a possible solution to the problem of Google crawling pages that are not in sitemaps anymore. But I guess that removing something from a sitemap does not stop Google from trying to access it. – wssbck Jul 14 '13 at 13:04
  • 1
    Google follows links and maybe there are still links to these webpages. In that case, apply a 301 redirects and Google will remove these webpages as 404. – Zistoloen Jul 14 '13 at 13:07
  • 1
    Just wait a little bit and Google will remove them. – Zistoloen Jul 14 '13 at 13:13

Once Google crawls a URL, it will continue to crawl that URL forever. It doesn't matter if it is no longer in your sitemap. It doesn't even matter if you never had a page there at all. If Google found a link to it at some point in the past, you may see it in your 404 report on an ongoing basis.

Google's John Mueller said that 404 errors don't hurt your site's indexing and ranking. There is really no reason to worry about 404 errors in this report unless you expected those pages to be found.

If you want to clean up that report you can always redirect the URLs to other similar pages, or return the 410 Gone status.

| improve this answer | |
  • Thanks, this is what I have done. I used 301s in the end, since these were the kind of redirects that fitted the situation best. – wssbck Jul 26 '13 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.