5

I tend to generate the sitemaps for my static sites (no CMS) by using a service such as xml-sitemaps.com. They way this works if you input your homepage and it crawls all the internal links of a site and then generates you a .xml file.

The problem with this is because its crawling from the homepage if a page if not linked to i.e. a 404 page it won't be included in the sitemap.

Is it good practice to have the 404 page in there?

Improve this question
1
  • Sounds like there is a bug in that site! – DisgruntledGoat Jun 14 '13 at 17:08
6

Sitemap.xml helps your website to be well and fast indexed. By submitting this file to search engines, I don't see any good reason to ask them to index a page that doesn't exist (404 HTTP status).

Improve this answer
6
  • 1
    Why wouldnt 404 exist ? It is a page /404.hmtl (although there is no reason to go there apart from in the incident of an error) – sam Jun 14 '13 at 14:00
  • Ok for a custom 404 webpage but not for URL that gets a webpage which doesn't exist. – Zistoloen Jun 14 '13 at 14:05
  • @sam: You seem to have answered your own question by your comment, "there is no reason to go there". So why index it as a possible soft-404? A custom error document probably should not be accessible directly over HTTP anyway, since by itself it probably does not contain anything useful. – MrWhite Jun 14 '13 at 14:17
  • 4
    @sam if a page is not found you should not be redirecting to the 404 page, the server should just show the contents of 404.html but keep the URL as example.com/missingpage.html – DisgruntledGoat Jun 14 '13 at 17:09
  • @DisgruntledGoat should i set header as 404 or 200 and show them 404 content ? – user43396 Aug 2 '15 at 5:39
0

keep your 404 links in sitemap and update it's lastmod or create a separate sitemap for 404 links to be removed faster with natural google bot crawl. this document is good Q&A about that

Google Support Q&A

Improve this answer
-4

I would say yes but not because you want them indexed but instead removed from the search results.

If you add the required noindex tag or HTTP header to those 404 pages, you will be informing the search engine (Google in this case) that those links should lose their place in the search results.

In Google's words:

When Googlebot next crawls that page and see the tag or header, Googlebot will drop that page entirely from Google Search results, regardless of whether other sites link to it.

Source: https://support.google.com/webmasters/answer/93710?hl=en

Improve this answer
3
  • how does someone add a noindex tag to a page that doesn't exist? – Steve Mar 2 '18 at 6:11
  • to the 404 page/template – mmttato Mar 3 '18 at 21:29
  • Don't bother adding noindex to 404 pages. It tells Google "Please don't index this page that doesn't exist," which is nonsensical. – Maximillian Laumeister May 18 '20 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.