5

I tend to generate the sitemaps for my static sites (no CMS) by using a service such as xml-sitemaps.com. They way this works if you input your homepage and it crawls all the internal links of a site and then generates you a .xml file.

The problem with this is because its crawling from the homepage if a page if not linked to i.e. a 404 page it won't be included in the sitemap.

Is it good practice to have the 404 page in there?

2
  • Sounds like there is a bug in that site! Jun 14, 2013 at 17:08
  • Google will assume pages listed in your sitemap are canonical. So it's definitely going to pad your 404 page's ego... Jan 4 at 3:43

4 Answers 4

6

Sitemap.xml helps your website to be well and fast indexed. By submitting this file to search engines, I don't see any good reason to ask them to index a page that doesn't exist (404 HTTP status).

7
  • 1
    Why wouldnt 404 exist ? It is a page /404.hmtl (although there is no reason to go there apart from in the incident of an error)
    – sam
    Jun 14, 2013 at 14:00
  • Ok for a custom 404 webpage but not for URL that gets a webpage which doesn't exist.
    – Zistoloen
    Jun 14, 2013 at 14:05
  • 1
    @sam: You seem to have answered your own question by your comment, "there is no reason to go there". So why index it as a possible soft-404? A custom error document probably should not be accessible directly over HTTP anyway, since by itself it probably does not contain anything useful.
    – MrWhite
    Jun 14, 2013 at 14:17
  • 4
    @sam if a page is not found you should not be redirecting to the 404 page, the server should just show the contents of 404.html but keep the URL as example.com/missingpage.html Jun 14, 2013 at 17:09
  • @DisgruntledGoat should i set header as 404 or 200 and show them 404 content ?
    – user43396
    Aug 2, 2015 at 5:39
0

keep your 404 links in sitemap and update it's lastmod or create a separate sitemap for 404 links to be removed faster with natural google bot crawl. this document is good Q&A about that

Google Support Q&A

0
  • Allowing your 404 pages to be indexed in search engines (make sure that your web server is configured to give a 404 HTTP status code or—in the case of JavaScript-based sites—include the noindex tag when non-existent pages are requested).
  • Blocking 404 pages from being crawled through the robots.txt file.
  • Providing only a vague message like "Not found", "404", or no 404 page at all.
  • Using a design for your 404 pages that isn't consistent with the rest of your site.

check the google search document:

https://developers.google.com/search/docs/beginner/seo-starter-guide

0
-4

I would say yes but not because you want them indexed but instead removed from the search results.

If you add the required noindex tag or HTTP header to those 404 pages, you will be informing the search engine (Google in this case) that those links should lose their place in the search results.

In Google's words:

When Googlebot next crawls that page and see the tag or header, Googlebot will drop that page entirely from Google Search results, regardless of whether other sites link to it.

Source: https://support.google.com/webmasters/answer/93710?hl=en

3
  • how does someone add a noindex tag to a page that doesn't exist?
    – Steve
    Mar 2, 2018 at 6:11
  • to the 404 page/template
    – mmttato
    Mar 3, 2018 at 21:29
  • Don't bother adding noindex to 404 pages. It tells Google "Please don't index this page that doesn't exist," which is nonsensical. May 18, 2020 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.