404 page vs. 404 server answer

Question

Could me somebody explain following topic:

On different sites i realized two scenarios, how not existing urls are handled. I'm not sure, which setup causes which scenario and which scenario should be preferred and why.

These scenarios are: on accessing of no longer existing url

• the url remains the same, but the page has the status code 404,
• an url changes to /404.html

Other differences are in what kind of 404 page is displayed. It can be a custom 404 page, which was setup with Error Document, otherwise it can be a default server 404 page or a default 404 page of CMS (if any in use).

The question is: which setup details cause the cases i listed above?

Answer 1

the url remains the same, but the page has the status code 404,

This is certainly the preferred option, as Martijn suggests. For a "custom" error page on Apache (ignoring the CMS option for a moment), this could be achieved with the ErrorDocument directive:

ErrorDocument 404 /errors/my404.php

it can be a default server 404 page

Using the default keyword clears any custom URL that might have been set and results in the servers default error document being served.

ErrorDocument 404 default

or a default 404 page of CMS

Since a CMS is often serving pages defined in a database, it's not possible to use the ErrorDocument directive to handle 404s for pages defined (or not) in the CMS, since whether a request results in a 404 is only known much later in the request. The CMS detects that the requested URL does not exist and manually serves a 404 Not Found header with some appropriate page content.

an url changes to /404.html

For the URL to change there must be some kind of external redirect. Most probably a 3xx status code. To me, this would suggest a misconfiguration. The user-agent first sees the 3xx status code, not a 404. The document being redirected to may or may not then serve a 404 (it should). The user-agent ends up issuing 2 requests (because of the redirect). Providing the end-point returns a 404 status it is still effectively a 404.

Other caveats with issuing a redirect to the error document is that you lose the original URL (to provide a meaningful message to your user) and other details (server variables) related to the error. (This information would need to be manually passed somehow if it was required.)

There is rarely a need to redirect to a 404. On Apache, this is possible again with the ErrorDocument directive. If you specify an absolute URL (with protocol and hostname) as the error document argument then Apache issues a 302 (temporary) redirect (regardless of whether the URL actually resides on the same server):

ErrorDocument 404 http://example.com/errors/my404.php

This is primarily for redirecting to an external resource (should that be a requirement). But the same caveats with issuing a redirect apply, as mentioned above.

Answer 2

The TLDR; answer: Send out a 404 header, don't change the url.

The longer answer:
There are more options then you suggest, which I'm not going to list here (because it's not really needed).

The most basic solution is just to serve a 404 header when the page does not exist. The reason for this is simple; This is what visitors expect. You open a URL and that either works or it doesn't.

Redirects to other pages are possible, but this should only be done when there is relevent content on the new page (do not redirect everything to home!). A better solution would be a custom page, with a few suggestions for your visitor, along the lines of:

"Oops! Page not found. Maybe you're looking for X, or perhaps Y? You can also go to Home"