1

I am confronted with a strange setup I never seen before - and I don't know, how to deal with it.

Details:

There is a bunch of no longer existing pages. All of them have

  • Status code 404,
  • Their previous URL,
  • Meta Robots: noindex,
  • Canonical: https://example.com/static/404. This is the custom Error Document from website setup. All 404 pages have this canonical.

These pages are listed in GSC - this moved me to look into their setup.

In my opinion, Meta Robots and Canonical should be removed. What would you think?

4
  • Does the "canonical URL" itself (ie. /static/404) also return a 404 HTTP status? What do you mean by "All of them have their previous URL"? You mean the URL stays in the address bar and is not redirected?
    – MrWhite
    Aug 9, 2023 at 14:58
  • 1
    Yes, the canonical URL itself return 404 and has self-referencing canonical. "All of them have their previous URL" means, a page has an URL like www.example.com/cat1/article-about-cats.html and status code 200. After some time the page becomes unavailable and has now status code 404. But the url of the page remains www.example.com/cat1/article-about-cats.html.
    – Evgeniy
    Aug 9, 2023 at 15:09
  • But presumably the content of the now 404 response is a "Not Found" page, not the original content just with a 404 status? (Otherwise that is confusing.)
    – MrWhite
    Aug 9, 2023 at 15:11
  • 1
    yes, the content changes: from the original content, as it was with status code 200, to the content of static/404.
    – Evgeniy
    Aug 9, 2023 at 15:12

2 Answers 2

3

In my opinion, Meta Robots and Canonical should be removed. What would you think?

The meta robots (noindex) tag is unnecessary, but it can stay. Providing the response has a 404 HTTP status then search engines will not index it anyway.

However, the rel="canonical" link element is in error and should be removed. (My guess is that this is perhaps a side effect of the CMS/framework?) For a 404 response there is no canonical URL - it just doesn't exist.

The 404 error document itself should not be a public (or publicised) URL. This should be hidden from users and search engines.

The concern here is whether direct requests to the error document itself (ie. /static/404) actually return a "200 OK" response, not a 404 (as it should).

2

The definition of 404 is

404 Not Found
The requested resource could not be found but may be available in the future. Subsequent requests by the client are permissible.

So you should be generating 410

410 Gone
Indicates that the resource requested was previously in use but is no longer available and will not be available again. This should be used when a resource has been intentionally removed and the resource should be purged. Upon receiving a 410 status code, the client should not request the resource in the future. Clients such as search engines should remove the resource from their indices. Most use cases do not require clients and search engines to purge the resource, and a "404 Not Found" may be used instead.

While external links exist to those pages, I think, GSE will keep listing them.

1
  • Such urls, with status code 410, are existing too. They have same setup: noindex and canonical to static/404.
    – Evgeniy
    Aug 9, 2023 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.