5

I have a custom 404.html file on my website that gets served with a 404 status code when a URL is not found, standard stuff. But if you visit the exact path /404.html on the server, the server says "hey, there's actually a file there" and shows you the 404 page but without the 404 status code.

I wanted to make sure that the /404.html path doesn't get accidentally indexed in search, so I added a <meta name="robots" content="noindex"> tag to it.

Of course, that means that my nonexistant URLs also have the meta noindex tag as part of the 404 page. Is there any way that the presence of this meta noindex tag could cause an issue for search engines / crawlers, or is this setup totally fine?

  • Alternately you could name the file /404-3pAhD3tM.html which would be unlikely to ever to get crawled unless you linked to it or had directory indexes on. – Stephen Ostermiller Jan 30 at 22:32
  • I also suspect that if Google did crawl /404.html it would treat it as a "soft 404" because it presumably has "Page Not Found" in the title and H1. – Stephen Ostermiller Jan 30 at 22:34
6

There's no problem having a meta noindex tag on the 404 page, to prevent 200 OK responses being indexed.

If this was a PHP page then you could obviously just send a 404 Not Found header as part of the standard response - to make sure that it always returns a 404.

You could also use .htaccess (mod_rewrite) to force any direct requests to 404.html to also trigger a 404:

RewriteEngine On
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^404\.html$ - [R=404]

The RewriteCond directive that checks the REDIRECT_STATUS environment variable is required if you want your custom 404 to be served (so that it only applies to direct requests). Otherwise, there will likely be a rewrite loop (500 error) when trying to serve the custom error document and you'll see the standard server generated 404 response (but it's still a 404).

  • 1
    Thank you for the good ideas. For my hosting setup it's easiest to just leave the meta noindex in place, so it's good to know it won't cause any issues if I do. – Maximillian Laumeister Jan 30 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.