Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
My website has an Admin area which requires a login. There are also other pages that should not be indexed. I know that Robots.Txt entries or meta name="robots" content="noindex" or an X-Robots-Tag: noindex are intended to inform the SE. There are, however, bad actors that ignore this and crawl the pages and links in spite of the commands.
I am wondering whether returning a 403 or 404 would prevent indexing?
As John said above, not only will it not prevent bad actors from indexing your admin pages, but this strategy can confuse your users and have other unintended consequences.
All of the ways you listed to keep the page out of most mainstream search engines' indexes are valid. And it's the mainstream ones you should be concerned with anyway, as Google, Bing, DuckDuckGo, etc. have the lion's share of internet searches. The bots that ignore these directives are probably not going to be the ones bringing you serious human visitor numbers.
Given that bad actors will do as they please, the best thing you can do is ensure that your cybersecurity practices are modern and effective, especially for admin and login areas.
You state that the admin area requires a login - assuming this is working correctly crawlers should not be able to access those pages and index them.
If a visitor attempts to access a secure URL without authenticating you have a couple of options:
If the resource is a web page, expecting to be accessed by web browsers, you would normally either issue a temporary redirect to a login page, that page would typically be set to no-imdex and shouldn't cause an issue, ideally this would have a mechanism to return the user to the desired page after they've logged in, or you could issue a 401 - Unauthorized response with the login form in-situ, and again ensure no-index tags are set.
Generally if you're trying to mitigate against bad crawlers, a redirect is likely to be safer, your server won't be returning any other indexable content.
If the resource is on an API, you've got some other options:
If the visitor isn't authenticated, a 401 - Unauthorized is appropriate, if they are authenticated, but don't have rights then a 403 forbidden would be correct.
