2

We have a website where we have a round 30 pages that are only visible for authorized users. in Google WMT we see errors appearing about this that Google bot can't read this.

Do we have to add them into our robots file to not let them be indexed or can we leave it this way?

1

You can do either. There is no harm in 401 error notices. They are just notices for your benefit.

I would still advise, if possible, adding them to your robots.txt file so that Google does not look for them. I assume these can easily be identified by the URL.

Others allow Google to index these pages to encourage new users, however, from a user experience (UX) point of view, I am not convinced this is a good thing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.