Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
We have a website where we have a round 30 pages that are only visible for authorized users. in Google WMT we see errors appearing about this that Google bot can't read this.
Do we have to add them into our robots file to not let them be indexed or can we leave it this way?
You can do either. There is no harm in 401 error notices. They are just notices for your benefit.
I would still advise, if possible, adding them to your robots.txt file so that Google does not look for them. I assume these can easily be identified by the URL.
Others allow Google to index these pages to encourage new users, however, from a user experience (UX) point of view, I am not convinced this is a good thing.
