I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization.
Does that mean I need to have a robots.txt file to Disallow these pages with authorization?
Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.Sign up to join this community
You don't have to block those pages via robots.txt, but it is still a good idea to do so. You never know if an error on your part, or changes in how crawlers deal with authentication, will result in crawlers being able to successfully find and crawl those pages in the future. And since explicitly blocking those pages is very easy to do, it is never a bad idea to block them via robots.txt (and x-robots-tag, too).
Depending on the website, its content, and how it's linked, disallowing crawling may be something to watch out for. By disallowing crawling, Google won't be able to see that the content requires authentication. This means that it may end up indexing the URLs and showing them in search (based on things like links to those URLs). So if having those URLs visible in search is a problem, then I'd recommend not blocking them in the robots.txt file. On the other hand, if having those URLs visible in search is helpful (eg if you know that your users are searching for your content), then disallowing them via robots.txt might be a possibility (better would be to just have an indexable landing page, and to put the rest behind authentication.)