Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization.

Does that mean I need to have a robots.txt file to Disallow these pages with authorization?

share|improve this question
up vote 2 down vote accepted

You don't have to block those pages via robots.txt, but it is still a good idea to do so. You never know if an error on your part, or changes in how crawlers deal with authentication, will result in crawlers being able to successfully find and crawl those pages in the future. And since explicitly blocking those pages is very easy to do, it is never a bad idea to block them via robots.txt (and x-robots-tag, too).

share|improve this answer
Blocking them might also cut down on the amount of work that your web server would be doing otherwise. – dan Jan 14 '14 at 3:17

Depending on the website, its content, and how it's linked, disallowing crawling may be something to watch out for. By disallowing crawling, Google won't be able to see that the content requires authentication. This means that it may end up indexing the URLs and showing them in search (based on things like links to those URLs). So if having those URLs visible in search is a problem, then I'd recommend not blocking them in the robots.txt file. On the other hand, if having those URLs visible in search is helpful (eg if you know that your users are searching for your content), then disallowing them via robots.txt might be a possibility (better would be to just have an indexable landing page, and to put the rest behind authentication.)

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.