4

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization.

Does that mean I need to have a robots.txt file to Disallow these pages with authorization?

2

You don't have to block those pages via robots.txt, but it is still a good idea to do so. You never know if an error on your part, or changes in how crawlers deal with authentication, will result in crawlers being able to successfully find and crawl those pages in the future. And since explicitly blocking those pages is very easy to do, it is never a bad idea to block them via robots.txt (and x-robots-tag, too).

| improve this answer | |
  • 1
    Blocking them might also cut down on the amount of work that your web server would be doing otherwise. – dan Jan 14 '14 at 3:17
2

Depending on the website, its content, and how it's linked, disallowing crawling may be something to watch out for. By disallowing crawling, Google won't be able to see that the content requires authentication. This means that it may end up indexing the URLs and showing them in search (based on things like links to those URLs). So if having those URLs visible in search is a problem, then I'd recommend not blocking them in the robots.txt file. On the other hand, if having those URLs visible in search is helpful (eg if you know that your users are searching for your content), then disallowing them via robots.txt might be a possibility (better would be to just have an indexable landing page, and to put the rest behind authentication.)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.