8

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?

Improve this question

2 Answers 2

Reset to default
6 
User-agent: *
Disallow: /
Allow: /lessons/
Allow: /other-dir/

This does disallow the entire website, but explicitly allows given directories.

Improve this answer
2
  • 1
    You should be careful with this as "Allow" is an unofficial 'extension' to the robots.txt standard. All the major search engines support it I believe, but minor ones or other services may not and ignore the entire site.
    – DisgruntledGoat
    Jul 30, 2011 at 15:14
  • 1
    and you need to put the Allow first than the Disallow
    – user18830
    Oct 2, 2012 at 18:39
2

You must list the Allow lines first as the file is read on first match basis.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

Reference: http://www.robotstxt.org/norobots-rfc.txt

Google provides a tool in webmaster tools for testing your file. I always recommend testing your file. See "Test a site's robots.txt file:" near bottom.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.