Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?

User-agent: *
Disallow: /
Allow: /lessons/
Allow: /other-dir/

This does disallow the entire website, but explicitly allows given directories.

| improve this answer | |
  • 1
    You should be careful with this as "Allow" is an unofficial 'extension' to the robots.txt standard. All the major search engines support it I believe, but minor ones or other services may not and ignore the entire site. – DisgruntledGoat Jul 30 '11 at 15:14
  • 1
    and you need to put the Allow first than the Disallow – user18830 Oct 2 '12 at 18:39

You must list the Allow lines first as the file is read on first match basis.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

Reference: http://www.robotstxt.org/norobots-rfc.txt

Google provides a tool in webmaster tools for testing your file. I always recommend testing your file. See "Test a site's robots.txt file:" near bottom.


| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.