Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?

2 Answers 2

User-agent: *
Disallow: /
Allow: /lessons/
Allow: /other-dir/

This does disallow the entire website, but explicitly allows given directories.

  • 1
    You should be careful with this as "Allow" is an unofficial 'extension' to the robots.txt standard. All the major search engines support it I believe, but minor ones or other services may not and ignore the entire site. Jul 30, 2011 at 15:14
  • 1
    and you need to put the Allow first than the Disallow
    – user18830
    Oct 2, 2012 at 18:39

You must list the Allow lines first as the file is read on first match basis.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

Reference: http://www.robotstxt.org/norobots-rfc.txt

Google provides a tool in webmaster tools for testing your file. I always recommend testing your file. See "Test a site's robots.txt file:" near bottom.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.