Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?

share|improve this question
User-agent: *
Disallow: /
Allow: /lessons/
Allow: /other-dir/

This does disallow the entire website, but explicitly allows given directories.

share|improve this answer
You should be careful with this as "Allow" is an unofficial 'extension' to the robots.txt standard. All the major search engines support it I believe, but minor ones or other services may not and ignore the entire site. – DisgruntledGoat Jul 30 '11 at 15:14
and you need to put the Allow first than the Disallow – user18830 Oct 2 '12 at 18:39

You must list the Allow lines first as the file is read on first match basis.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

Reference: http://www.robotstxt.org/norobots-rfc.txt

Google provides a tool in webmaster tools for testing your file. I always recommend testing your file. See "Test a site's robots.txt file:" near bottom.


share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.