Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Using robots.txt, how do you disallow the root of a site (http://www.example.com/) but allow a subdirectory (http://www.example.com/lessons/)?

share|improve this question

2 Answers 2

User-agent: *
Disallow: /
Allow: /lessons/
Allow: /other-dir/

This does disallow the entire website, but explicitly allows given directories.

share|improve this answer
1  
You should be careful with this as "Allow" is an unofficial 'extension' to the robots.txt standard. All the major search engines support it I believe, but minor ones or other services may not and ignore the entire site. –  DisgruntledGoat Jul 30 '11 at 15:14
1  
and you need to put the Allow first than the Disallow –  user18830 Oct 2 '12 at 18:39

You must list the Allow lines first as the file is read on first match basis.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

Reference: http://www.robotstxt.org/norobots-rfc.txt

Google provides a tool in webmaster tools for testing your file. I always recommend testing your file. See "Test a site's robots.txt file:" near bottom.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.