Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?

share|improve this question
add comment

migrated from stackoverflow.com Sep 19 '11 at 13:11

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 3 down vote accepted

At the very bottom of the page.

To exclude all files except one

This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory.

share|improve this answer
add comment

@Alex answer is partially correct. I subsequently discovered this:

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.