Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
2 Answers
At the very bottom of the page.
To exclude all files except one
This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory.
@Alex answer is partially correct. I subsequently discovered this:
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive