1

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?

2 Answers 2

4

At the very bottom of the page.

To exclude all files except one

This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory.

1

@Alex answer is partially correct. I subsequently discovered this:

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.