Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
migrated from stackoverflow.com Sep 19 '11 at 13:11
|
At the very bottom of the page.
|
|||
|
|
|
@Alex answer is partially correct. I subsequently discovered this: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive |
|||
|
|