Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I'm fairly new to robots.txt but have the basics. My first problem is that I'd like to disallow all the translated pages, except in the help file.
So I have this:
Disallow: /de/
But I want, at the same time, to ALLOW one subfolder of the translations, namely:
Allow: /de/help/
How can I show this in robots.txt?
You would have to explicitly allow the subfolder you want crawled, while disallowing the rest of the folder. So something like:
User-agent: *
Allow: /de/help/
Disallow: /de/
This will tell all search engines that you're allowing this one subfolder (and anything in it, including other subfolders - you're allowing a path, essentially) but otherwise disallowing everything else in this folder. Best to include the Allow directive above Disallow.
On the subject of robots.txt, more info from Google here.
