I'm fairly new to robots.txt but have the basics. My first problem is that I'd like to disallow all the translated pages, except in the help file.

So I have this:

Disallow: /de/

But I want, at the same time, to ALLOW one subfolder of the translations, namely:

Allow: /de/help/

How can I show this in robots.txt?

1 Answer 1


You would have to explicitly allow the subfolder you want crawled, while disallowing the rest of the folder. So something like:

User-agent: * 
Allow: /de/help/ 
Disallow: /de/

This will tell all search engines that you're allowing this one subfolder (and anything in it, including other subfolders - you're allowing a path, essentially) but otherwise disallowing everything else in this folder. Best to include the Allow directive above Disallow.

On the subject of robots.txt, more info from Google here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.