If you literally only have a few "groups" you want to block then you would do something like:
User-agent: *
Disallow: /lang/group1
Disallow: /lang/group2
...and everything else would be allowed. This would work with all robots that obey the original "standard". Or, you could block all groups (group1
, group2
, etc.) and make an exception for "group3", like:
User-agent: *
Disallow: /lang/group
Allow: /lang/group3
Note that the Allow
directive is not part of the original "standard", but has universal support. The URL path is simply a prefix.
HOWEVER, I wouldn't use robots.txt to block the pages being "crawled". What about stray visitors? And bad bots? And robots.txt doesn't prevent pages from being indexed if they are inadvertently linked to. I would use .htaccess or your server config to actually block all traffic to these URLs. Something like the following in .htaccess:
RewriteEngine On
RewriteRule ^lang/group[12] - [R=404]
To respond with a 404 for all requests to these invalid URLs. Or, use the F
flag to respond with a 403 "Forbidden".