Restrict crawling of region/lang combinations other than the provided ones in robots.txt

Question

I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other hand crawler should be able to crawl / or /about, etc. For example:

example.com/us/en/videos  # should be allowed
example.com/de/en/videos  # should be blocked
example.com/users/mark    # should be allowed

Again, it should be blocked only if it starts with unaccepted region/language combinations. What I did so far does not work:

Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css?
Allow: /*.js?
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

I tested it with google's online robots.txt tester.

Answer 1

Assuming you have the User-agent: * directive on the first line then what you've posted appears to work as intended in "Google's online robots.txt tester". In that it satisfies the 3 (or more) criteria you've stated:

1. example.com/us/en/videos IS allowed
2. example.com/de/en/videos IS BLOCKED
3. example.com/users/mark IS allowed
4. example.com/ IS allowed
5. example.com/about IS allowed
6. example.com/us/en/ (with or without the trailing slash) IS allowed
7. example.com/gb/en/ (with or without the trailing slash) IS allowed
8. example.com/fr/fr/ (with or without the trailing slash) IS allowed

As per the screenshots below...

1. example.com/us/en/videos

2. example.com/de/en/videos

3. example.com/users/mark

4. example.com/

5. example.com/about

Answer 2

I haven't seen the ? character in a robots.txt file before. Are you sure you don't mean to use $ instead, which denotes the end of the string?

Furthermore, your file needs to specify user-agent at the top for which bots to affect. Here's a revised file:

User-agent: *
Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css$
Allow: /*.js$
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

This is closer to what you want. To affect only Googlebot, change User-agent: * to User-agent: Googlebot.

But it does seem like different robots.txt checking tools read it differently. Google's tool blocks /us/en/cart/, for instance, while technicalseo's tool and seobook.com's tool both allow the same URL. Not sure how to reconcile that one.