I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are
fr/fr. Other language combinations must be restricted. On the other hand crawler should be able to crawl
/about, etc. For example:
example.com/us/en/videos # should be allowed example.com/de/en/videos # should be blocked example.com/users/mark # should be allowed
Again, it should be blocked only if it starts with unaccepted region/language combinations. What I did so far does not work:
Disallow: /*? Disallow: /*/*/cart/ Disallow: /*/*/checkout/ Disallow: /*/*/ Allow: /*.css? Allow: /*.js? Allow: /us/en/ Allow: /gb/en/ Allow: /fr/fr/
I tested it with google's online robots.txt tester.