I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en
, gb/en
and fr/fr
. Other language combinations must be restricted. On the other hand crawler should be able to crawl /
or /about
, etc. For example:
example.com/us/en/videos # should be allowed
example.com/de/en/videos # should be blocked
example.com/users/mark # should be allowed
Again, it should be blocked only if it starts with unaccepted region/language combinations. What I did so far does not work:
Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css?
Allow: /*.js?
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/
I tested it with google's online robots.txt tester.
User-agent: *
line then what you've posted works as intended in "Google's online robots.txt tester". If you've missed the user-agent line then the tool reports a stack of errors: "No user-agent specified". (?) – MrWhite Feb 23 '20 at 0:23/us/en/videos
IS allowed,/de/en/videos
IS blocked and/users/mark
IS allowed. (?) – MrWhite Feb 24 '20 at 20:54/*/*/
rule is the reason of blocking. – Eziz Durdyyev Feb 24 '20 at 22:30