I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other hand crawler should be able to crawl / or /about, etc. For example:

example.com/us/en/videos  # should be allowed
example.com/de/en/videos  # should be blocked
example.com/users/mark    # should be allowed

Again, it should be blocked only if it starts with unaccepted region/language combinations. What I did so far does not work:

Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css?
Allow: /*.js?
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

I tested it with google's online robots.txt tester.

  • Assuming you have the User-agent: * line then what you've posted works as intended in "Google's online robots.txt tester". If you've missed the user-agent line then the tool reports a stack of errors: "No user-agent specified". (?) – MrWhite Feb 23 at 0:23
  • @MrWhite, yes I added user-agent line. but it does not work as I expect. – Eziz Durdyyev Feb 24 at 11:06
  • "it does not work as I expect" - and what do you expect? When I said it "works as intended", I mean it satisfies the 3 criteria you've stated. ie. /us/en/videos IS allowed, /de/en/videos IS blocked and /users/mark IS allowed. (?) – MrWhite Feb 24 at 20:54
  • @MrWhite, nope. when i test it at google's tool it blocks them all. and it says that /*/*/ rule is the reason of blocking. – Eziz Durdyyev Feb 24 at 22:30
  • Are you including the slash prefix on your input? – MrWhite Feb 25 at 0:58

Assuming you have the User-agent: * directive on the first line then what you've posted appears to work as intended in "Google's online robots.txt tester". In that it satisfies the 3 (or more) criteria you've stated:

  1. example.com/us/en/videos IS allowed
  2. example.com/de/en/videos IS BLOCKED
  3. example.com/users/mark IS allowed
  4. example.com/ IS allowed
  5. example.com/about IS allowed
  6. example.com/us/en/ (with or without the trailing slash) IS allowed
  7. example.com/gb/en/ (with or without the trailing slash) IS allowed
  8. example.com/fr/fr/ (with or without the trailing slash) IS allowed

As per the screenshots below...

1. example.com/us/en/videos


2. example.com/de/en/videos


3. example.com/users/mark


4. example.com/


5. example.com/about


| improve this answer | |
  • ok, now I see what is my mistake. I tested it with https://sites.google.com/site/ezizdurdy/. sorry to waste your time. :/ – Eziz Durdyyev Feb 26 at 9:26

I haven't seen the ? character in a robots.txt file before. Are you sure you don't mean to use $ instead, which denotes the end of the string?

Furthermore, your file needs to specify user-agent at the top for which bots to affect. Here's a revised file:

User-agent: *
Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css$
Allow: /*.js$
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

This is closer to what you want. To affect only Googlebot, change User-agent: * to User-agent: Googlebot.

But it does seem like different robots.txt checking tools read it differently. Google's tool blocks /us/en/cart/, for instance, while technicalseo's tool and seobook.com's tool both allow the same URL. Not sure how to reconcile that one.

| improve this answer | |
  • The ? is just a literal ? - so this would match URLs that contain a query string (in the case of CSS and JS files, this might be used to indicate version/cache busting). "technicalseo's tool" appears to disallow that URL when I tried, not "allow" it? Although "seobook.com's tool" does appear to (incorrectly) allow it. (?) – MrWhite Feb 23 at 0:13
  • "seobook.com's tool" appears to just process every directive and whatever end result spits out is what it goes with - which (AFAIK) is not how any real crawler behaves. – MrWhite Feb 23 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.