1

I want to allow crawling of my website only if the URL starts with accepted region/language combinations which are us/en, gb/en and fr/fr. Other language combinations must be restricted. On the other hand crawler should be able to crawl / or /about, etc. For example:

example.com/us/en/videos  # should be allowed
example.com/de/en/videos  # should be blocked
example.com/users/mark    # should be allowed

Again, it should be blocked only if it starts with unaccepted region/language combinations. What I did so far does not work:

Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css?
Allow: /*.js?
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

I tested it with google's online robots.txt tester.

7
  • Assuming you have the User-agent: * line then what you've posted works as intended in "Google's online robots.txt tester". If you've missed the user-agent line then the tool reports a stack of errors: "No user-agent specified". (?)
    – MrWhite
    Feb 23 '20 at 0:23
  • @MrWhite, yes I added user-agent line. but it does not work as I expect. Feb 24 '20 at 11:06
  • "it does not work as I expect" - and what do you expect? When I said it "works as intended", I mean it satisfies the 3 criteria you've stated. ie. /us/en/videos IS allowed, /de/en/videos IS blocked and /users/mark IS allowed. (?)
    – MrWhite
    Feb 24 '20 at 20:54
  • @MrWhite, nope. when i test it at google's tool it blocks them all. and it says that /*/*/ rule is the reason of blocking. Feb 24 '20 at 22:30
  • Are you including the slash prefix on your input?
    – MrWhite
    Feb 25 '20 at 0:58
1

Assuming you have the User-agent: * directive on the first line then what you've posted appears to work as intended in "Google's online robots.txt tester". In that it satisfies the 3 (or more) criteria you've stated:

  1. example.com/us/en/videos IS allowed
  2. example.com/de/en/videos IS BLOCKED
  3. example.com/users/mark IS allowed
  4. example.com/ IS allowed
  5. example.com/about IS allowed
  6. example.com/us/en/ (with or without the trailing slash) IS allowed
  7. example.com/gb/en/ (with or without the trailing slash) IS allowed
  8. example.com/fr/fr/ (with or without the trailing slash) IS allowed

As per the screenshots below...

1. example.com/us/en/videos

example.com/us/en/videos

2. example.com/de/en/videos

example.com/de/en/videos

3. example.com/users/mark

example.com/users/mark

4. example.com/

example.com/

5. example.com/about

example.com/about

1
  • ok, now I see what is my mistake. I tested it with https://sites.google.com/site/ezizdurdy/. sorry to waste your time. :/ Feb 26 '20 at 9:26
0

I haven't seen the ? character in a robots.txt file before. Are you sure you don't mean to use $ instead, which denotes the end of the string?

Furthermore, your file needs to specify user-agent at the top for which bots to affect. Here's a revised file:

User-agent: *
Disallow: /*?
Disallow: /*/*/cart/
Disallow: /*/*/checkout/
Disallow: /*/*/
Allow: /*.css$
Allow: /*.js$
Allow: /us/en/
Allow: /gb/en/
Allow: /fr/fr/

This is closer to what you want. To affect only Googlebot, change User-agent: * to User-agent: Googlebot.

But it does seem like different robots.txt checking tools read it differently. Google's tool blocks /us/en/cart/, for instance, while technicalseo's tool and seobook.com's tool both allow the same URL. Not sure how to reconcile that one.

2
  • The ? is just a literal ? - so this would match URLs that contain a query string (in the case of CSS and JS files, this might be used to indicate version/cache busting). "technicalseo's tool" appears to disallow that URL when I tried, not "allow" it? Although "seobook.com's tool" does appear to (incorrectly) allow it. (?)
    – MrWhite
    Feb 23 '20 at 0:13
  • "seobook.com's tool" appears to just process every directive and whatever end result spits out is what it goes with - which (AFAIK) is not how any real crawler behaves.
    – MrWhite
    Feb 23 '20 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.