2

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt

Could someone elaborate on the difference between these two:

http://example.com/page.htm

allow: /page
disallow: /*.htm

Applicable rule: disallow: /*.htm, because it matches more characters in the URL, so it's more specific.


http://example.com/page.php5

allow: /page
disallow: /*.ph

Applicable rule: allow: /page, because in case of matching rules, Google uses the least restrictive rule.

They look rather similar but one disallows while the other allows.

2

I'm answering this myself.

If both allow and disallow match, then compare the length of the paths. The longer one wins. If both are of the same length, then the least restrictive (allow) wins.

2
  • While counting the lengths of the paths, should we include the * (wildcard)? (for example, should the length of /*.htm be 6?)
    – Kannan
    Jul 9 at 2:48
  • 1
    @Kannan yes, count every character as if it's a non special character. In your case, 6 is correct.
    – annette
    Jul 9 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.