8

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says

the /fish rule matches any path that starts with /fish (...) it doesn't match the path /desert/fish

but they also say

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/

That means that as long as a path is defined within the rule (like /something/), a match occurs wherever that pattern is found within the URL path. But if the rule doesn't define a path, having a leading / but no trailing /, the path starts from root?

(regexes would be something like

  • /fish^/fish.*
  • /fish/^.*/fish/.*

the .* and the second ^ are not necessary, it's just to emphasize the regex patterns)

That sounds quite counter-intuitive to me (I thought that could be a mistake in the Google doc). Can someone confirm this understanding of the syntax?

3
  • 1
    FWIW the latest official specification is datatracker.ietf.org/doc/html/draft-koster-rep-06 ; do note that it is still not a full fledged IETF RFC. Feb 25 at 8:43
  • Interesting, thanks. But the pattern matching rules chapter is not detailed enough to answer the question.
    – Déjà vu
    Feb 25 at 8:58
  • Yes (which is why I just did a comment, not an answer, as I don't have a true answer for your question), which may mean you are in "uncharted" territory and the final results will depend on the robot parsing the robots.txt file so the behaviour might not be uniform. If you care only about GoogleBot then Google instructions should be good enough but for other robots you will need to consult their own respective documentation. Feb 25 at 12:37

1 Answer 1

7

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/

This does look like a mistake in the Google docs IMO (as you suggest) for several reasons:

  • That example does not work in Google's own robots.txt Tester:

    Screenshot of Google's robots.txt Tester tool

  • That would seem to be the only example (AFAICS) that does not follow the expected prefix-matching nature of robots.txt pattern matching. Patterns always match from the document root (from the left). So this is indeed counter-intuitive.

  • If you needed to match /fish/ anywhere in the URL-path then you can use an explicit wildcard, eg. /*/fish/ (or even */fish/). So having /fish/ also match anywhere is not required and again counter-intuitive.

  • If the pattern /fish/ matches anywhere then you are going to have to always have rules that negate matching /fish/ in other parts of the URL. This would create more work and is counter-intuitive. If for instance you had Disallow: /fish/ and I link to your homepage with /?/fish/ then I've effectively created a nofollow link that can be shared. (This should not be possible, at least not so easily/accidentally.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.