Is it true to generalize that (in 2021) the second or later forward slash acts like a wildcard global (asterisk) in robots.txt files?
In another approach, it true to say that /MY_NAMESPACE:/
could have been in regex: /MY_NAMESPACE:*
?
In robots.txt
syntax forward slashes are not equivalent to asterisks. Robots.txt
rules are all "starts with" rules. Your Disallow: /MY_NAMESPACE:/
rule will match any URL path that starts with /MY_NAMESPACE:/
. The following URLs would be disallowed:
https://example.com/MY_NAMESPACE:/
https://example.com/MY_NAMESPACE:/foo
https://example.com/MY_NAMESPACE:/foo/bar
Some user agents including the major search engine bots do support glob wildcards where the asterisk means "any number of any character". However, most bots don't support wildcards in rules. It is best to avoid them when possible. If you do use wildcards, they will apply to the major search engines, but not to most other bots. If you are going to use them, it is often best to do so is User-Agent: Googlebot
sections that only apply to a specific bot that supports wildcards, or to use them for things that only need to apply to search engine bots.
Because robots.txt rules are "starts with", an asterisk at the end of a rule is the same as a rule without the asterisk, but it will only be understood by advanced crawlers. It is more compatible just to write the rule as Disallow: /MY_NAMESPACE:
. That rule (with or without the asterisk would disallow the following URLs:
https://example.com/MY_NAMESPACE:/
https://example.com/MY_NAMESPACE:/foo
https://example.com/MY_NAMESPACE:/foo/bar
https://example.com/MY_NAMESPACE:
https://example.com/MY_NAMESPACE:foo
https://example.com/MY_NAMESPACE:foo/bar
The only difference between your two rules is whether or not URLs without the slash immediately following the colon are disallowed.
In summary:
robot.txt
rule; it is redundant.