1

Is it true to generalize that (in 2021) the second or later forward slash acts like a wildcard global (asterisk) in robots.txt files?

In another approach, it true to say that /MY_NAMESPACE:/ could have been in regex: /MY_NAMESPACE:*?

1 Answer 1

3

In robots.txt syntax forward slashes are not equivalent to asterisks. Robots.txt rules are all "starts with" rules. Your Disallow: /MY_NAMESPACE:/ rule will match any URL path that starts with /MY_NAMESPACE:/. The following URLs would be disallowed:

  • https://example.com/MY_NAMESPACE:/
  • https://example.com/MY_NAMESPACE:/foo
  • https://example.com/MY_NAMESPACE:/foo/bar

Some user agents including the major search engine bots do support glob wildcards where the asterisk means "any number of any character". However, most bots don't support wildcards in rules. It is best to avoid them when possible. If you do use wildcards, they will apply to the major search engines, but not to most other bots. If you are going to use them, it is often best to do so is User-Agent: Googlebot sections that only apply to a specific bot that supports wildcards, or to use them for things that only need to apply to search engine bots.

Because robots.txt rules are "starts with", an asterisk at the end of a rule is the same as a rule without the asterisk, but it will only be understood by advanced crawlers. It is more compatible just to write the rule as Disallow: /MY_NAMESPACE:. That rule (with or without the asterisk would disallow the following URLs:

  • https://example.com/MY_NAMESPACE:/
  • https://example.com/MY_NAMESPACE:/foo
  • https://example.com/MY_NAMESPACE:/foo/bar
  • https://example.com/MY_NAMESPACE:
  • https://example.com/MY_NAMESPACE:foo
  • https://example.com/MY_NAMESPACE:foo/bar

The only difference between your two rules is whether or not URLs without the slash immediately following the colon are disallowed.

In summary:

  • You should never use a wildcard at the end of a robot.txt rule; it is redundant.
  • You should avoid using wildcards when possible because only a few bots understand them. If you do use them, know that the major search engine bots support them, but few other crawlers do.
  • Forward slashes and wildcards are not interchangeable in robots.txt rules.
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.