Adding this to my robots.txt
User-agent: * Disallow: /*action=*$
How does robots not recognizing wild cards handle this?
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.Sign up to join this community
Robots that do not recognize wildcards (which is not in the official spec) will treat
* as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.
This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.
If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.
User-agent: * Disallow: / User-Agent: Googlebot Disallow: /*action=*$
Which bans all robots except Googlebot which will honor the wildcard configuration.