Robots.txt and pattern matching

Question

Adding this to my robots.txt

User-agent: *
Disallow: /*action=*$

How does robots not recognizing wild cards handle this?

Answer 1

Good point Kris. (+1)

Also, I completely agreed with an htacess suggestion but I should point out that this will require some decent RegEx skills and the ability to pin point user-agent used by SE (unlike robots.txt where you just need to name the robot).

When using robots.txt you basically "transfer" all responsibility to the accessing crawler, which should identify the request and act accordingly.

On the contrary, when using htaccess, you are taking full control and manage the whole traffic on the server level. This is why you need to know all the user-agents used by a specific SE, detect a pattern and use it in your RegEx rules. (to see just how much of an issue this is, just read some forums about Baidu spider and the mess it can make)

I suggest using Botopedia to go over the relevant user-agent lists and to verify IP ranges, if needed.

Answer 2

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.

This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.

If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.

E.g.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow: /*action=*$

Which bans all robots except Googlebot which will honor the wildcard configuration.