Adding this to my robots.txt
User-agent: *
Disallow: /*action=*$
How does robots not recognizing wild cards handle this?
-
Good point Kris. (+1) Also, I completely agreed with an htacess suggestion but I should point out that this will require some decent RegEx skills and the ability to pin point user-agent used by SE (unlike robots.txt where you just need to name the robot). When using robots.txt you basically "transfer" all responsibility to the accessing crawler, which should identify the request and act accordingly. On the contrary, when using htaccess, you are taking full control and manage the whole traffic on the server level. This is why you need to know all the user-agents used by a specific SE, detect a– Igal ZeifmanOct 3, 2012 at 10:28
Add a comment
|
1 Answer
Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.
This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.
If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.
E.g.
User-agent: *
Disallow: /
User-Agent: Googlebot
Disallow: /*action=*$
Which bans all robots except Googlebot which will honor the wildcard configuration.
-
Good point with 'others who do honor robots.txt'. There are a number of SEs out there that don't play by the rules (notably Chinese and Russian spiders!!), and if you're quite serious about it, editing your .htaccess file is a more complete option. google search on blocking bad bots with htaccess– huzzahOct 1, 2012 at 19:04
-