Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Adding this to my robots.txt

User-agent: *
Disallow: /*action=*$

How does robots not recognizing wild cards handle this?

share|improve this question

2 Answers 2

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.

This will depend a bit on the exact implementation of the crawlers robot.txt honoring scheme and can not be entirely counted on.

If you want to avoid this you could have a separate configuration for googlebot (and others who do honor robots.txt.

E.g.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow: /*action=*$

Which bans all robots except Googlebot which will honor the wildcard configuration.

share|improve this answer
    
Good point with 'others who do honor robots.txt'. There are a number of SEs out there that don't play by the rules (notably Chinese and Russian spiders!!), and if you're quite serious about it, editing your .htaccess file is a more complete option. google search on blocking bad bots with htaccess –  huzzah Oct 1 '12 at 19:04

Good point Kris. (+1)

Also, I completely agreed with an htacess suggestion but I should point out that this will require some decent RegEx skills and the ability to pin point user-agent used by SE (unlike robots.txt where you just need to name the robot).

When using robots.txt you basically "transfer" all responsibility to the accessing crawler, which should identify the request and act accordingly.

On the contrary, when using htaccess, you are taking full control and manage the whole traffic on the server level. This is why you need to know all the user-agents used by a specific SE, detect a pattern and use it in your RegEx rules. (to see just how much of an issue this is, just read some forums about Baidu spider and the mess it can make)

I suggest using Botopedia to go over the relevant user-agent lists and to verify IP ranges, if needed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.