Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

The definition of User-Agent states that several tokens can be included, as deemed necessary by the client.

I want to block certain bots via robots.txt and I am confused as to which part of the User-Agent string to use, especially for more obscure bots. For example:

Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)"
JS-Kit URL Resolver, http://js-kit.com/
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html

Do I use the second token? Can tokens contain spaces, or did the SEOkicks folks forget a semicolon after SEOkicks-Robot? I don't actually intend on making my question specific to a couple bots - I want to know the guideline: which part of UA do I place in robots.txt for these exotic bots with UA as long as a haiku?

User-agent: uMBot-LN/1.0
Disallow: /

PS: Thank you but I do not need to hear that undesirable bots are better blocked with mod_security. I already have commercial mod_sec rules in place.

share|improve this question
add comment

migrated from serverfault.com Oct 28 '13 at 17:04

This question came from our site for professional system and network administrators.

1 Answer

up vote 0 down vote accepted

Web crawlers that support robots.txt often publish a page about their crawler with instructions about how to block the crawler in robots.txt:

  • Google:

    The Google user-agent is (appropriately enough) Googlebot.

  • Yandex:

    Examples: User-agent: Yandex

  • Yahoo:

    Yahoo Slurp obeys the first entry in the robots.txt file with a User-agent containing "Slurp."

There is also a database of robot names that can be used in robots.txt on the robots website: http://www.robotstxt.org/db.html

Unfortunately, neither of the robots that you post as examples have pages that I could find, nor are they listed in the robots database. However, as a pattern, I would expect that using a slash in the User-agent line of robots.txt would not be appropriate. None of the examples that I have come across recommend that. So I would use:

User-agent: uMBot-LN
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /
share|improve this answer
    
Thanks. I have no doubt I shouldnt use the semicolon. I mentioned it because the token to be used is usually the token which contains (what resembles) a botname (followed by a slash w/ version number when available), so I didn't think that the entire token is SEOkicks-Robot +seokicks.de/robot.html, but instead only SEOkicks-Robot. As for your answer, with the exception of omitting the version number, your guess seems as good as mine. But the goal of the question is to find out if there is a standard I can follow, instead of just guessing the proper token/section to use. –  Gaia Oct 29 '13 at 0:29
    
seokicks.de/robot.html confirms that it is SEOkicks-Robot. If you don't have a URL for the robot, chances are that it is a rogue bot that doesn't implement robots.txt properly anyway and it doesn't matter what you put in the file. Legit bots have documentation. –  Stephen Ostermiller Oct 29 '13 at 0:34
    
I saw that, and it's what I am using. What about JS-Kit URL Resolver, js-kit.com? There's got to be a standard, a guideline, of at least what it should be. –  Gaia Oct 29 '13 at 0:37
1  
Notice how Yahoo also uses a "contains" rule for this case. So you could say "Yahoo-Slurp" or "Yahoo Slurp/1.78" and they would still obey it. Good crawlers will be liberal in interpreting the user-agent rules. –  Stephen Ostermiller Oct 29 '13 at 10:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.