Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Can user-agents be listed together, followed by their common rules like so in a robots.txt?

User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot #Microsoft
User-agent: MSNBot #Microsoft
User-agent: Slurp #Yahoo
User-agent: Ask Jeeves/Teoma #Ask
Disallow: /adm30_buds/
share|improve this question
Is there a reason why you don't use user-agent: * followed by the disallow? Is the list of "allowed" user-agents longer than the list that you want to disallow? – Mike Hudson Mar 18 '14 at 6:24
See also on SO: Is it possible to list multiple user-agents in one line? – unor Mar 19 '14 at 9:28
up vote 7 down vote accepted

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

share|improve this answer
The original robots.txt spec agrees: "If more than one User-agent field is present the record describes an identical access policy for more than one robot." – Ilmari Karonen Mar 18 '14 at 15:41

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.