Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Can user-agents be listed together, followed by their common rules like so in a robots.txt?

User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot #Microsoft
User-agent: MSNBot #Microsoft
User-agent: Slurp #Yahoo
User-agent: Ask Jeeves/Teoma #Ask
Disallow: /adm30_buds/
share|improve this question
1  
Is there a reason why you don't use user-agent: * followed by the disallow? Is the list of "allowed" user-agents longer than the list that you want to disallow? –  Mike Hudson Mar 18 at 6:24
    
See also on SO: Is it possible to list multiple user-agents in one line? –  unor Mar 19 at 9:28

1 Answer 1

up vote 7 down vote accepted

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

share|improve this answer
1  
The original robots.txt spec agrees: "If more than one User-agent field is present the record describes an identical access policy for more than one robot." –  Ilmari Karonen Mar 18 at 15:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.