10

Can user-agents be listed together, followed by their common rules like so in a robots.txt?

User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot #Microsoft
User-agent: MSNBot #Microsoft
User-agent: Slurp #Yahoo
User-agent: Ask Jeeves/Teoma #Ask
Disallow: /adm30_buds/
|improve this question
9

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

|improve this answer
  • 1
    The original robots.txt spec agrees: "If more than one User-agent field is present the record describes an identical access policy for more than one robot." – Ilmari Karonen Mar 18 '14 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.