0

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.

| improve this question | | | | |
2

There are many "Yandex bots". If you want to block all the Yandex bots then:

User-agent: Yandex
Disallow: /

However, if you only want to block the main indexing "YandexBot" then:

User-agent: YandexBot
Disallow: /

Source:
https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml

You don't need to do anything for Googlebot (or any other bot) if you want the other bots to crawl your site, since allow (crawling) is the default action.

| improve this answer | | | | |
  • Yeah, I generally use robots.txt to prevent something, because by default all bots are allowed to crawl. For example wikipedia site blocking Google-ads bot(Mediapartners-Google*), because they are not using any ads. So it helps them to save some bandwidth. – Goyllo Mar 25 '17 at 11:37
  • ok so if i want block all yandex bot, i need add in robots.txt this: User-agent: Yandex Disallow: / ? – nikola99 Mar 25 '17 at 17:42
  • @nikola99 Yes, that should do it. – MrWhite Mar 25 '17 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.