2

Can you show me how robots.txt will look when I block only Yandex bot?

I want to allow Googlebot and block Yandex bot.

0

1 Answer 1

6

There are many "Yandex bots". If you want to block all the Yandex bots then:

User-agent: Yandex
Disallow: /

However, if you only want to block the main indexing "YandexBot" then:

User-agent: YandexBot
Disallow: /

Source:
https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml

You don't need to do anything for Googlebot (or any other bot) if you want the other bots to crawl your site, since allow (crawling) is the default action.

3
  • Yeah, I generally use robots.txt to prevent something, because by default all bots are allowed to crawl. For example wikipedia site blocking Google-ads bot(Mediapartners-Google*), because they are not using any ads. So it helps them to save some bandwidth.
    – Goyllo
    Mar 25, 2017 at 11:37
  • ok so if i want block all yandex bot, i need add in robots.txt this: User-agent: Yandex Disallow: / ?
    – nikola99
    Mar 25, 2017 at 17:42
  • @nikola99 Yes, that should do it.
    – MrWhite
    Mar 25, 2017 at 22:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.