I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots.
I want to disallow content and media (images, videos) crawlers.
The reason behind doing this is that my website is targetted only for google and US market and located on hosting with limited resources.
I found different rules while googling and merge everything together:
# Block Bing User-agent: bingbot Disallow: / User-agent: msnbot Disallow: / # Block Yahoo User-agent: slurp User-agent: yahoo Disallow: / # Block Ask User-agent: jeeves User-agent: teoma Disallow: / # Block Baidu User-agent: baidu Disallow: / # Block Yandex User-agent: yandex Disallow: /
Are these rules correct?
Or I missed something?
Or maybe I added something redundant?
Is there any official robots.txt rules for each web crawler?