1

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots.

I want to disallow content and media (images, videos) crawlers.

The reason behind doing this is that my website is targetted only for google and US market and located on hosting with limited resources.

I found different rules while googling and merge everything together:

# Block Bing
User-agent: bingbot
Disallow: /

User-agent: msnbot
Disallow: /

# Block Yahoo
User-agent: slurp
User-agent: yahoo
Disallow: /

# Block Ask
User-agent: jeeves
User-agent: teoma
Disallow: /

# Block Baidu
User-agent: baidu
Disallow: /

# Block Yandex
User-agent: yandex
Disallow: /

Are these rules correct?

Or I missed something?

Or maybe I added something redundant?

Is there any official robots.txt rules for each web crawler?

  • What's your reasoning behind this? Understanding your purpose might help provide alternative answers. – Andrew Lott Nov 26 '15 at 15:16
  • @AndrewLott Thanks for your suggestion. I updated the question with the reason. – webvitaly Nov 26 '15 at 15:20
  • The User-agent field is case-insensitive (AFAIK all the main bots treat it this way), so there is no point checking MSNBot and msnbot for instance. And if you are checking for msnbot there would not seem to be any reason to check for msnbot-media as well (unless you wanted to specify different rules in a different group)? – MrWhite Nov 26 '15 at 15:50
  • Targeting for Google is fine, but blocking yourself from other search engines just feels like shooting yourself in the foot; you'll still lose visitors. If hosting is so limited that it can't handle Bing crawlers, that would be the first thing I'd try to fix. My two cents.... – Andrew Lott Nov 26 '15 at 15:54
  • @w3d Thanks for your note. I updated the rules because User-agent is case-insensitive and now rules look much cleaner. – webvitaly Nov 26 '15 at 16:16
2

If you test the robots.txt in one of the many robot.txt validators you'll see it does what you want.

For instance using the seobook robot txt validator shows that when tested for the url / these bots should not spider your website.

If you really want that is another question. If a bot crawling the website is too much of a strain on the resources, then maybe you need to look the performance of the website and or server as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.