3

Bing removed my top pages from results, 3 days ago they were showing at first page. Maybe i updated robots.txt file few weeks ago. Can someone check below template if there is something wrong that is blocking bingbot to index my top pages. Thanks in advance

User-agent: * 
Crawl-Delay: 20

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: Yahoo-slurp
Disallow: 

User-agent: Msnbot
Disallow:
2

Msnbot is just one of the user agents that Bing users, and it isn't even the main one. According to https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0

  • Bingbot - Main bing crawler
  • Msnbot - Old crawler, still used some
  • MSNBot-Media - Image and video crawler
  • AdIdxBot - Ads crawler
  • BingPreview - Preview crawler

To allow Bing to index your site you need to allow Bingbot at the very least:

User-agent: Bingbot
Disallow:

You could keep the Msnbot section and also add the other Bing user agents.

Your user agent for Yahoo is not correct either. According to https://help.yahoo.com/kb/learn-submit-website-yahoo-robotstxt-directive-sln2213.html it is just Slurp rather than Yahoo-slurp.

While you have the main user agent for Googlebot correct, Google many user agents that it crawls with. You may want to allow crawling for some of their others as well. Their documentation is here: https://support.google.com/webmasters/answer/1061943?hl=en

Another problem is that you have two User-agent: * sections. I would say that you should combine them, but the Crawl-Delay directive won't be used by anything. You disallow all crawlers that would use that section. If you want to specify a crawl delay for Googlebot, Yahoo, or Bing, you need to add the crawl delay into their section. Once you create a specific section for a bot, it ignores the * section completely. Note that Googlebot doesn't support crawl delays at all.

To do what you want I'd recommend at least the following robots.txt. You also might want to allow some ads crawlers from Google and Bing.

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow:

User-agent: Slurp
Crawl-Delay: 20
Disallow: 

User-agent: MSNBot
Crawl-Delay: 20
Disallow:

User-agent: MSNBot-Media
Crawl-Delay: 20
Disallow:

User-agent: Bingbot
Crawl-Delay: 20
Disallow:
  • What I understand, according to my robots.txt file I am blocking all search engines with below command. User-agent: * Disallow: / And after that we are requesting specific search engines. Bing Yahoo and Google to index our site with other related commands. If above understanding is correct. What should be format to get maximum traffic from any where. Because according to my analytics, i am getting traffic from different sources including duckduckgo.com Netdeals.com Etc etc. i fear i ll loose my traffic from other sources as well. stay blessed – Rebbica Apr 25 '18 at 0:41
  • I would not recommend blocking all robots except specific ones. As you point out there are many lesser search engines. Plus a new search engine could come along at any point. – Stephen Ostermiller Apr 25 '18 at 1:05
  • can you share a template of robots.txt according to my need ? you send me a very good reply + template in first response but now you understand that my need is little different. Sorry for bothering you again and again. stay blessed always – Rebbica Apr 25 '18 at 22:29
  • Just User-agent: * then Disallow: works fine to allow all crawling of your site. Then you could disallow specific bots that you find problematic. – Stephen Ostermiller Apr 26 '18 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.