26

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which was a factor in us blocking them.

[Yahoo's Spider has] been flooding us with 70-200 requests/second since about 5:30 AM EST. I've never seen anything like this, even from their rogue "experimental" crawlers that have abused us in the past. Robots.txt ban isn't enough: I'm having to block them by IP at the load-balancer level. Since they've abused Stack Overflow in the past, I figured you might want to know.

It looks like Yahoo's web search is "Powered by Bing!" now. This is good, because we've never had a problem with Bing's spider (aka MSNBOT).

So, is Yahoo's search the same as Bing search now, or should we be looking at potentially unblocking them if they've somehow, against all odds, stopped sucking so hard?

Improve this question
1
  • 2
    Literally, and figuratively.
    – Tim Post
    Jul 28, 2010 at 17:03

4 Answers 4

Reset to default
10

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back.

From the document Joel linked to there is another document here. In it Yahoo states "In mid-July, we began testing the delivery of organic results from Bing to Yahoo!" To me that means that Yahoo is not tightly integrating with the Bing Engine. They are simply calling the Bing Engine API. That also would make the most sense from the standpoint of work required since the Yahoo Engine is going away in the long term.

That means we should see no more Yahoo Bot crawling the web once 100% of the results are coming out of the Bing Engine. Therefore, there is no reason to unblock it.

Improve this answer
1
  • 3
    Ding dong .. the bot is dead. Rub your eyes. Get out of bed. Ding dong that wicked old bot is deaaaad!
    – Tim Post
    Jul 28, 2010 at 15:30
6

It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.

Improve this answer
5
  • 2
    Interestingly, Yahoo in Japan will begin using Google search results: bbc.co.uk/news/business-10773532
    – DisgruntledGoat
    Jul 27, 2010 at 23:48
  • @DisgruntledGoat: [Excuse me Joel for replying on your answer] "Yahoo in Japan" is basically a different company from Yahoo in America, it's run by SoftBank. That's why Geocities is still alive in Japan. Also "Yahoo Auctions" in Japan is the equivalent of Ebay, etc. As that article notes, Yahoo is dominant in search in Japan (believe it or not).
    – delete
    Jul 28, 2010 at 1:54
  • @Kinopiko - Yahoo Japan is relevant but hardly dominant. They control 20% of the market share according to this, gs.statcounter.com/#search_engine-JP-daily-20080701-20100717
    – Ben Hoffman
    Jul 28, 2010 at 21:21
  • 1
    @RandomBen: Severe doubts about where that graph's data is from. If you have any traffic for Japanese-language search keywords on your website, or if you have a Japanese translation of your website, go and look at which search engine sends you more traffic, Google or Yahoo.co.jp.
    – delete
    Jul 28, 2010 at 23:54
  • @Kinopiko: Very interesting. I will do some more research. The numbers from that site for the US seem correct but it worries me that it could that far off for Japan. Makes me wonder if any other countries are incorrect.
    – Ben Hoffman
    Jul 29, 2010 at 13:21
4

I raised a search enquiry with Yahoo on Monday - the acknowledgement included the following note on the transition to Bing which you might find relevant:

Yahoo! Search is excited to reach a new stage in the Yahoo! and Microsoft Search Alliance; the integration of Microsoft algorithmic results is near complete for the U.S. and Canada marketplace. Yahoo! Search is currently testing the new integration, and you may notice a different experience during your searches. Within this test, some basic search listings will use Microsoft Web, Image, and Video results. All the enhancements you have come to expect from Yahoo! Search are preserved, and we will continue to innovate new improvements to the Search experience.

As this is only a testing stage, not every search will produce different results, and the changes may not be obvious. Results may differ for the same keyword from one search to the next. Since the ranking and indexing algorithms are different, search results may appear in a different order. Search results that did not appear before may be included, and you may not see search results that were included before. We appreciate your patience as we thoroughly test the new results.

Thank you again for contacting Yahoo! Search.

Improve this answer
1

FYI, On Monday, August 23rd (or within a couple of days of it) Yahoo will officially start serving BING search results.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.