Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl pages, and seems to ignore the Gone status code when returned (as it keeps coming back).

Is there any reason why I wouldn't want to block Yahoo! Slurp via robots.txt or by IP (since it tends to ignore robots.txt in some cases anyways)?

I've confirmed that when the bot does hit it is from Yahoo! IPs, so I believe this is a legit instance of the bot.

Is Yahoo Search the same as Bing Search now? is a related question, but I don't think it completely answers whether one should add a new block of the bot.

share|improve this question

1 Answer 1

up vote 3 down vote accepted

I wish I could I find the article I read about 3 months ago. However, the short version is that Yahoo is using Bing's index (entirely) and algorithm (mostly). Yahoo is still crawling with their own bot to make small amendments to the search results and for other projects.

In short, banning the bot could drop you entirely out of the Yahoo SERPs (but would have no effect on Bing).

UPDATE: This thread on webmasterworld has some interesting posts.

share|improve this answer
    
Thanks. I had seen the thread on Webmaster World, but stopped when all I was was user agent information. Based on your response I did some searching and found gordonchoi.com/… , which backs up what you've said above. –  James Skemp Nov 29 '11 at 19:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.