1
vote
2answers
129 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
4
votes
1answer
592 views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
23
votes
4answers
851 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...