A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?

2 Answers 2


"Hammer" is subjective, but there does seem to be a general belief that Slurp is let's say...maybe a little more aggressive than it really needs to be, and that it's been going on for years, ie. 2007, 2009, 2011. A basic search for something like slurp robots delay will turn up a giant pile of complaints.

In case you're tempted to actually block Slurp, as suggested at one of those links, you might want to also see this recent question here for a different take.

  • Another thing to consider is that the forum complaints (at least the 2011 thread that I read) are from users who mostly come off as a bit clueless and also complaining just to complain. Commented Dec 23, 2011 at 12:41
  • 2
    A little dusty, but this question on Meta Stack Overflow helps to explain it.
    – Tim Post
    Commented Dec 23, 2011 at 17:17

Yes, from personal experience, Slurp is aggressive. Baidu even more so.

If your website can withstand Google, Bing, Slurp and Baidu indexing your site within the same week while serving your customers or content viewers with minimal lag, you've chosen your platform specifications and software setups quite well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.