Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 2 down vote favorite

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?

share|improve this question

2 Answers

up vote 2 down vote accepted

"Hammer" is subjective, but there does seem to be a general belief that Slurp is let's say...maybe a little more aggressive than it really needs to be, and that it's been going on for years, ie. 2007, 2009, 2011. A basic search for something like slurp robots delay will turn up a giant pile of complaints.

In case you're tempted to actually block Slurp, as suggested at one of those links, you might want to also see this recent question here for a different take.

share|improve this answer
Another thing to consider is that the forum complaints (at least the 2011 thread that I read) are from users who mostly come off as a bit clueless and also complaining just to complain. – Lèse majesté Dec 23 '11 at 12:41
2  
A little dusty, but this question on Meta Stack Overflow helps to explain it. – Tim Post♦ Dec 23 '11 at 17:17
up vote 1 down vote

Yes, from personal experience, Slurp is aggressive. Baidu even more so.

If your website can withstand Google, Bing, Slurp and Baidu indexing your site within the same week while serving your customers or content viewers with minimal lag, you've chosen your platform specifications and software setups quite well.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.