1

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and such?

  • FWIW Google allows you to control the crawl rate of its Googlebot from between 0.002 (1 request every 8.33 minutes) and 2 requests per second. The default seems to be towards the low end (eg. 0.003) - although this obviously varies per site. – MrWhite Mar 12 '16 at 11:09
1

I've written a search engine bot before. Building it was fun! There is a lot of cleanup that you'll have to do on the URLs, as munged URLs will constantly crash your search engine bot when it runs.

I'd set it to a 5 second sleep timer. Websites probably won't care as long as you aren't trying to hit them with 10K requests in <1 second. It's pretty easy to tell that if a user isn't looking at a page for 5+ seconds / domain, that they are a bot. Users can't read 2+ pages simultaneously. However in those 5 seconds, you can be scanning 10-15 other websites & indexing their pages. So it's not like your bot will be blocked for 5 seconds while it sleeps. It only sleeps on each site for 5 seconds.

Make sure to name/identify your bot too in the code, so that other webmasters will know that it's a search engine bot & will want to white-list it. You should be able to do that by setting a header.

0

Google will not make a request less than 2 seconds apart unless you specifically set a crawl rate. That is a good guideline.

My question is, why write a bot when there are so many open source bots fully developed and fully vetted? It would be a lot safer to use one already thrashed out than make all the same mistakes and be banned and possibly face retaliation.

  • 3
    that's a strange question.. why write any software at all? There are all kinds of open source programs from games to data backup, surely nobody should ever create anything in the same genre ever again. – user81993 Mar 12 '16 at 17:59
  • 3
    @user81993 I did not say all that for Christ sake! I did say that writing a spider comes with a lot of pitfalls that can be easily avoided by using one of many existing open source bots. Why recreate the wheel?? Especially when it is not necessary. – closetnoc Mar 12 '16 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.