2

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and such?

Improve this question
4
  • FWIW Google allows you to control the crawl rate of its Googlebot from between 0.002 (1 request every 8.33 minutes) and 2 requests per second. The default seems to be towards the low end (eg. 0.003) - although this obviously varies per site.
    – MrWhite
    Mar 12, 2016 at 11:09
  • @MrWhite any references please?
    – Peyman Mohamadpour
    May 16, 2020 at 14:48
  • @Pmpr Change Googlebot crawl rate - The crawl rates I stated above are the default entry-level rates as reported in GSC (and still appears to hold true today). However, for larger/popular sites (that have capacity) then the Googlebot (minimum) crawl rates can be considerably higher.
    – MrWhite
    May 17, 2020 at 0:47
  • I can't see reported numbers in that URL
    – Peyman Mohamadpour
    May 17, 2020 at 5:43

2 Answers 2

Reset to default
1

I've written a search engine bot before. Building it was fun! There is a lot of cleanup that you'll have to do on the URLs, as munged URLs will constantly crash your search engine bot when it runs.

I'd set it to a 5 second sleep timer. Websites probably won't care as long as you aren't trying to hit them with 10K requests in <1 second. It's pretty easy to tell that if a user isn't looking at a page for 5+ seconds / domain, that they are a bot. Users can't read 2+ pages simultaneously. However in those 5 seconds, you can be scanning 10-15 other websites & indexing their pages. So it's not like your bot will be blocked for 5 seconds while it sleeps. It only sleeps on each site for 5 seconds.

Make sure to name/identify your bot too in the code, so that other webmasters will know that it's a search engine bot & will want to white-list it. You should be able to do that by setting a header.

Improve this answer
0

Google will not make a request less than 2 seconds apart unless you specifically set a crawl rate. That is a good guideline.

My question is, why write a bot when there are so many open source bots fully developed and fully vetted? It would be a lot safer to use one already thrashed out than make all the same mistakes and be banned and possibly face retaliation.

Improve this answer
2
  • 3
    that's a strange question.. why write any software at all? There are all kinds of open source programs from games to data backup, surely nobody should ever create anything in the same genre ever again.
    – user81993
    Mar 12, 2016 at 17:59
  • 3
    @user81993 I did not say all that for Christ sake! I did say that writing a spider comes with a lot of pitfalls that can be easily avoided by using one of many existing open source bots. Why recreate the wheel?? Especially when it is not necessary.
    – closetnoc
    Mar 12, 2016 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.