Bot calling my php script too fast

Question

I have a hobby website, where a php script presents a html page displaying information for a particular surname. I am capable of displaying information for over 100,000 surnames. So the script is called like this to generate the http output:

/surname.php?surname=Smith

The script accesses a database to acquire some of the data for the surname. The script takes some 20ms to execute.

I have an issue where a bot called https://domainsproject.org was calling my script some 50 times a second, triggering 508 errors on some occasions.

So, I'm seeing something like this in my logs:

/surname.php?surname=Smith 200
/surname.php?surname=Jones 200
/surname.php?surname=Kelly 200
/surname.php?surname=Walsh 508
/surname.php?surname=Boyle 200

If I search for random surnames in search engines, I can find my website for that particular surname. So it is a good thing that my website is being crawled in this manner. However, I would like to slow down the bots. Is this possible? I am hosting my website with a common web hosting company.

Answer 1

There are 3 options to my knowledge:

1. If it's a Robots.txt compliant crawler, you can ask it to delay the requests by adding the Crawl-delay directive to your robots.txt file:

   User-agent: Botname
   Disallow: /private/
   Crawl-delay: 10
   

2. If your hosting allows and/or you have a content firewall/frontend like Cloudflare or similar, you can configure a rate-limit option so that it slows down the request.

3. You can always block the offender, by IP or name so that it doesn't hammer your site at all.

Answer 2

As https://domainsproject.org maintainer I'd like to say that there are limits in place (random delay - 1-15 seconds, concurrency limited to 2 parallel requests at point of writing this message).

However, having the ability to specify Crawl-Delay is actually a good idea.

It is now added to crawler code: https://github.com/tb0hdan/idun/blob/main/robots.go#L63

and will be deployed gradually across crawler nodes.

Regarding IP block - it is better to use that in combination with webserver level checks, i.e. https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/ (unfortunately for me - Domains Project is there already)

Answer 3

This crawler crashed a t3.micro server by issuing 240 requests to a Wordpress site in a two-minute window today. The log records showed it was making 2-8 parallel requests every few seconds.

Rather than implement rate limiting, you can choose to block this bot by adding the following to your robots.txt:

# Block Domains Project crawler
User-agent: Domains Project
Disallow: /