4

I have a hobby website, where a php script presents a html page displaying information for a particular surname. I am capable of displaying information for over 100,000 surnames. So the script is called like this to generate the http output:

/surname.php?surname=Smith

The script accesses a database to acquire some of the data for the surname. The script takes some 20ms to execute.

I have an issue where a bot called https://domainsproject.org was calling my script some 50 times a second, triggering 508 errors on some occasions.

So, I'm seeing something like this in my logs:

/surname.php?surname=Smith 200
/surname.php?surname=Jones 200
/surname.php?surname=Kelly 200
/surname.php?surname=Walsh 508
/surname.php?surname=Boyle 200

If I search for random surnames in search engines, I can find my website for that particular surname. So it is a good thing that my website is being crawled in this manner. However, I would like to slow down the bots. Is this possible? I am hosting my website with a common web hosting company.

2
  • The webserver software may have rate limiting options. However, I am assuming you are on a shared hosting plan with no control over the server configuration. – Anonymous Jan 6 at 19:36
  • Yes thats true. For now I have blocked the ip used by domainsproject.org. – Baz Jan 6 at 20:04
5

There are 3 options to my knowledge:

1 - If it's a Robots.txt compliant crawler, you can ask it to delay the requests by adding the Crawl-delay directive to your robots.txt file:

User-agent: Botname
Disallow: /private/
Crawl-delay: 10

2 - If your hosting allows and/or you have a content firewall/frontend like Cloudflare or similar, you can configure a rate-limit option so that it slows down the request.

3 - You can always block the offender, by IP or name so that it doesn't hammer your site at all.

2

As https://domainsproject.org maintainer I'd like to say that there are limits in place (random delay - 1-15 seconds, concurrency limited to 2 parallel requests at point of writing this message).

However, having the ability to specify Crawl-Delay is actually a good idea.

It is now added to crawler code: https://github.com/tb0hdan/idun/blob/main/robots.go#L63

and will be deployed gradually across crawler nodes.

Regarding IP block - it is better to use that in combination with webserver level checks, i.e. https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/ (unfortunately for me - Domains Project is there already)

2
  • 1
    What other information are you (the "domains project") collecting? From the website, it only appears to collect "domain names"? But then why does this necessitate crawling websites at all, let alone at "50 times a second"?! – MrWhite Jan 19 at 14:39
  • Robots.txt, Favicon, DNS records, SSL info. Considering screenshots. – Bohdan Turkynewych Jan 20 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.