I have a hobby website, where a php script presents a html page displaying information for a particular surname. I am capable of displaying information for over 100,000 surnames. So the script is called like this to generate the http output:
The script accesses a database to acquire some of the data for the surname. The script takes some 20ms to execute.
I have an issue where a bot called https://domainsproject.org was calling my script some 50 times a second, triggering 508 errors on some occasions.
So, I'm seeing something like this in my logs:
/surname.php?surname=Smith 200 /surname.php?surname=Jones 200 /surname.php?surname=Kelly 200 /surname.php?surname=Walsh 508 /surname.php?surname=Boyle 200
If I search for random surnames in search engines, I can find my website for that particular surname. So it is a good thing that my website is being crawled in this manner. However, I would like to slow down the bots. Is this possible? I am hosting my website with a common web hosting company.