I tend to think it's probably not Googlebot, it's quite easy to spoof the User Agent string, and with a bit more effort to report a false IP address.
Regardless, you're basically experiencing a Denial of Service attack so a service designed to prevent that (query 'edge server' on the search engine of your choice), or anything else that lightens the load on ...
I changed hosting and right after I tested the ahrefs crawler and everything worked perfectly, probably something from the host was blocking ahrefs ips.
Incredible is that the hosting staff did not know how to solve it.
You can't make search engines forget about old URLs on your site. Once search engines find content at a URL, they will continue to periodically crawl that URL indefinitely, even after you remove the content change the URL to show a 404 error.
Instead of returning a 404 error you could change to a more relevant status:
301 Permanent Redirect - For the ...
I tried the robots.txt method and it didn't work. So I contacted the website on their email firstname.lastname@example.org:
Can you please remove my personal website dimitarnestorov.com from
And I got the following answer:
The Internet Archive can exclude websites from the Wayback Machine
Sites get rearranged all of the time, some for the better, some for the worse.
Use a 301 Permanent Redirect. This way Google will understand that you want it to update it's index to use the new URLs, not the old ones. Assuming your restructuring directs users to better content, you should see better performance in the long run. In the short run, you ...