1

I noticed a newly registered domain is being hit by non-existent URLs like

  • /pithy/597/47363-1117.doc
  • /pithy/597/47363-1117.doc
  • /2015/150728.html
  • etc...

It's being hit for thousands per day which is causing high server load.

I also noticed the IP appears to be coming from google 66.249.79.85

I figured it's probable that this are old links from the old domain owner or I could simply be attacked by some other thing.

Is there a way I can stop this bombardment from happening?

  • What is the user agent? – Stephen Ostermiller Nov 25 '15 at 15:50
  • The IP address is Google trying to chase down URLs that probably only existed a very short while ago. The simplest thing to do is just to let Google do what it does and let the pages 404 naturally. It will take time. Ultimately, you can issue a 410 for these pages, but I would not personally work that hard. Thousands of requests should not cause a high server load. I often will get as much as 60,000 requests in just one or two hours without hardly a huff or a puff on a Dell 1750. Google paces requests 2 seconds apart by default so this should not be an issue at least not from Google. – closetnoc Nov 25 '15 at 16:05
2

I noticed a newly registered domain is being hit by non-existent URLs... It's being hit for thousands per day which is causing high server load.

There are ways to mitigate the load from all sources (including google) that you believe are causing it.

  1. Cut down on the amount of code in the error pages.

This means no error pages with fancy templates, and no error pages with graphics. Heck, making an error page consisting of only one line of text would be sufficient. I do that for all my potential hackers to other sections of my server all the time. The price... maybe 29 bytes per potential hacker instead of several kilobytes per hacker. The end result: bandwidth savings which may result in a lower server bill.

  1. Unload unnecessary modules.

Determine which modules you need and only stick with them. Loading extra modules causes the server to use additional memory. If the total amount of memory required for all programs including the server program (example: apache) exceed available memory, then the hard drive will be used as additional memory and this can slow things down.

  1. Add rate-limiting.

If your server is linux-based, you can use iptables to rate limit connections from certain IP addresses. I think you can do similar with windows servers as well. Rate limiting means determining the maximum number of connections per second an IP address can make.

If you are using a free or low priced service, suggestions 2 and 3 are something your administrator will need to look after.

I also noticed the IP appears to be coming from google 66.249.79.85

Google scans websites all the time normally at a maximum rate of two requests per second. My server isn't innocent. Mine is checked too.

If google does more than two per second, then someone may be using google webmaster tools and could be managing your domain with it and decided to crank up the requests per second bar under "site settings". In this case, you can try to reverse the verification so that google will ask the offender to verify the site is once again his. Try the following if you haven't done so:

  1. In the document root folder, delete a file that begins with google, ends with .html and in between it, contains a series of hexadecimal numbers. For example: google2da9166c91fd3e7e.html

  2. Remove the meta tag in your homepage that begins with <meta name="google-site-verification"

  3. Remove the DNS TXT record from your domain that contains google-site-verification=

  4. Remove the google analytics asynchronous tracking code from your pages.

  5. Remove the google tag manager container snippet.

I gave half of these options as inverted steps from the methods google webmaster tools allows for one to verify their domain. You may also want to check out google's help pages or try contacting them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.