7

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74.125.226.

I then go here http://www.iplists.com/ and find out there's a bunch more IP addresses for google.

I even did a whois lookup in a command prompt for google and see "Aborting search 50 records found ....." along with a whole bunch of servers with unusual names such as GOOGLE.COM.ANGRYPIRATES.COM and GOOGLE.COM.AFRICANBATS.ORG. along with different IP addresses I haven't seen before such as 91.218.229.20.

What I want to do is whitelist real google IP addresses. Should I just go ahead and white list every single IP address I found thus far or are there other precautions I should take before white listing a google address?

I don't want to rely on others searching for IP addresses for me such as the source http://www.iplists.com/ because the IP addresses can change at any time and I want to update my white list frequently.

Improve this question
2
  • 2
    Sadly, blocking IP's using a white list collected from yourself or anywhere is pointless. Harmful bots, scrapers and so forth, frequently change IP's. They buy and sell IP addresses in blocks, as any other hosting company would. Rather than wasting your time your better of using a setup that throttles users for requesting to much information to fast, the same way most busy sites work these days, in fact also the same as this network your using now. You can also setup a black list them for X hours if they repeat offend, by far the most effective method, never mind the safest method.
    – Simon Hayter
    Commented Oct 8, 2015 at 20:21
  • @SimonHayter I do the same thing. I have a white list and a black list. The black list is almost empty. Between the two is a grey list where all users are watched and determined to be human or a bot. If they are a bot, then I check the white list and if they are not on the list, then they get junk content and the nature of their access logged. Every user is compared to the log file immediately. Anything in the log gets purged over time if they do not come back or begin to behave. This has worked fairly well and I plan to make the system better soon.
    – closetnoc
    Commented Oct 8, 2015 at 23:15

2 Answers 2

Reset to default
7

From the official docs to verify Googlebot / Google:

Note that Google does not recommend using a static "whitelist".

You can verify if a web crawler accessing your server really is Googlebot (or another Google user-agent). This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard-coded them, so you must run a DNS lookup as described next.

To verify Googlebot as the caller:

  1. Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
  2. Verify that the domain name is in either googlebot.com or google.com.
  3. Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.

(Emphasis my own)

See the Google docs for a worked example:
https://support.google.com/webmasters/answer/80553?hl=en

Improve this answer
5

First things first.

There are two domain names that we are talking about; Google.com and Googlebot.com. Doing a reverse look-up for Google.com would be misleading.

Here is an answer I gave quite a while ago where I did an audit of the Google IP addresses used to crawl my site over 8 years. This is not the be-all end-all list, but could give an historical view.

What does Google use 173.194.90.xxx IP's for? Why is it not a googlebot.com pointer?

In this list, you can see the IP address range 64.249 you are seeing is correct.

I did a quick audit for this answer. This is my current white-list for googlebot (agent name). Please forgive the wildcarding. I have updated my code to understand ranges more efficiently, but have not updated the list. Sorry. In this case, I just use a simple regular expression to compare the IP address to the list with a minimum of trickery to replace the * with something that regex understands. I can compare an IP address to the entire list using my code so it is still fast enough.

64.233.160.*
64.233.161.*
64.233.162.*
64.233.163.*
64.233.164.*
64.233.165.*
64.233.166.*
64.233.167.*
64.233.168.*
64.233.169.*
64.233.170.*
64.233.171.*
64.233.172.*
64.233.173.*
64.233.174.*
64.233.175.*
64.233.176.*
64.233.177.*
64.233.178.*
64.233.179.*
64.233.180.*
64.233.181.*
64.233.182.*
64.233.183.*
64.233.184.*
64.233.185.*
64.233.186.*
64.233.187.*
64.233.188.*
64.233.189.*
64.233.190.*
64.233.191.*
66.249.64.*
66.249.65.*
66.249.66.*
66.249.67.*
66.249.68.*
66.249.69.*
66.249.70.*
66.249.71.*
66.249.72.*
66.249.73.*
66.249.74.*
66.249.75.*
66.249.76.*
66.249.77.*
66.249.78.*
66.249.79.*
66.249.80.*
66.249.82.*
66.249.83.*
66.249.84.*
66.249.85.*
66.249.86.*
66.249.87.*
66.249.88.*
66.249.89.*
66.249.90.*
66.249.91.*
66.249.92.*
66.249.93.*
66.249.94.*
66.249.95.*
74.125.*
173.194.*

GOOGLE.COM.ANGRYPIRATES.COM and GOOGLE.COM.AFRICANBATS.ORG are what is known as a domain or sub-domain hack. These do not belong to Google at all. This is something anyone can do to any domain name at any time. Without getting into the details of how this is done, you can find information on the net. I do not want to encourage this behavior by explaining it here.

Improve this answer
1
  • Thanks! Any way you'd be able to provide the updated list, even if not in a nice list format (i.e. even if it's in a regex)? EDIT: And any other bots besides Google, too, if you have them and don't mind sharing them.
    – Pamela
    Commented Feb 4, 2016 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.