3

Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page.

While we do have these restrictions, our (financial) products are perfectly legal and regulated in other countries in Europe, Middle East and Asia.

Since we don't want to be considered as cloaking, we redirect all IP's - humans and bots alike.

As far as I understand most, if not all, Googlebot's IP ranges come from the U.S. So my question is how can we allow Google to index our website for the benefit of non-U.S. based users and avoid cloaking (or post-cloaking). Can Googlebot access from non-U.S. IP ranges? What would the preferred course of action be?

Improve this question
2

Googlebot does have IP addresses located in other countries. For a (likely incomplete) list, see the following link and click on each IP address listed there to see its country of origin:

Bots vs Browsers - User Agent Details for "+http://www.googlebot.com/bot.html"

You could add these IP's, and others you might find elsewhere, as exclusions to your IP blocking rules. However, there is no guarantee that those specific IP addresses will crawl your site if you block the U.S. based ones.

Therefore to help with this, you should indicate a Geotarget for a country outside of the U.S. in Google Webmaster Tools, as detailed here: Geotargeting

Using a country code top-level domain (ccTLD) might also help get your site listed in country specific versions of Google search, such as google.de (for Germany), and hopefully crawled by servers located outside the U.S.

Improve this answer
0

You will probably have problems with Google indexing if you try and block US traffic. Googlebot crawls mainly from the US.

Google says

"Google does not, right now, have any crawling that happens from non-US IP addresses. Right now - we only crawl from the United States"

Source http://www.youtube.com/watch?v=7paVYBgH0Hw

A javascript solution has been suggested by google a few years ago stating :

One potential solution would be to use a JavaScript-based interstitial that verifies the IP address and otherwise blocks access to your site. I assume you have to use JavaScript within your site, correct? If so, there would be no simple way for a user to selectively block the JavaScript interstitial and allow the JavaScript casino content. Assuming the JavaScript is in an external file that is disalllowed through your robots.txt file, Googlebot would not be able to view the interstitial and would be able to crawl the site normally.

Source http://productforums.google.com/d/msg/webmasters/thqhs2Mrmtw/LdSu7vj8Oi4J

Improve this answer
5
  • It's well documented that some IP's for Google's user-agents come from countries outside the U.S., and that Google doesn't like to disclose things like where its data centers are located. This video by Matt Cutts from 2.5 years ago doesn't match-up with what's showing up in server logs. – dan♦ Jul 26 '13 at 0:10
  • It might be 2.5 years ago, but I have not yet read or seen anything which contradicts that. – user29671 Jul 26 '13 at 5:15
  • Confirmation just over 1 year ago : productforums.google.com/d/msg/webmasters/HMOak-mLGcA/… really dont think much has changed in the past year. Site owner will find out quick enough anyway when USA is blocked. – user29671 Jul 26 '13 at 5:31
  • As my answer already stated "there is no guarantee that those specific IP addresses will crawl your site if you block the U.S. based ones." The post you referenced states, "makes it hard to crawl". What I stated in my comment is that some IP's for Google user-agents do come from outside the U.S., as documented in the link I provided in my answer. – dan♦ Jul 26 '13 at 5:36
  • BTW, the question states the OP already "redirect[s] all IP's - humans and bots alike." – dan♦ Jul 26 '13 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.