How to serve for International Googlebots if we redirect users to subdomains for language matching?

Question

So, we have a site and a few sub-domains with content in different languages and we're using Hreflang and canonical implementations for SEO.

We do detect the language that users use in their browsers and we redirect them to the pages that match their language.

Since Googlebots are not only coming from USA, but from international IP addresses as well, there might be some things that we didn't implement on our end to correctly optimize.

UPDATE: In January of this year Google announced its new locale-aware crawling configuration. Googlebot can now make requests using international IP addresses and can include Accept-Language headers that’s two different languages. This is definitely good news for locale-adaptive pages, as the different variation can now be crawled, indexed, and can rank accordingly in the different language, country-specific versions of Google. It’s important to say that Google still recommends to use separate URLs with hreflang annotations.

As more sites use the Accept-Language Header to dynamically change the language of a given page, Google's new locale-adaptive crawling is sending it.

Do we need to set up any of the following HTTP headers (or any others) on any of our language subdomains or the main domain?:

• Accept-Language
• Content-Language
• Vary: Accept-Language

What else setup do we need to become fully optimized in this area?

Answer 1

First of all, beware: canonical and hreflang at the same time sounds dangerous. You would have to use canonical only for duplication within the same language.

Back to the main topic: Automatic redirection is not a good idea...

You can read this from Google's documentation for international targeting:

Avoid automatic redirection based on the user’s perceived language. These redirections could prevent users (and search engines) from viewing all the versions of your site.

I strongly recommend to:

• Use hreflang

• Having links on a language switcher with do follow so the bot can crawl all the versions

• Not using canonical between language versions

• Implement a popup based country selector as www.sarenza.com does, leaving the user the decision of remaining in the actual market/language or changing to the right one.

Regarding the points in bold:

- Accept-Language

Could be useful for showing a message to the user telling them you have a different version of the content and then show a country/language selection popup as Sarenza.com does but never redirecting the user. Depending on the logic, it could be more useful to use a geoip database, if your site is an ecommerce probably you prefer to care for the market and not the language of the user (as the example does, see screenshot), on the other hand, if your site is only about content and the subdomains are not mapped to a market but only to a language, then it's ok to use accept-language. Make sure the code (probably a hidden div with the message) is at the end of the body so it does not add noise to the content of the page.

- Content-Language: Google does not use content-language neither other meta fields to guess the content language, they use their own algorithm to detect even if you have several languages in a single url. The answer for this is: not to use

- Vary: Accept-Language: Here it depends on your cache solution, I'm afraid it's not possible to give a "good-for-all" question. I recommend reading this question about vary:accept language in StackOverflow and this about the misuse of vary If you implement the accept language with a popup that changes depending on users' language settings and you have full page cache, then you would need to tell the proxies that a different version of accept-language produces a different cache version. In this case example it seems necessary.