Multilingual sites SEO and canonical

Question

I'm in the process of improving the SEO in one of our client's multilingual sites. Currently the site passes a user the site in their preferred language by checking for a language cookie set by the system last time it was visited. If the cookie does not exist it looks for an "Accept-Language" header from the browser and checks for a language supported by the site. If that fails it defaults to a language based on some preset defaults for GEO-Location. Once we have a language determined the site will return the page in the language determined with no change to the URL and with all the proper meta tags for that language. So if a Spanish speaking user requests domain.com/venue/access/ then they we'll see Spanish while an English speaker would see English. Of course if the user clicks a language select link with a query string including lang=?? (?? would be es,en, ect.) their cookie will be changed to that language and they'll get that language from that point forward.

The problem with this approach is, while usually great for a person, crawlers (currently) don't pass an "Accept-Language" header when they request a page. This means that a result shown for people using Google, Bing, ect. will usually be in the default language or sometimes a completely different language depending on the path the crawler has taken to a page and how it decided to index it. This is killing our SEO and click-though outside of the default language.

We're working with two ideas for how we're going to change handling of user language.

---

Version 1 - All languages have a sub-directory If a user hits a URL without a language directory we'll check for the users language by following the following steps in order until we get a match.

1. User has cookie for language preference.
2. Check the browser headers for a supported language
3. Fallback to a site default language based on GEO-Location.

Once a language has been determined the system will set a cookie, do a 301 direct and add the language sub-directory to the URL which will be parsed by mod-rewrite and passed as a query string parameter.

---

Version 2 - All NEW languages have a sub-directory The first time a user hits the site if they hit a URL without a language directory we'll check for the users language by following the following steps in order until we get a match.

1. User has cookie for language preference.
2. Check the browser headers for a supported language

If a supported language has been detected the system will set a cookie. If the language is not the default language the system will do a 301 direct and add the language sub-directory to the URL if the language is not the default site language which will be parsed by mod-rewrite and passed as a query string parameter.

---

Any input on the better option?

How should we deal with Canonical URL's depending on which option we choose, should canonical URL's point to the main URL without the language sub-directory or should they point to the version for the current language sub-directory?

The client would like, if possible, to have counters for social networking reflect the totals for a page in all languages and we're assuming having separate sub-directories for language will prevent this, unless we use a canonical to a single URL which will probably screw up crawlers and negate the whole point of the language changes.

Is there an ideal solution I might be missing?

Answer 1

I do not know the capabilities of the CMS you are using but this is what I would suggest.

First of all, try avoiding URL parameters and go for a clear URL scheme. For a "Contact" page this could look like this:

English version: http://domain.com/en/contact Polish version: http://domain.com/pl/kontakt

Or:

English version: http://en.domain.com/contact Polish version: http://pl.domain.com/kontakt

This way the URL is always easy to understand for the user. Do not use English URLs for non-English pages, so for a Polish page use "kontakt" instead of "contact". After all, if an URL is to be comprehensible for a visitor, it needs to be in the language he or she has chosen to use.

If you want to inform search engines about the language of the current page, you have three options that you can use in tandem:

1. Send the HTTP header Content-Language: xx.
2. Use the HTML META section: <meta http-equiv="Content-Language" content="xx" />. Remember that this is obsolete in HTML5.
3. Use the 'lang" attribute in the HTML element. <html lang="xx">

Hope this helps!