Multiple Language Website and SEO [duplicate]

Question

Possible Duplicate:
How can I get search engines to crawl my site and see a localised view of my data?

Sorry, this post became a bit loquacious - but I haven't found any satisfactory answers on the web thus far.

I've been developing a website for a while now, into which I've built support for multiple languages. All the terms, words and phrases are stored in a file in JSON format which is decoded and converted into an array. With this, it's easy to create the entire site in a different language. The website consists of 100% home-cooked code... no Wordpress, Drupal or any other framework, so none of the multi-language plugins for the aforementioned will work.

To switch language, the user clicks a link at the top of the page (positioned at the top, in reality, it's at the bottom of the HTML structure). Each page of the site has this menu, whereby a query string parameter is added. /hotels/index.html becomes /hotels/index.html?lang=fr. The script sees the lang parameter, changes the user's cookie and loads the appropriate JSON file, and they are returned back to the same page they were on without the lang parameter. This has been a very convenient way to switch languages, as the user remains on the same page, and the links are exactly the same. I built it this way after another site that required multiple language support where SEO was not a priority (an internal company-only program).

If a visitor comes to the site for the first time, the script looks at the HTTP_ACCEPT_LANG constant and loads the appropriate language and sets the cookie (instead of geo-locating, which takes longer, and I personally dislike - I live in Thailand, and even Google serves the Thai version even though I have my OS and browser in English).

My question is - I'd eventually like Google and other SE's to index the site in these other languages, but without the lang attribute in the URL. If the user uses Google in French or Russian, they get served the site in French or Russian, respectively. Since the URL /hotels/index.html would have a version in English (default), French and Russian, but still having the same URL, I assume this is a problem for Google and other SE's.

Can someone explain how Google indexes multiple-language sites setup in this manner? Would using a sub-domain work best to separate them? I do not wish to alter the way URL's are handled on the site, as I like to keep as few query strings as possible from being indexed.

On another note, I accidentally opened the site for indexing with the language links, and somehow, the redirects did not work properly, and I have links such as /something/index.html?lang=ru?lang=ru indexed in Google. I've detected these in the script and placed a 301 redirect to remove these lang parameters... will these bad links be replaced with the redirected URL's in Google over time?

Answer 1

from your description i would say ... your site is in no way SEO optimized for multiple languages and it will be a pain to trim it in this direction.

there is now this approach: http://googlewebmastercentral.blogspot.com/2010/09/unifying-content-under-multilingual.html

basically you can use it to push language switch URLs to google. but this is more than a fix for not SEO optimized sites than a real solution.

Answer 2

This is why it's a good idea to take a look at popular frameworks and CMSes before designing your own. These are very common problems that have been solved countless times by many different developers. Some of them are not obvious when you first sit down to design a CMS, so it helps to take a look at what's out there and see what some common problems/solutions/conventions are when building a CMS.

First off, basic content ought to be served statelessly from unique URLs. It's fine to save user language preferences by a cookie or session variable, but what you should do is redirect the user to the right URL/subsite to view the content in their preferred language. Otherwise, you're effectively cloaking your content from search engines (not in a malicious way, but in a self-harming way).

So, whatever primary language is used on your site, that version of a page would be:

http://foo/bar

Then the localizations will be:

http://ru.foo/bar
http://fr.foo/bar
http://de.foo/bar
etc.

You don't necessarily need to use clean-URLs so long as each page can be accessed from a unique URL without needing a cookie; so you could still do something like:

http://foo/bar?lang=ru
http://foo/bar?lang=fr
http://foo/bar?lang=de

But it's generally accepted that clean URLs are more user-friendly and thus also search-engine-friendly.

Now, implementing this change should be fairly easy on a conventional CMS or CMS framework, since most have a routing engine that allows one to easily add custom routes to content. If you don't, and your URLs are not decoupled from your application architecture, then it may take more work.