I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per language. As far as I understand, each sub-domain should be indexed by search engines, meaning they will actually need to crawl 40M x supported-languages. So I thought it might be best to direct each subdomain crawler, to pages that are fully in that language (titles + UGC).

Is there a way to do this? Should search engines understand this on their own?

2 Answers 2


Google says that you should avoid mixing two languages in one page. So you should not let Google index your pages that have translated titles, but original language body.

One way to do so would be to put all of the pages with the translated titles into a single subdirectory like:

  • en.example.com/translated/this-was-spanish.html
  • en.example.com/translated/this-was-german.html
  • es.example.com/translated/this-was-english.html
  • de.example.com/translated/this-was-english.html

then put Disallow: /translated/ into robots.txt. That would prevent Googlebot from both crawling and indexing the content that hasn't been fully translated.

If you can't move all the partially translated content to a single directory, then you could use the meta robots noindex tag on any page that has a translated title but an untranslated body. Google would still crawl these pages, but it would not include them in the index.

  • Can you please provide reference to where they said this? These translated pages are being served in SERP according to the searching user, so I'm pretty confused as to what is the right thing to do. Titles and descriptions are also language oriented.
    – Noam
    Commented Jun 29, 2013 at 8:47
  • From Google's official webmaster blog: googlewebmastercentral.blogspot.com/2008/08/… "Avoid mixing languages on each page, as this may confuse Googlebot as well as your users. Keep navigation and content in the same language on each page." Commented Jun 29, 2013 at 11:09
  • But if the translated content is not indexed how will people who search in one of those languages find your content? Am I understanding your answer correctly? Commented Mar 25, 2014 at 22:58

I do not recommend keeping same content in all sub-domain as it is against search engine's guidelines and may result in poor performance.

If you are operating different country, you can have same content but still you will have to change the offer price/services location etc. To redirecting crawlers is not a good practice as there is not country specific crawlers. Search engine may rank page based on relevancy and 200+ other factors.

  • Could you provide a reference to the "search engine guidelines" you speak of? Commented Mar 25, 2014 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.