I have a site with multilingual website with many pages (40M). The site has UGC, and each translation is actually for the titles. Each sub-domain points to the same content with different titles per language. As far as I understand, each sub-domain should be indexed by search engines, meaning they will actually need to crawl 40M x supported-languages. So I thought it might be best to direct each subdomain crawler, to pages that are fully in that language (titles + UGC).

Is there a way to do this? Should search engines understand this on their own?


Google says that you should avoid mixing two languages in one page. So you should not let Google index your pages that have translated titles, but original language body.

One way to do so would be to put all of the pages with the translated titles into a single subdirectory like:

  • en.example.com/translated/this-was-spanish.html
  • en.example.com/translated/this-was-german.html
  • es.example.com/translated/this-was-english.html
  • de.example.com/translated/this-was-english.html

then put Disallow: /translated/ into robots.txt. That would prevent Googlebot from both crawling and indexing the content that hasn't been fully translated.

If you can't move all the partially translated content to a single directory, then you could use the meta robots noindex tag on any page that has a translated title but an untranslated body. Google would still crawl these pages, but it would not include them in the index.

  • Can you please provide reference to where they said this? These translated pages are being served in SERP according to the searching user, so I'm pretty confused as to what is the right thing to do. Titles and descriptions are also language oriented. – Noam Jun 29 '13 at 8:47
  • From Google's official webmaster blog: googlewebmastercentral.blogspot.com/2008/08/… "Avoid mixing languages on each page, as this may confuse Googlebot as well as your users. Keep navigation and content in the same language on each page." – Stephen Ostermiller Jun 29 '13 at 11:09
  • But if the translated content is not indexed how will people who search in one of those languages find your content? Am I understanding your answer correctly? – Jeremy Cook Mar 25 '14 at 22:58

I do not recommend keeping same content in all sub-domain as it is against search engine's guidelines and may result in poor performance.

If you are operating different country, you can have same content but still you will have to change the offer price/services location etc. To redirecting crawlers is not a good practice as there is not country specific crawlers. Search engine may rank page based on relevancy and 200+ other factors.

  • Could you provide a reference to the "search engine guidelines" you speak of? – Jeremy Cook Mar 25 '14 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.