Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm managing a CMS-based website. The primary language of the CMS is English. Our content is mostly in English, but user postings are sometimes in other languages. The supporting mySQL database of my site is set to UTF-8 encoding.

Until recently, "other language" (e.g. Russian) postings displayed correctly. This in spite of the fact that the CMS declares in the header of each generated page:

<meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1' />

My best guess is that in the past, browsers commonly took this header declaration as advice, and also looked at content to determine the actual encoding. If, for example, UTF-8 encoding was detected, the heading declaration was overridden. That's how "other language" text was correctly displayed. It helps that iso-8859-encoded text will be correctly interpreted by a UTF-8 decoder, that is, iso-8859-1 is effectively a subset of UTF-8.

Question 1: iso-8859-1 is effectively a subset of UTF-8, right?

As far as I know, nothing about the site, the CMS installation, or the server has changed recently. But, of course, browser tech is constantly evolving and new browser releases arrive all the time.

In the past few weeks, "other language" content is showing up as garbage (mojibake) in the latest versions of Safari (5.1) Chrome (15.0.874.81), and Firefox (7.0.1), at least. All indications are that the "auto" text encoding mode of each is now determining the site pages are iso-8859-1 encoded, end-of-story. When I manually set the browser to use UTF-8, the garbage disappears, and all text is correctly rendered, as before.

Question 2: Has the standard/practice changed, so that recent browser releases obey the header declaration of text encoding and won't override it no matter what encoding is actually present?

Question 3: If not, what other factors could account for the sudden appearance of garbage where there was none before? (For example, could a subtle change to Apache made by the hosting provider have this effect?)

I have no idea why the CMS primary distribution is iso-8859-1 encoded. I've looked at the CMS docs and submitted several queries to support channels, but … no answer. The CMS is definitely capable of supporting other languages; many "official" alternates are available, and so far as I've checked, they all declare "utf-8" in the headers of generated pages

There may be some 8859-1 encoding-dependent code deep in the CMS, I suppose. (No, I'm not going to try to find it!) But the existence of a large number of alternative language packs would seem to argue against that.

Question 4: (bonus!) If there is no encoding-dependent code in the CMS, what kind of technical reasons might make the CMS developers reluctant to move their primary distribution to UTF-8?

Question 5: Am I totally missing some point? Am I totally or partially confused about how text encoding works?

share|improve this question

3 Answers 3

As I understand it, the character encoding used by the browser is decided in the following order:

  1. The Content-Type response header as sent from the server.
  2. If not #1 then the Content-Type META tag.
  3. If neither of the above then the browser default, which I assume is based initially on the default language on the system.

AFAIK the default encoding in the browser has not changed. And I very much doubt that simply upgrading to the latest version of a browser would touch the default that was already set.

Also, I don't think it is even possible to accurately deduce the correct character encoding by analysing the page content. It needs to be told, or it falls back to the browser default. Incidentally, on the machine I'm sat at, both Firefox 3.6 and Chrome 14 both default to ISO-8859-1.

Question 1: iso-8859-1 is effectively a subset of UTF-8, right?

Unfortunately, that's not the case - US-ASCII is a subset of utf-8, but non-ASCII characters in iso-8859-1 are encoded differently than in utf-8.

Question 2: Has the standard/practice changed, so that recent browser releases obey the header declaration of text encoding and won't override it no matter what encoding is actually present?

I'm unaware of any change. This would surely break a lot of sites? A browser cannot override the character encoding, as specified by the site, by what it thinks should be the character encoding! Can it?!

Question 3: If not, what other factors could account for the sudden appearance of garbage where there was none before? (For example, could a subtle change to Apache made by the hosting provider have this effect?)

Yes, I would guess a change in server config could be causing this. The Content-Type response header (as mentioned above) might have changed.

Question 4: (bonus!) If there is no encoding-dependent code in the CMS, what kind of technical reasons might make the CMS developers reluctant to move their primary distribution to UTF-8?

How old is the CMS? What is it written in? Historically PHP does not handle multi-byte strings too well. There are many functions in PHP that only work on single byte strings. Some of the multi-byte functions are only available with PHP 5.

share|improve this answer
    
I thought I saw extensive work by Mozilla to guess the encoding -- but I could be wrong. The CMS is written in PHP. The project is several years old and actively supported. I am also running a parallel site using the same CMS, same version, in another language (Polish). With the addition of a language extension, it operates very well -- with the header declaring "utf-8". At first examination, the English support is a co-equal extension. Hmmm, maybe by studying the differences I can see the underlying issue. But it seems that fundamentally, multi-byte is supported. –  hen3ry Oct 7 '11 at 19:03

Utf-8 and non English chars do not display well in iso-8859-1 and iso-8859-1 chars do not display well in utf-8.

If you want to know the encoding your browser has detected and is using on a page:

In Firefox just go to the menu "View" > "Character encoding".

http://support.mozilla.com/en-US/kb/Menu%20Reference

  • Make sure UF-8 is automatically selected when you read a page and when you enter text in the CMS to be posted.

  • If possible configure the cms to use html character entities.

  • Try replacing the meta tag to utf-8, or to the char encoding that displays well in your site.

  • Please note that invisible chars may appear if you are using the wrong encoding

share|improve this answer
    
Presumably your first statement should be "...and [non English] iso-8859-1 chars do not display well in utf-8"? –  w3d Oct 6 '11 at 23:28

Resume

Your server had and haven't now AddDefaultCharset direcive (namely - AddDefaultCharset utf-8).

This is single reason for described past and current view.

Return (try) it back over htaccess

Bonus answer on q.4

  • charset-dependent code changes to text-manipulation exist always - most string functions have pairs as mb*()
  • when you start to think about 8bit-texts in (My)SQL, you have to think about DB-charset|client charset|connection charset|natural sorting|string size. Closed eyes and bad memory seems better choice ("English rule the world" mantra)

Bonus answer on q.5

No, you miss nothing. Old good results was just spoofed, you must got headache with 8bit from first strings on site

share|improve this answer
    
That is absolutely brilliant, and completely correct. I accidentally deleted this command from .htaccess. Putting it back solved the problem. большое спасибо! –  hen3ry Oct 7 '11 at 18:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.