2

Working with a client, I've just noticed that all of their files are being saved as Windows-1252, but they're serving them with charset=utf-8 on the Content-Type header (e.g., Content-Type: text/html; charset=utf-8 and similar for their JS and CSS).

I've recommended to them that they actually use UTF-8, which they're happy to do. But their primary authoring tool is VS.Net 2012, which defaults to Windows-1252 (in English locale Windows installs) unless the file has a signature telling it otherwise. (I was very surprised not to find a setting for this, but I've found multiple answers on Stack Overflow that seem to confirm it doesn't: 1, 2, 3.)

So we can fix this by saving their files as UTF-8 with a BOM (and possibly updating the templates similarly so new files get created that way), because if VS.Net sees the BOM it remembers to save them that way later. The Unicode standard (PDF) says that using a BOM with UTF-8 is allowed but (oddly, to my mind) "not recommended":

Use of a BOM is neither required nor recommended for UTF-8, but may be encountered...where the BOM is used as a UTF-8 signature.

Are there any significant downsides to serving UTF-8 with a BOM to general web users? Issues with user agents somehow getting it wrong, or...? I mean, anything that understands Unicode is required to understand the BOM, so it should be okay, but we all know that reality sometimes diverges from theory...

2
  • 1
    You are citing an old version of the Unicode standard. The current standard has the same text here, however. – Jukka K. Korpela Jan 13 '14 at 20:45
  • @JukkaK.Korpela: Thanks. I have to admit having stolen the link from an SO answer, I didn't check it was up-to-date. :-) – T.J. Crowder Jan 14 '14 at 7:52
2

No, there are no significant downsides to serving HTML documents as UTF-8 with BOM. Statements to the contrary are still common, but they are based on misunderstanding. Some very early browsers, which you now might find in a museum if you are very lucky, rendered a BOM literally in some encoding. Even in our times, PHP software still cannot handle BOM properly, so you should not use BOM at the start of a PHP file, as it may cause trouble when such a file is concatenated or inserted by PHP. But this is a problem intrinsic to PHP.

Software used to operate on HTML documents need to handle BOM. It’s a rather basic requirement, and UTF-8 with BOM is so common that such software should be avoided. Inconveniening people who still use such programs should not be counted as significant downside.

The W3C page The byte-order mark (BOM) in HTML no longer mentions any browser problems. It mentions issues in processing HTML documents with program code, but this just means that when you write code to process UTF-8 encoded HTML pages, or anything UTF-8 encoded, you need to be prepared to the BOM.

8
  • When serving a whole document this way, the BOM appears before the doctype causing IE to go into quirks mode. I do not know if current versions of IE still do this. – Rob Jan 13 '14 at 22:06
  • 1
    @Rob, the BOM in UTF-8 has never put IE to quirks mode. Statements to the contrary are common, but erroneous. They typically deal with PHP pages. I just checked on IE 6 (in a virtual machine), and a page with BOM is displayed in standards mode. And there was no quirks mode before IE 6. (After all, the quirks mode was invented for simulating IE 5.*.) – Jukka K. Korpela Jan 13 '14 at 23:28
  • Anything appearing before the doctype, including the BOM or spaces or any character, will put IE into quirks mode and that's been true for IE6-8 minimum and well known and documented. stackoverflow.com/questions/5063789/… – Rob Jan 13 '14 at 23:33
  • @Rob, no, the BOM has never caused quirks mode; in the question you refer to, it was probably (can’t analyze for sure any more) a line break and U+FEFF that triggered quirks mode–they do, but a U+FEFF in any position other than the very start is not a BOM. – Jukka K. Korpela Jan 14 '14 at 0:19
  • 1
    @Rob, did you actually test this on IE 6? I did. Can you cite a single documented case that can be verified? – Jukka K. Korpela Jan 14 '14 at 7:13
2

Part of the advantage of UTF-8 is that software that only knows about ASCII can still read the files. When a byte order mark is present in the file, some of software that expects ASCII text may complain that the file is "binary".

Modern web browsers are all capable of consuming UTF-8 with a BOM. I would still recommend omitting the BOM because it makes compatibility with Unix tools such as grep less straightforward.

Also, I am not aware of any advantages of including a BOM for UTF-8, so it seems like a no-brainer to omit it. (This is different than UTF-16 which has big endian and little endian variants that need to be distinguished with a BOM).

1
  • GNU grep (my version, anyway, v2.9) isn't phased at all by the BOM, but I know that was just an example. I take your general point about tools, but in this day and age, plain text just isn't plain text. If I use a tool that may strip the BOM out (running a file through grep and taking the output as the starting point of a new file, for instance), it's down to me to make sure I put it back and make sure the text hasn't gotten modified inappropriately. (And similarly if I use cat to combine files and end up with BOM-in-the-middle, etc.) But again, it's a good point to raise. – T.J. Crowder Jan 13 '14 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.