Special characters from ISO-8859-1 encode website come out mangled (�) in Google search results

Question

My website encoding is ISO-8859-1. ISO-8859-1 is defined as charset in the web pages and Google Search results have always looked good.

However, for several weeks now, special characters (é, à, è, â, etc.) are replaced by � in the Google Search results, for both page titles and page descriptions.

The charset is defined on each page: And the website looks good with all web browsers, there is no encoding errors.

Answer 1

Google is getting confused because while the page is ISO-8859-1, some content is loaded into the page in UTF-8. This causes Googlebot to have to re-encode the content page content as UTF-8 so that it can process it. Something is going wrong during that process and characters are getting mangled.

For example, you use a JavaScript library for consenting to cookies. It loads UTF-8 encoded text and writes it into the page.

Ideally Google would be able to deal with this situation without getting the characters garbled. I contacted Google about this and a bug has been filed on their end. However, some other sites that are getting re-encoded are working. Whatever is happening with your site isn't affecting a lot of other sites, so it may be a lower priority fix for Google.

As a workaround you could ensure that your page and JavaScript all use the same character set. Since you don't have control over third party libraries that use UTF-8 and can't convert them to ISO-8859-1, you would have to convert your site to UTF-8.

In general, there is no good reason to use ISO-8859-1 these days. That character set only support 256 characters. UTF-8 doesn't make the page size significantly larger and it supports all unicode characters:

• The extra French characters Œ, œ, and Ÿ
• The Euro sign (€), ellipses (…), non-breaking space ( )
• Fun characters like arrows and emoji

Using UTF-8 allows you support user generated content from any language. At the very least, it allows users names to be written correctly, no matter their national origin.

Answer 2

So far i see, french diacritical signs are not a part of ISO-8859-1 (beside of apostrophe). The cache version of Google doesn't contain these characters.

I see two workarounds:

1. Encode the whole content as UTF-8 (would be the way i prefer)
2. Encode diacritical signs as HTML entities. Example: séries becomes séries. This can be done in an editor like Notepad++ and HTML Tag plugin. This is a bunch of manual work - i wouldn't do it.

BTW. you have a page https://www.hypnoweb.net/www/, which mirrors the startpage - not good.

Answer 3

� is mojibake for the Replacement Character ("�").

What's happening is that your website encoding is ISO 8859-1 ("Western"), while Google is trying to interpret the bytes as UTF-8. In UTF-8, bytes between 0 and 127 (0x00 to 0x7F) are only valid as standalone bytes, whereas bytes between 128 and 255 (0x80 to 0xFF) are only valid in groups of two or more.*

Code pages, however, do allow standalone bytes between 0x80 and 0xFF, which is where Western has the special characters located, and a common side effect is seeing the replacement character where the invalid bytes are.

The problem in your case is that the bytes that represent the special characters are also getting corrupted, since something along the way thinks that the replacement character was part of the original document and is encoding it in UTF-8. In UTF-8, the replacement character is represented as EF BF BD, and this sequence is what has taken the place of every “illegal” byte. If EF BF BD is interpreted as Western, EF becomes ï, BF becomes ¿, and BD becomes ½.

Since all of the invalid bytes have been replaced with the replacement character, all of the special characters now become �, regardless of what they were originally.

My personal recommendation is to re-encode your entire site as UTF-8, since the vast majority of websites on the Internet use UTF-8 nowadays and there are almost no good reasons to use legacy encodings anymore.

Here is a UTF-8 to Western debugging table showing Mojibake for the most common special characters.

---

* The reason why this is done is to allow UTF-8 to maintain backwards compatibility with ASCII while eliminating the need to set a specified code page or switch between code pages when encoding characters located beyond U+007F. During UTF-8's development, an algorithm to convert code points to byte sequences was developed to allow a somewhat easy way to tell which bytes should be treated as groups and how many bytes are in that group:

• If we are reading UTF-8 encoded text and the byte after the one we are currently on starts with C or D (i.e. it has 1100 or 1101 as the first four bits), this tells us that the next two bytes should be interpreted as a group, which is valid for all Unicode characters up to U+07FF, the end of the block for N'Ko characters.
• If the next byte starts with E (1110), then we need to interpret the next three bytes as a group, which is needed for all characters from U+0800 (the start of the Samaritan block) up to U+FFFF (the end of the Basic Multilingual Plane).
• If the next byte starts with F (1111), we are now beyond the Basic Multilingual Plane, so we need to treat the next four, five, or six bytes as a group, depending on the latter four bits of the next byte.

When an application converts text encoded in Western or Windows-1252 into UTF-8, standalone bytes 128 and greater are converted to multi-byte sequences (because, again, UTF-8 does not allow individual bytes greater than 127). For example, the German word für is encoded as 66 FC 72 in Western, but becomes 66 C3 BC 72 in UTF-8.