7

My website encoding is ISO-8859-1. ISO-8859-1 is defined as charset in the web pages and Google Search results have always looked good.

However, for several weeks now, special characters (é, à, è, â, etc.) are replaced by � in the Google Search results, for both page titles and page descriptions.

Screenshot of the Google Search rendering

The charset is defined on each page: And the website looks good with all web browsers, there is no encoding errors.

0

3 Answers 3

9

Google is getting confused because while the page is ISO-8859-1, some content is loaded into the page in UTF-8. This causes Googlebot to have to re-encode the content page content as UTF-8 so that it can process it. Something is going wrong during that process and characters are getting mangled.

For example, you use a JavaScript library for consenting to cookies. It loads UTF-8 encoded text and writes it into the page.

Ideally Google would be able to deal with this situation without getting the characters garbled. I contacted Google about this and a bug has been filed on their end. However, some other sites that are getting re-encoded are working. Whatever is happening with your site isn't affecting a lot of other sites, so it may be a lower priority fix for Google.

As a workaround you could ensure that your page and JavaScript all use the same character set. Since you don't have control over third party libraries that use UTF-8 and can't convert them to ISO-8859-1, you would have to convert your site to UTF-8.

In general, there is no good reason to use ISO-8859-1 these days. That character set only support 256 characters. UTF-8 doesn't make the page size significantly larger and it supports all unicode characters:

  • The extra French characters Œ, œ, and Ÿ
  • The Euro sign (€), ellipses (…), non-breaking space ( )
  • Fun characters like arrows and emoji

Using UTF-8 allows you support user generated content from any language. At the very least, it allows users names to be written correctly, no matter their national origin.

9
  • Thanks a lot for your detailed answer, let's hope Google will fix it one day... I would love to migrate to UTF-8 (and to a supported version of PHP), but the code base is a mess. If someone reads me and is looking for a new challenge, let me know ;-)
    – Eckinox
    May 25, 2020 at 17:23
  • 1
    i wonder, why this issue is absolutely rare to see. In the german-speaking web there are plenty of websites still using ISO-8859-1 - and since a year it is a must to include cookie banners, cookieconsent from your analyze is pretty popular cookie banner solution. With this setup there should be around each tenth website in german-speaking web messed on this kind...
    – Evgeniy
    May 25, 2020 at 18:06
  • 1
    I think there is more to it than just the cookie consent script. John Mueller said that many other sites are working fine. He said it is somehow related to the UTF-8 insertion into ISO-8859-1 and the "re-encoding" that needs to happen to handle that. I don't have more information or a good suggestion about how to diagnose the issue further. May 25, 2020 at 21:29
  • In fact the issue appeared several days after I changed my PHP editor from phpEd to PhpStorm. At first I thought the issue could come from the files encoding in PhpStorm: I was using ISO-8859-1 and in phpEd I used windows-1252. I tried to change to windows-1252 in PhpStorm, but it did not fix the problem. Would it be possible that one of my PHP files are wrongly encoded, for instance in UTF-8. Is there a way to detect that?
    – Eckinox
    May 26, 2020 at 3:18
  • 1
    I may have found the issue, I have a PHP file with charset=unknown-8bit included in all my pages. I'll try to find how to fix that...
    – Eckinox
    May 26, 2020 at 3:27
6

� is mojibake for the Replacement Character ("�" a.k.a. the "black diamond with a question mark").

What's happening is that your website encoding is ISO 8859-1 ("Western"), while Google is trying to interpret the bytes as UTF-8. In UTF-8, bytes between 0 and 127 (0x00 to 0x7F) are only valid as standalone bytes, whereas bytes between 128 and 255 (0x80 to 0xFF) are only valid in groups of two or more.*

Code pages, however, do allow standalone bytes between 0x80 and 0xFF, which is where Western has the special characters located.

For example, é is represented as just 0xE9 in Western, but in UTF-8, it is 0xC3 0xA9.

When an application tries to encode text as Unicode but comes across a byte or byte sequence that is invalid, a common side effect is seeing the replacement character where the invalid bytes or byte sequences are. Ideally, the replacement character should only be seen when displaying the text, without changing any of the underlying data.

The problem in your case is that the bytes representing the special characters are also getting corrupted. Before your webpages reach Google, something is trying to interpret the Western-encoded data stream as UTF-8, realising it can't, and then replacing all of the invalid bytes with 0xEF 0xBF 0xBD (the UTF-8 representation of the replacement character) to produce something that is valid in UTF-8 and to tell the end-user "this byte or byte sequence is erroneous". By the time the text reaches Google, it's already been corrupted, but Google doesn't know this, so when it gets the text and sees that you've declared Western as your website's encoding, it just interprets all of the bytes as Western. As a result, EF becomes ï, BF becomes ¿, and BD becomes ½.

Since all of the invalid bytes have been replaced with the replacement character, all of the special characters are going to show up as �, regardless of what they were originally. Once the text is corrupted, the only way to fix it is manual proofreading.


  1. So here's a little excerpt from your website's main page (http://www.hypnoweb.net) as of 7/4/2023:

DÉCOUVREZ LES SÉRIES DU MOMENT

Hypnoweb est un site spécialisé dans les séries TV et leurs acteurs quelle que soit la plateforme de diffusion: Netflix, Prime Video, Disney+ ou la télévision. Fun et auto-géré par ses membres, il te permet de nouer des liens, de découvrir des séries télévisées nouvelles ou anciennes mais aussi de jouer.

Grâce à notre communauté, notre base de données de séries, saisons, épisodes et acteurs est l'une des plus grosses du web! Tu pourras y retrouver toutes les informations utiles: le nombre de saisons, le résumé des épisodes, les actualités liées à tes séries préférées, et surtout, des forums de discussions sur les séries.

  1. Here's what you get if the text is encoded in UTF-8 but you try to interpret the bytes as Western:

DÉCOUVREZ LES SÉRIES DU MOMENT

Hypnoweb est un site spécialisé dans les séries TV et leurs acteurs quelle que soit la plateforme de diffusion: Netflix, Prime Video, Disney+ ou la télévision. Fun et auto-géré par ses membres, il te permet de nouer des liens, de découvrir des séries télévisées nouvelles ou anciennes mais aussi de jouer.

"Grâce à notre communauté, notre base de données de séries, saisons, épisodes et acteurs est l'une des plus grosses du web! Tu pourras y retrouver toutes les informations utiles: le nombre de saisons, le résumé des épisodes, les actualités liées à tes séries préférées, et surtout, des forums de discussions sur les séries.

  1. Here's what you get if the text is encoded in Western but you try to interpret the bytes as UTF-8:

D�COUVREZ LES S�RIES DU MOMENT

Hypnoweb est un site sp�cialis� dans les s�ries TV et leurs acteurs quelle que soit la plateforme de diffusion: Netflix, Prime Video, Disney+ ou la t�l�vision. Fun et auto-g�r� par ses membres, il te permet de nouer des liens, de d�couvrir des s�ries t�l�vis�es nouvelles ou anciennes mais aussi de jouer.

Gr�ce � notre communaut�, notre base de donn�es de s�ries, saisons, �pisodes et acteurs est l'une des plus grosses du web! Tu pourras y retrouver toutes les informations utiles: le nombre de saisons, le r�sum� des �pisodes, les actualit�s li�es � tes s�ries pr�f�r�es, et surtout, des forums de discussions sur les s�ries.

I have the "Override Text Encoding" addon on Firefox on my computer and I was able to force the browser to interpret the website as UTF-8. Not only did all of the special characters get replaced with the replacement character, but I also found this error message sitting at the very top of the console:

The byte stream was erroneous according to the character encoding that was declared. The character encoding declaration may be incorrect.

And the blind replacement of every special character with the replacement character means that...

  1. Here's what Google would output after it sees that your site has declared Western as its encoding, after the text has been corrupted:

D�COUVREZ LES S�RIES DU MOMENT

Hypnoweb est un site sp�cialis� dans les s�ries TV et leurs acteurs quelle que soit la plateforme de diffusion: Netflix, Prime Video, Disney+ ou la t�l�vision. Fun et auto-g�r� par ses membres, il te permet de nouer des liens, de d�couvrir des s�ries t�l�vis�es nouvelles ou anciennes mais aussi de jouer.

Gr�ce � notre communaut�, notre base de donn�es de s�ries, saisons, �pisodes et acteurs est l'une des plus grosses du web! Tu pourras y retrouver toutes les informations utiles: le nombre de saisons, le r�sum� des �pisodes, les actualit�s li�es � tes s�ries pr�f�r�es, et surtout, des forums de discussions sur les s�ries.


My personal recommendation is to re-encode your entire site as UTF-8, since the vast majority of websites on the Internet use UTF-8 nowadays and there are almost no good reasons to use legacy encodings anymore.

Here is a UTF-8 to Western debugging table showing Mojibake for the most common special characters.


* The reason why this is done is to allow UTF-8 to maintain backwards compatibility with ASCII while eliminating the need to set a specified code page or switch between code pages when encoding characters located beyond U+007F. During UTF-8's development, an algorithm to convert code points to byte sequences was developed to allow a somewhat easy way to tell which bytes should be treated as groups and how many bytes are in that group:

  • If the current byte starts with C or D (i.e. its first four bits are 1100 or 1101), this tells us that we need to interpret this byte and the next one as a group, which is valid for all Unicode characters starting from U+0080 (where ASCII ends) up to U+07FF, the end of the block for N'Ko characters.
  • If the current byte starts with E (1110 as its first four bits), then we need to interpret this byte and the next two bytes as a group, which is needed for all characters from U+0800 (the start of the Samaritan block) up to U+FFFF (the end of the Basic Multilingual Plane).
  • If the next byte starts with F (1111 as its first four bits), we are now beyond the Basic Multilingual Plane, so depending on the latter four bits we will need to treat this byte plus the next three, four, or five bytes as a group.

When an application converts text encoded in Western or Windows-1252 into UTF-8, standalone bytes 128 and greater are converted to multi-byte sequences (because, again, UTF-8 does not allow individual bytes greater than 127). For example, the German word für is encoded as 0x66 0xFC 0x72 in Western, but becomes 0x66 0xC3 0xBC 0x72 in UTF-8.

0
3

So far i see, french diacritical signs are not a part of ISO-8859-1 (beside of apostrophe). The cache version of Google doesn't contain these characters.

I see two workarounds:

  1. Encode the whole content as UTF-8 (would be the way i prefer)
  2. Encode diacritical signs as HTML entities. Example: séries becomes séries. This can be done in an editor like Notepad++ and HTML Tag plugin. This is a bunch of manual work - i wouldn't do it.

BTW. you have a page https://www.hypnoweb.net/www/, which mirrors the startpage - not good.

3
  • 3
    French diacritic characters are all part of ISO-8859-1. There should be no need to use entities for them for that character set. The only French characters not covered by ISO-8859-1 are Œ, œ, and the very rare Ÿ. Also missing commonly used characters are the Euro sign (€), ellipses (…), non-breaking space ( ), and various arrows (→). ISO-8859-15 would be a better choice for French than ISO-8859-1 because it includes the missing French characters and the Euro symbol. UTF-8 would be even better. May 25, 2020 at 13:57
  • Good point, the characters are not present in the Google cache version. As I cannot move easily to UTF-8, I will update my PHP script to encode special characters in HTML entities. I'm going to start with the page titles, let's see if it will help!
    – Eckinox
    May 29, 2020 at 4:22
  • It worked, I just had to encode as HTML entities! Thanks!
    – Eckinox
    May 29, 2020 at 9:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.