Same Arabic phrase encoding into two different urls, why? How to handle?

Question

I have a multi-lingual site which two arabic phrases which to my eye (and to the Arabic speakers I've asked) looks the same, but they encode differently.

إعادةصياغةالنصتلقائياباللغةالعربيةمجانا becomes %D8%A7%D9%95%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D9%8A%D9%94%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

إعادةصياغةالنصتلقائياباللغةالعربيةمجانا becomes %D8%A5%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D8%A6%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

The difference is minor, but how can I control these examples? Also, does anyone have any insights into what is happening above?

Answer 1

The difference comes from normalization.

I know nothing about Arabic language but if I use both strings in a Unicode viewer, the difference starts right at the left part of the string.

First string is composed of these Unicode characters:

 ‎0627 ARABIC LETTER ALEF
 ‎0655 ARABIC HAMZA BELOW

when for the second string it is:

0625 ARABIC LETTER ALEF WITH HAMZA BELOW

The graphical display is the same, because the above are just two representations, one decomposed and one composed, of the same glyph. You can normalize strings in one direction, decomposing all characters, or in the other, composing them back.

But that explains what you have in string as UTF8 since percent encoded:

U+0627 is %D8%A7 
U+0655 is %D9%95

where

U+0625 is %D8%A5

Both versions are technically correct and the fact they exist are mostly because of how Unicode was created and the need to try keeping some compatibility with earlier situations (that were not able to handle "decomposed" characters).

However "internalized" characters in URLs are governed by a standard called "IRI", like an extension of URI. This is RFC3987 "Internationalized Resource Identifiers (IRIs)"

It explains how to go from an arbitrary Unicode character to an URL/URI/IRI.

Inside "5.3.2.2. Character Normalization" there is this:

To avoid false negatives and problems with transcoding, IRIs SHOULD be created by using NFC.

As such the NFC form of your string is really the following list of characters:

 ‎0625 ARABIC LETTER ALEF WITH HAMZA BELOW
 ‎0639 ARABIC LETTER AIN
 ‎0627 ARABIC LETTER ALEF
 ‎062F ARABIC LETTER DAL
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0635 ARABIC LETTER SAD
 ‎064A ARABIC LETTER YEH
 ‎0627 ARABIC LETTER ALEF
 ‎063A ARABIC LETTER GHAIN
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0646 ARABIC LETTER NOON
 ‎0635 ARABIC LETTER SAD
 ‎062A ARABIC LETTER TEH
 ‎0644 ARABIC LETTER LAM
 ‎0642 ARABIC LETTER QAF
 ‎0627 ARABIC LETTER ALEF
 ‎0626 ARABIC LETTER YEH WITH HAMZA ABOVE
 ‎064A ARABIC LETTER YEH
 ‎0627 ARABIC LETTER ALEF
 ‎0628 ARABIC LETTER BEH
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0644 ARABIC LETTER LAM
 ‎063A ARABIC LETTER GHAIN
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0627 ARABIC LETTER ALEF
 ‎0644 ARABIC LETTER LAM
 ‎0639 ARABIC LETTER AIN
 ‎0631 ARABIC LETTER REH
 ‎0628 ARABIC LETTER BEH
 ‎064A ARABIC LETTER YEH
 ‎0629 ARABIC LETTER TEH MARBUTA
 ‎0645 ARABIC LETTER MEEM
 ‎062C ARABIC LETTER JEEM
 ‎0627 ARABIC LETTER ALEF
 ‎0646 ARABIC LETTER NOON
 ‎0627 ARABIC LETTER ALEF

which gives this URL encoding:

%D8%A5%D8%B9%D8%A7%D8%AF%D8%A9%D8%B5%D9%8A%D8%A7%D8%BA%D8%A9%D8%A7%D9%84%D9%86%D8%B5%D8%AA%D9%84%D9%82%D8%A7%D8%A6%D9%8A%D8%A7%D8%A8%D8%A7%D9%84%D9%84%D8%BA%D8%A9%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A%D8%A9%D9%85%D8%AC%D8%A7%D9%86%D8%A7

(you can get it from https://r12a.github.io/app-conversion/index.html for example)

So in theory that should be the only form used, especially if you are the producer of those URLs.