Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
How does Punycode solve (or plan to solve) the issue with identical characters?
In most browsers now Punycode is disabled unless you enable it in language preferences.
This works great except many alphabets like Cyrillic have an a that looks identical to the the a used in Latin scripts. You can't ignore the Latin script for backwards compatibility reasons which leads me to believe they must ignore the Cyrillic a.
Different clients have different mitigations, but the common thread is that they usually prevent an attacker from mixing homographs from different alphabets by falling back to punycode if more than one alphabet is used:
Google Chrome versions 51 and later use an algorithm similar to the one used by Firefox. Previous versions display an IDN only if all of its characters belong to one (and only one) of the user's preferred languages.
Safari's approach is to render problematic character sets as Punycode. This can be changed by altering the settings in Mac OS X's system files.
Mozilla Firefox versions 22 and later display IDNs if either the TLD prevents homograph attacks by restricting which characters can be used in domain names or labels do not mix scripts for different languages. Otherwise IDNs are displayed in Punycode.
To speak to your specific example of aa.com in Cyrillic script, here is Google Chrome's rule that detects that and displays punycode. Other browsers generally use similar rules:
- If a hostname belongs to an non-IDN TLD(top-level-domain) such as 'com', 'net', or 'uk' and all the letters in a given label belong to a set of Cyrillic letters that look like Latin letters (e.g. Cyrillic Small Letter IE - е ), show punycode.
.com rules (it depends on the registry and even TLD) Verisign does not allow mixing of cyrillic and latin characters in the same label, per ICANN rules, for any of the languages they allow which are based on CYrillic. You can mix ASCII digits with cyrillic characters, but not other ASCII characters with Cyrillic characters. So aa.com can not exist if one a is latin and one is Cyrillic. See my extended future answer for more details.
Apr 11, 2019 at 3:36
How does Punycode solve (or plan to solve) the issue with identical characters?
It does not (solve nor plan to solve), since it was never designed to do this (its sole purpose is to convert a label using Unicode characters into a label using only ASCII lowercase letters, digits and hyphens, and back) and what you imply is far more complicated than you think or at least describe.
First, it is not a problem starting with IDNs, you have the exact same problem in ASCII! In some fonts, 1 (the digit one), l (the letter l) and i/I (the letter i in lower or uppercase) may appear very close, and creating ambiguity.
It is not even just a one character problem: for example, r + n side by side could look very similar to m in some fonts...
The ambiguity is most often resolved by humans immediately because of the context surrounding this, which is more difficult for computers to understand.
There are no algorithms to "solve" this issue.
This point is rightfully mentioned in ICANN IDN guidelines at https://www.icann.org/en/system/files/files/idn-guidelines-10may18-en.pdf which say on page 5:
It is important to understand that not all visual similarity issues can be addressed by IDN Tables and IDN policies. Other policies such as dispute resolution policies may be necessary to mitigate against abusive registrations exploiting visually similar characters. For example, even for ASCII letters,digits and hyphen (LDH)basedrepertoire, where the small letter "l" and digit "1" may be considered visually confusable characters,the mitigation policy for abuse is often addressed by dispute resolution policies, leveraging other bodies of knowledge (e.g. Trademark Law) to evaluate whether similarities between domain names causes confusion and abuse.
IDNs are at a junction of various norms and regulations:
.DE); this faq at http://www.unicode.org/faq/idn.html shows the differences, similarities and issues between thembg in Cyrillic which comes to be БГ, deemed to be too close to br which stands fro Brazil -- and in fact ICANN just published some guidelines for "risk mitigation" of such cases: https://www.icann.org/en/system/files/files/guideline-risk-mitigation-measures-evaluation-28mar19-en.pdf), and IDN gTLDs were allowed to be applied for during the 2010-2012 ICANN opening of new TLDs, so there are some there, but not a lot (see bottom of https://www.iana.org/domains/root/db for examples; why at bottom? Because they are ordered by their LDH form, that is xn--something, so obviously starting with x they do not come first alphabetically).cn, .tw, .kr and .jp that share some characters) since not everyone started at the same time, the rules and the list of allowed characters may have been different (there are works to converge on common sets, at various levels, but this takes time).Remember that not all registries are gTLDs and hence are not bound by ICANN rules, and not all registries are following IDNA2008. This is why for example you can register domain names with emojis characters in .ws because this ccTLD just decided to accept that, as it is their jurisdictional right. Try http://👓.ws/!
But see this ICANN document at
https://www.icann.org/en/system/files/files/idn-emojis-domain-names-13feb19-en.pdf for explanations about problems with emojis in domain names (even more cases of possible graphic collisions, not even starting to take into account the newer FitzPatrick modifiers).
Not all registrars handle IDNs and not all in the same way. Typically, registries expect registrars to send a "language tag" along the domain name to be registered. This language tag allows to select a specific list of allowed characters because otherwise, in the absence of any language tag, the only rule that will prevail, besides the lists of characters allowed, is that you can not mix two characters from different Unicode scripts (a script being the set of characters used in a given writing system).
The language tag is needed for some languages that indeed need to mix characters: if you go to Japan you will see everywhere that numbers such as phone numbers or floor number are written in Latin characters, not with the native Japanese characters. This is one example among others.
To echo a point raised in above comments, Verisign for .COM publish different tables of allowed characters for some language tags of languages using Cyrillic (like Russian, Ukrainian, Bulgarian, Kurdish, etc... see them at https://www.verisign.com/en_US/channel-resources/domain-registry-products/idn/idn-policy/registration-rules/index.xhtml). If you look at all of them you will see some subset of the Cyrillic Unicode script plus ASCII digits from 0 to 9. And if you do not provide a language tag, you just need to use any of the allowed characters by Verisign as long as all of them are in the same script, whatever script it is.
So two options:
aа.com that is in fact xn--a-8sb.com by providing any language tag using Cyrillic character, it will fail because no tables using Cyrillic characters also use Latin characters (besides digits) at the same timeSo that specific case can not happen in real, but of course there are indeed others that make the domain "confusingly similar" to another.
This is the specific problem you seem to focus on, ambiguities in display because multiple characters look like the same, as they have a similar graphic representation (homographic).
First an observation: this is known for a long time, and for a long time anyone discovering this is soon to predict world doom because of all the confusion that will be created. In practice, we do not see it so much or in fact so little so it just appears in newspapers from time to time with one specific case, but the rest of the time there are very few cases of domain cybersquatting based on IDNs, and very little phishing based on IDNs (for that last part because attackers do not need to do the most perfect illusion to succeed they just need the lowest one that succeeds good enough for their targets; if you look at phishing emails and the kind of URLs you find in them, sometimes you have ugly things like http://johndoe.tretre89.webhosting.somewhere.example.com/perso/misc/895f/bank.php?account=login and with some crafted text around it and good link some people will still click on that thinking that will put them at the official website of their bank! Note also that various companies provide domain monitoring services to brands, so any registration of a "confusingly similar" domain to a big brand will trigger an alert to them, while using URLs as above, after having penetrated any webhosting, will not trigger any alert by itself before people report it as phishing).
So the problem/attack exists on paper but in practice it stays for now as a curiosity and an edge cases.
The above rules, and specifically the non mixing of scripts, were specifically put in place exactly to limit problems like that. Like described in Maximillian Laumeister's answer, browsers (but please remember that we are dealing with domain names here that do indeed appear in URLs, but they appear in many other things like email addresses or XMPP identifiers... so any UI issues is certainly not constrained to web browsers) implement typically a rule to go back to the Punycode form if they detect some script mixing or for some TLD they deem "dangerous".
They mitigate some problems, but not all.
Let us take раураӏ.com. This is a valid domain name and it can be registered because all characters (for a given label) are in one script. But in fact it has nothing to do with the ASCII version paypal.com because the first one is using only Cyrillic characters and translate in LDH to xn--80aa0cbo65f.com.
Like explained at the beginning this problem does not appear with IDNs, you have the exact same problem if you stay in pure ASCII.
To forbid cases like that it will be a very difficult problem with many edge cases and false positive. You could imagine: let us take each character one by one and find out all other characters in Unicode that are graphically displayed similarly for some definition of similar, and once we have that we can compare strings and determine that раураӏ (100% Cyrillic) and paypal (100% Latin) are the "same" and if one exists then the other one should be forbidden. Please look at UTS#39 for an algorithm to do so and all its caveats.
Besides even the problem of the graphical display that depends on the font, once you take into account Asian or Arabic characters things become magnitude more complex (even not taking into account here that not all languages are written left to right, Arabic isn't for one), which brings us exactly to the problem of variants.
Chinese is a famous and "simple" example. There is in fact "Traditional" Chinese and "Simplified" Chinese. Both still exist, depending on the context or the place, so domain names have to deal with both. But does it make sense to allow two separate registrants to register one the traditional Chinese version of a name, and the other the simplified one? Probably not. So many registries define variants and blocking rule: once you define that character X is a variant of character Y, if X appears in a name then any other name where X is replaced by Y is either forbidden to be registered or be registered only by the same registrant.
Sometimes the rules are ephemeral also: for example .FR introduced IDNs far after the fact that they were millions of .FR domain names existing, so they implemented a "grandfathering" approach where for a given amount of time, if you had cafe.fr for example you were the only eligible registrant for café.fr because this was the IDN close "version" of it. After that period elapsed, any domain name or variation of existing domain name just playing on characters was open to registration by anyone. But again just by reading this, one may think this leads to chaos where in practice, as observed during this case or similar ones, there was no real problems (that is no massive influx of people just trying to register confusingly IDNs for existing pure ASCII based domain names).
This snippet from Verisign FAQ about IDNs touches the subject and delivers the sad truth:
Different thought leaders in the technical community have suggested different approaches to address the character variant issue. Each approach has both positive and negative aspects. However, the IDN community is in agreement that the character variant issue may never fully be addressed because languages are always in a state of change. New character variants between languages will continue to be introduced into languages.
If you look at IDN tables you will see that registries not only list characters allowed, per language tag, but also variants of each character, if any. This creates a huge computational problem as the lists become huge.
And they do not even encode everything possible because in some languages some characters can not appear at some given spots or after or before some other character and so on. To handle that and as the future of IDN Tables, the IETF created a new standard called Label Generation Rules, or LGR, which allows to encode all rules about IDNs and variants in an XML-based structure. Do not hope to understand that specification without a solid grasp already on Unicode, regular expressions, and domain names.
