2

How does Punycode solve (or plan to solve) the issue with identical characters?

In most browsers now Punycode is disabled unless you enable it in language preferences.

This works great except many alphabets like Cyrillic have an a that looks identical to the the a used in Latin scripts. You can't ignore the Latin script for backwards compatibility reasons which leads me to believe they must ignore the Cyrillic a.

list of homographs

2

Different clients have different mitigations, but the common thread is that they usually prevent an attacker from mixing homographs from different alphabets by falling back to punycode if more than one alphabet is used:

  • Google Chrome versions 51 and later use an algorithm similar to the one used by Firefox. Previous versions display an IDN only if all of its characters belong to one (and only one) of the user's preferred languages.

  • Safari's approach is to render problematic character sets as Punycode. This can be changed by altering the settings in Mac OS X's system files.

  • Mozilla Firefox versions 22 and later display IDNs if either the TLD prevents homograph attacks by restricting which characters can be used in domain names or labels do not mix scripts for different languages. Otherwise IDNs are displayed in Punycode.

IDN homograph attack - Wikipedia

To speak to your specific example of aa.com in Cyrillic script, here is Google Chrome's rule that detects that and displays punycode. Other browsers generally use similar rules:

  • If a hostname belongs to an non-IDN TLD(top-level-domain) such as 'com', 'net', or 'uk' and all the letters in a given label belong to a set of Cyrillic letters that look like Latin letters (e.g. Cyrillic Small Letter IE - е ), show punycode.

IDN in Google Chrome

  • So what about aa.com ? Is that supposed to be Latin or Cyrillic? – William Nov 30 '18 at 1:28
  • @William To the best of my knowledge a domain like that is considered "ambiguous" by these algorithms, and will fall back to punycode when the user is visiting a domain of a different language than their browser is set to. – Maximillian Laumeister Nov 30 '18 at 1:34
  • @William See my update for details on that specific example. – Maximillian Laumeister Nov 30 '18 at 1:39
  • 1
    If you look specifically at .com rules (it depends on the registry and even TLD) Verisign does not allow mixing of cyrillic and latin characters in the same label, per ICANN rules, for any of the languages they allow which are based on CYrillic. You can mix ASCII digits with cyrillic characters, but not other ASCII characters with Cyrillic characters. So aa.com can not exist if one a is latin and one is Cyrillic. See my extended future answer for more details. – Patrick Mevzek Apr 11 at 3:36
0

How does Punycode solve (or plan to solve) the issue with identical characters?

It does not (solve nor plan to solve), since it was never designed to do this (its sole purpose is to convert a label using Unicode characters into a label using only ASCII lowercase letters, digits and hyphens, and back) and what you imply is far more complicated than you think.

Same problem purely in ASCII

First, it is not a problem starting with IDNs, you have the exact same problem in ASCII! In some fonts, 1 (the digit one), l (the letter l) and i/I (the letter i in lower or uppercase) may appear very close, and creating ambiguity.

The ambiguity is most often resolved by humans immediately because of the context surrounding this, which is more difficult for computers to understand.

There are no algorithms to "solve" this issue.

IDNs

IDNs are at a junction of various norms and regulations:

  1. there is the IETF standard: IDNA2003 (IDNA itself in RFC3490, Nameprep in RFC3491, Punycode in RFC3492 and Stringprep in RFC3454) and then IDNA2008 (definitions in RFC5890, protocol in RFC5891, codepoints in RFC5892, right-to-left scripts in RFC5893 and a generic document for background in RFC5894) done in 2010 in fact, with some major differences between the two, in summary: IDNA2003 was blocked with a specific version of Unicode (hence characters available) where IDNA2008 is not, both uses Punycode to go from the internationalized form to the LDH one but IDNA2003 had a "StringPrep" preliminary step that IDNA2008 does not have anymore where certain characters where remapped (which creates problems for one character in German and one in Greek, so much that they are still exceptions for registries having switch to IDNA2008 like .DE); this faq at http://www.unicode.org/faq/idn.html shows the differences, similarities and issues between them
  2. some Unicode recommendation, notably UTS#36 "Unicode Security Considerations" with its section 2.8 devoted to IDNA and "Punycode spoofs" and UTS#46 "Unicode IDNA Compatibility Processing" which mostly deals about going from IDNA2003 to IDNA2008. You can also have a look at UTS#39 "Unicode Security Mechanism", there is even a full section on "Confusable Detection", and an associated file that provides list of "confusing" (confusingly similar) characters (see http://www.unicode.org/Public/security/latest/confusables.txt)
  3. ICANN rules, which can be mostly summarized as:
    1. no single character TLD, 2 ASCII characters are ccTLDs, the rest gTLDs
    2. TLDs can be IDNs, under two different programs: IDN ccTLDs have a fast track, following a government request (and famous "homographic" attacks or illusion of such did have impact there, for example Bulgaria had to ask multiple times for its IDN TLDs that it bg in Cyrillic which comes to be БГ, deemed to be too close to br which stands fro Brazil -- and in fact ICANN just published some guidelines for "risk mitigation" of such cases: https://www.icann.org/en/system/files/files/guideline-risk-mitigation-measures-evaluation-28mar19-en.pdf), and IDN gTLDs were allowed to be applied for during the 2010-2012 ICANN opening of new TLDs, so there are some there, but not a lot (see bottom of https://www.iana.org/domains/root/db for examples; why at bottom? Because they are ordered by their LDH form, that is xn--something, so obviously starting with x they do not come first alphabetically)
    3. the list of characters permissible per TLD should be a subset of the lists ICANN is working on, in conjunction with linguistic experts in the relevant languages: for example https://www.icann.org/resources/pages/root-zone-lgr-2015-06-21-en for the root zone (hence for the permissible TLDs) or others per language at https://www.icann.org/resources/pages/second-level-lgr-2015-06-21-en
    4. registries are expected to publish their IDN tables (available at https://www.iana.org/domains/idn-tables) which list the allowed characters per "language" or script, other rules, and handling of variants
    5. and finally one of the most important rule and directly related to your problem: except for some specific case because it is needed in some languages, you are not allowed to mix, in the same label of a domain name, characters coming from different Unicode scripts
  4. and finally the registry own rules, that can be a subset of above points, or not. Specifically in the past, even for registries sharing the same challenges (like .cn, .tw, .kr and .jp that share some characters) since not everyone started at the same time, the rules and the list of allowed characters may have been different (there are works to converge on common sets, at various levels, but this takes time).

Remember that not all registries are gTLDs and hence are not bound by ICANN rules, and not all registries are following IDNA2008. This is why for example you can register domain names with emojis characters in .ws because this ccTLD just decided to accept that, as it is their jurisdictional right. Try http://👓.ws/! But see this ICANN document at https://www.icann.org/en/system/files/files/idn-emojis-domain-names-13feb19-en.pdf for explanations about problems with emojis in domain names (even more cases of possible graphic collisions, not even starting to take into account the newer FitzPatrick modifiers).

In practice at domain registration

Not all registrars handle IDNs and not all in the same way. Typically, registries expect registrars to send a "language tag" along the domain name to be registered. This language tag allows to select a specific list of allowed characters because otherwise, in the absence of any language tag, the only rule that will prevail, besides the lists of characters allowed, is that you can not mix two characters from different Unicode scripts (a script being the set of characters used in a given writing system).

The language tag is needed for some languages that indeed need to mix characters: if you go to Japan you will see everywhere that numbers such as phone numbers or floor number are written in Latin characters, not with the native Japanese characters. This is one example among others.

To echo a point raised in above comments, Verisign for .COM publish different tables of allowed characters for some language tags of languages using Cyrillic (like Russian, Ukrainian, Bulgarian, Kurdish, etc... see them at https://www.verisign.com/en_US/channel-resources/domain-registry-products/idn/idn-policy/registration-rules/index.xhtml). If you look at all of them you will see some subset of the Cyrillic Unicode script plus ASCII digits from 0 to 9. And if you do not provide a language tag, you just need to use any of the allowed characters by Verisign as long as all of them are in the same script, whatever script it is.

So two options:

  • if you try to register aа.com that is in fact xn--a-8sb.com by providing any language tag using Cyrillic character, it will fail because no tables using Cyrillic characters also use Latin characters (besides digits) at the same time
  • and if you try to register it without specifying any language tags, then the rule of not mixing characters from more than one script applies and the registration will be refused.

So that specific case can not happen in real, but of course there are indeed others that make the domain "confusingly similar" to another.

IDN Homographic Attacks

This is the specific problem you seem to focus on, ambiguities in display because multiple characters look like the same, as they have a similar graphic representation (homographic).

First an observation: this is known for a long time, and for a long time anyone discovering this is soon to predict world doom because of all the confusion that will be created. In practice, we do not see it so much or in fact so little so it just appears in newspapers from time to time with one specific case, but the rest of the time there are very few cases of domain cybersquatting based on IDNs, and very little phishing based on IDNs (for that last part because attackers do not need to do the most perfect illusion to succeed they just need the lowest one that succeeds good enough for their targets; if you look at phishing emails and the kind of URLs you find in them, sometimes you have ugly things like http://johndoe.tretre89.webhosting.somewhere.example.com/perso/misc/895f/bank.php?account=login and with some crafted text around it and good link some people will still click on that thinking that will put them at the official website of their bank! Note also that various companies provide domain monitoring services to brands, so any registration of a "confusingly similar" domain to a big brand will trigger an alert to them, while using URLs as above, after having penetrated any webhosting, will not trigger any alert by itself before people report it as phishing).

So the problem/attack exists on paper but in practice it stays for now as a curiosity and an edge cases.

The above rules, and specifically the non mixing of scripts, were specifically put in place exactly to limit problems like that. Like described in Maximillian Laumeister's answer, browsers (but please remember that we are dealing with domain names here that do indeed appear in URLs, but they appear in many other things like email addresses or XMPP identifiers... so any UI issues is certainly not constrained to web browsers) implement typically a rule to go back to the Punycode form if they detect some script mixing or for some TLD they deem "dangerous".

They mitigate some problems, but not all.

Let us take раураӏ.com. This is a valid domain name and it can be registered because all characters (for a given label) are in one script. But in fact it has nothing to do with the ASCII version paypal.com because the first one is using only Cyrillic characters and translate in LDH to xn--80aa0cbo65f.com. Like explained at the beginning this problem does not appear with IDNs, you have the exact same problem if you stay in pure ASCII.

To forbid cases like that it will be a very difficult problem with many edge cases and false positive. You could imagine: let us take each character one by one and find out all other characters in Unicode that are graphically displayed similarly for some definition of similar, and once we have that we can compare strings and determine that раураӏ (100% Cyrillic) and paypal (100% Latin) are the "same" and if one exists then the other one should be forbidden. Please look at UTS#39 for an algorithm to do so and all its caveats.

Besides even the problem of the graphical display that depends on the font, once you take into account Asian or Arabic characters things become magnitude more complex (even not taking into account here that not all languages are written left to right, Arabic isn't for one), which brings us exactly to the problem of variants.

The variations of variants

Chinese is a famous and "simple" example. There is in fact "Traditional" Chinese and "Simplified" Chinese. Both still exist, depending on the context or the place, so domain names have to deal with both. But does it make sense to allow two separate registrants to register one the traditional Chinese version of a name, and the other the simplified one? Probably not. So many registries define variants and blocking rule: once you define that character X is a variant of character Y, if X appears in a name then any other name where X is replaced by Y is either forbidden to be registered or be registered only by the same registrant.

Sometimes the rules are ephemeral also: for example .FR introduced IDNs far after the fact that they were millions of .FR domain names existing, so they implemented a "grandfathering" approach where for a given amount of time, if you had cafe.fr for example you were the only eligible registrant for café.fr because this was the IDN close "version" of it. After that period elapsed, any domain name or variation of existing domain name just playing on characters was open to registration by anyone. But again just by reading this, one may think this leads to chaos where in practice, as observed during this case or similar ones, there was no real problems (that is no massive influx of people just trying to register confusingly IDNs for existing pure ASCII based domain names).

This snippet from Verisign FAQ about IDNs touches the subject and delivers the sad truth:

Different thought leaders in the technical community have suggested different approaches to address the character variant issue. Each approach has both positive and negative aspects. However, the IDN community is in agreement that the character variant issue may never fully be addressed because languages are always in a state of change. New character variants between languages will continue to be introduced into languages.

If you look at IDN tables you will see that registries not only list characters allowed, per language tag, but also variants of each character, if any. This creates a huge computational problem as the lists become huge.

And they do not even encode everything possible because in some languages some characters can not appear at some given spots or after or before some other character and so on. To handle that and as the future of IDN Tables, the IETF created a new standard called Label Generation Rules, or LGR, which allows to encode all rules about IDNs and variants in an XML-based structure. Do not hope to understand that specification without a solid grasp already on Unicode, regular expressions, and domain names.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.