How does Punycode solve (or plan to solve) the issue with identical characters?
It does not (solve nor plan to solve), since it was never designed to do this (its sole purpose is to convert a label using Unicode characters into a label using only ASCII lowercase letters, digits and hyphens, and back) and what you imply is far more complicated than you think.
Same problem purely in ASCII
First, it is not a problem starting with IDNs, you have the exact same problem in ASCII! In some fonts,
1 (the digit one),
l (the letter l) and
I (the letter i in lower or uppercase) may appear very close, and creating ambiguity.
The ambiguity is most often resolved by humans immediately because of the context surrounding this, which is more difficult for computers to understand.
There are no algorithms to "solve" this issue.
IDNs are at a junction of various norms and regulations:
- there is the IETF standard: IDNA2003 (IDNA itself in RFC3490, Nameprep in RFC3491, Punycode in RFC3492 and Stringprep in RFC3454) and then IDNA2008 (definitions in RFC5890, protocol in RFC5891, codepoints in RFC5892, right-to-left scripts in RFC5893 and a generic document for background in RFC5894) done in 2010 in fact, with some major differences between the two, in summary: IDNA2003 was blocked with a specific version of Unicode (hence characters available) where IDNA2008 is not, both uses Punycode to go from the internationalized form to the LDH one but IDNA2003 had a "StringPrep" preliminary step that IDNA2008 does not have anymore where certain characters where remapped (which creates problems for one character in German and one in Greek, so much that they are still exceptions for registries having switch to IDNA2008 like
.DE); this faq at http://www.unicode.org/faq/idn.html shows the differences, similarities and issues between them
- some Unicode recommendation, notably UTS#36 "Unicode Security Considerations" with its section 2.8 devoted to IDNA and "Punycode spoofs" and UTS#46 "Unicode IDNA Compatibility Processing" which mostly deals about going from IDNA2003 to IDNA2008. You can also have a look at UTS#39 "Unicode Security Mechanism", there is even a full section on "Confusable Detection", and an associated file that provides list of "confusing" (confusingly similar) characters (see http://www.unicode.org/Public/security/latest/confusables.txt)
- ICANN rules, which can be mostly summarized as:
- no single character TLD, 2 ASCII characters are ccTLDs, the rest gTLDs
- TLDs can be IDNs, under two different programs: IDN ccTLDs have a fast track, following a government request (and famous "homographic" attacks or illusion of such did have impact there, for example Bulgaria had to ask multiple times for its IDN TLDs that it
bg in Cyrillic which comes to be
БГ, deemed to be too close to
br which stands fro Brazil -- and in fact ICANN just published some guidelines for "risk mitigation" of such cases: https://www.icann.org/en/system/files/files/guideline-risk-mitigation-measures-evaluation-28mar19-en.pdf), and IDN gTLDs were allowed to be applied for during the 2010-2012 ICANN opening of new TLDs, so there are some there, but not a lot (see bottom of https://www.iana.org/domains/root/db for examples; why at bottom? Because they are ordered by their LDH form, that is
xn--something, so obviously starting with
x they do not come first alphabetically)
- the list of characters permissible per TLD should be a subset of the lists ICANN is working on, in conjunction with linguistic experts in the relevant languages: for example https://www.icann.org/resources/pages/root-zone-lgr-2015-06-21-en for the root zone (hence for the permissible TLDs) or others per language at https://www.icann.org/resources/pages/second-level-lgr-2015-06-21-en
- registries are expected to publish their IDN tables (available at https://www.iana.org/domains/idn-tables) which list the allowed characters per "language" or script, other rules, and handling of variants
- and finally one of the most important rule and directly related to your problem: except for some specific case because it is needed in some languages, you are not allowed to mix, in the same label of a domain name, characters coming from different Unicode scripts
- and finally the registry own rules, that can be a subset of above points, or not. Specifically in the past, even for registries sharing the same challenges (like
.jp that share some characters) since not everyone started at the same time, the rules and the list of allowed characters may have been different (there are works to converge on common sets, at various levels, but this takes time).
Remember that not all registries are gTLDs and hence are not bound by ICANN rules, and not all registries are following IDNA2008. This is why for example you can register domain names with emojis characters in
.ws because this ccTLD just decided to accept that, as it is their jurisdictional right. Try http://👓.ws/!
But see this ICANN document at
https://www.icann.org/en/system/files/files/idn-emojis-domain-names-13feb19-en.pdf for explanations about problems with emojis in domain names (even more cases of possible graphic collisions, not even starting to take into account the newer FitzPatrick modifiers).
In practice at domain registration
Not all registrars handle IDNs and not all in the same way.
Typically, registries expect registrars to send a "language tag" along the domain name to be registered. This language tag allows to select a specific list of allowed characters because otherwise, in the absence of any language tag, the only rule that will prevail, besides the lists of characters allowed, is that you can not mix two characters from different Unicode scripts (a script being the set of characters used in a given writing system).
The language tag is needed for some languages that indeed need to mix characters: if you go to Japan you will see everywhere that numbers such as phone numbers or floor number are written in Latin characters, not with the native Japanese characters. This is one example among others.
To echo a point raised in above comments, Verisign for
.COM publish different tables of allowed characters for some language tags of languages using Cyrillic (like Russian, Ukrainian, Bulgarian, Kurdish, etc... see them at https://www.verisign.com/en_US/channel-resources/domain-registry-products/idn/idn-policy/registration-rules/index.xhtml). If you look at all of them you will see some subset of the Cyrillic Unicode script plus ASCII digits from
9. And if you do not provide a language tag, you just need to use any of the allowed characters by Verisign as long as all of them are in the same script, whatever script it is.
So two options:
- if you try to register
aа.com that is in fact
xn--a-8sb.com by providing any language tag using Cyrillic character, it will fail because no tables using Cyrillic characters also use Latin characters (besides digits) at the same time
- and if you try to register it without specifying any language tags, then the rule of not mixing characters from more than one script applies and the registration will be refused.
So that specific case can not happen in real, but of course there are indeed others that make the domain "confusingly similar" to another.
IDN Homographic Attacks
This is the specific problem you seem to focus on, ambiguities in display because multiple characters look like the same, as they have a similar graphic representation (homographic).
First an observation: this is known for a long time, and for a long time anyone discovering this is soon to predict world doom because of all the confusion that will be created. In practice, we do not see it so much or in fact so little so it just appears in newspapers from time to time with one specific case, but the rest of the time there are very few cases of domain cybersquatting based on IDNs, and very little phishing based on IDNs (for that last part because attackers do not need to do the most perfect illusion to succeed they just need the lowest one that succeeds good enough for their targets; if you look at phishing emails and the kind of URLs you find in them, sometimes you have ugly things like
http://johndoe.tretre89.webhosting.somewhere.example.com/perso/misc/895f/bank.php?account=login and with some crafted text around it and good link some people will still click on that thinking that will put them at the official website of their bank! Note also that various companies provide domain monitoring services to brands, so any registration of a "confusingly similar" domain to a big brand will trigger an alert to them, while using URLs as above, after having penetrated any webhosting, will not trigger any alert by itself before people report it as phishing).
So the problem/attack exists on paper but in practice it stays for now as a curiosity and an edge cases.
The above rules, and specifically the non mixing of scripts, were specifically put in place exactly to limit problems like that. Like described in Maximillian Laumeister's answer, browsers (but please remember that we are dealing with domain names here that do indeed appear in URLs, but they appear in many other things like email addresses or XMPP identifiers... so any UI issues is certainly not constrained to web browsers) implement typically a rule to go back to the Punycode form if they detect some script mixing or for some TLD they deem "dangerous".
They mitigate some problems, but not all.
Let us take
раураӏ.com. This is a valid domain name and it can be registered because all characters (for a given label) are in one script. But in fact it has nothing to do with the ASCII version
paypal.com because the first one is using only Cyrillic characters and translate in LDH to
Like explained at the beginning this problem does not appear with IDNs, you have the exact same problem if you stay in pure ASCII.
To forbid cases like that it will be a very difficult problem with many edge cases and false positive. You could imagine: let us take each character one by one and find out all other characters in Unicode that are graphically displayed similarly for some definition of similar, and once we have that we can compare strings and determine that
раураӏ (100% Cyrillic) and
paypal (100% Latin) are the "same" and if one exists then the other one should be forbidden. Please look at UTS#39 for an algorithm to do so and all its caveats.
Besides even the problem of the graphical display that depends on the font, once you take into account Asian or Arabic characters things become magnitude more complex (even not taking into account here that not all languages are written left to right, Arabic isn't for one), which brings us exactly to the problem of variants.
The variations of variants
Chinese is a famous and "simple" example. There is in fact "Traditional" Chinese and "Simplified" Chinese. Both still exist, depending on the context or the place, so domain names have to deal with both. But does it make sense to allow two separate registrants to register one the traditional Chinese version of a name, and the other the simplified one? Probably not.
So many registries define variants and blocking rule: once you define that character X is a variant of character Y, if X appears in a name then any other name where X is replaced by Y is either forbidden to be registered or be registered only by the same registrant.
Sometimes the rules are ephemeral also: for example
.FR introduced IDNs far after the fact that they were millions of
.FR domain names existing, so they implemented a "grandfathering" approach where for a given amount of time, if you had
cafe.fr for example you were the only eligible registrant for
café.fr because this was the IDN close "version" of it. After that period elapsed, any domain name or variation of existing domain name just playing on characters was open to registration by anyone. But again just by reading this, one may think this leads to chaos where in practice, as observed during this case or similar ones, there was no real problems (that is no massive influx of people just trying to register confusingly IDNs for existing pure ASCII based domain names).
This snippet from Verisign FAQ about IDNs touches the subject and delivers the sad truth:
Different thought leaders in the technical community have suggested
different approaches to address the character variant issue. Each
approach has both positive and negative aspects. However, the IDN
community is in agreement that the character variant issue may never
fully be addressed because languages are always in a state of change.
New character variants between languages will continue to be
introduced into languages.
If you look at IDN tables you will see that registries not only list characters allowed, per language tag, but also variants of each character, if any.
This creates a huge computational problem as the lists become huge.
And they do not even encode everything possible because in some languages some characters can not appear at some given spots or after or before some other character and so on. To handle that and as the future of IDN Tables, the IETF created a new standard called Label Generation Rules, or LGR, which allows to encode all rules about IDNs and variants in an XML-based structure. Do not hope to understand that specification without a solid grasp already on Unicode, regular expressions, and domain names.