Late delimiters: NFKC normalization and host confusion in CPython's urlsplit

urlsplit takes a string and hands back the pieces: scheme, netloc, path, query, fragment. It finds them by scanning for delimiters. // opens the netloc, and the first /, ? or # closes it. Inside the netloc, the last @ divides the userinfo from the host. Pure string work. No network, no DNS, no encoding.

＃ is not one of those delimiters. It is FULLWIDTH NUMBER SIGN, a printable character in its own right. urlsplit walks straight past it the way it walks past a letter. Same for ℀, the ACCOUNT OF sign.

Both of them turn into delimiters slightly later, in a different part of the program.

Jonathan Birch (Microsoft) and Panayiotis Panayiotou reported that to the Python Security Response Team on 16 February 2019. Steve Dower filed the public issue on 6 March. His one-line summary of the mechanism is the whole bug:

URLs encoded with Punycode/IDNA use NFKC normalization to decompose characters. This can result in some characters introducing new segments into a URL.

IDNA and NFKC normalization#

DNS does not speak Unicode. An internationalised hostname has to be converted to Punycode before anything can resolve it, and the conversion is not a straight transliteration. IDNA, following Unicode Technical Standard #46, first maps the name to a canonical form using NFKC. That is Normalization Form KC, the one that includes compatibility decomposition. NFKC is the aggressive normalisation. It decides that a fullwidth ＃ and an ASCII # are the same character for matching purposes. It also treats ℀ as a stand-in for the letters a/c.

So ＃ normalizes to #, and ℀ normalizes to a/c, complete with the slash.

That gives you two programs looking at the same URL string and disagreeing about where the host ends. Python parsed the raw characters and found no delimiter. The thing that eventually resolves the name normalizes first, and finds one.

Where the netloc ends#

Take https://attacker.example＃@bank.example/.

urlsplit sees a netloc of attacker.example＃@bank.example, with no # anywhere in it. So it splits on the last @. Python's answer for .hostname is bank.example.

Normalize that string first and it reads attacker.example#@bank.example. The # is a fragment delimiter, so it terminates the netloc. Everything after it is fragment. The host is attacker.example, and the @bank.example part is inert decoration.

Now put an application in the middle. It calls urlsplit on a user-supplied URL and gets bank.example. It looks up whatever it has cached against that name: a session cookie, a stored credential, an allowlist entry. Then it hands the original URL to a browser or an HTTP client to actually fetch. The advisory's phrasing is careful and accurate:

A specially crafted URL could be incorrectly parsed to locate cookies or authentication data and send that information to a different host than when parsed correctly.

That is it. No memory corruption, no parser crash. Two components agreeing to disagree about a string.

First patch#

Python cannot fix this by normalizing for you. urlsplit does not know what the caller will do with the result, and normalising would change data the caller may need intact. So the fix refuses instead. A new _checknetloc was added, called from urlsplit before it builds the result:

def _checknetloc(netloc):
    if not netloc or netloc.isascii():
        return
    # looking for characters like \u2100 that expand to 'a/c'
    # IDNA uses NFKC equivalence, so normalize for this check
    import unicodedata
    netloc2 = unicodedata.normalize('NFKC', netloc)
    if netloc == netloc2:
        return
    _, _, netloc = netloc.rpartition('@') # anything to the left of '@' is okay
    for c in '/?#@:':
        if c in netloc2:
            raise ValueError("netloc '" + netloc2 + "' contains invalid " +
                             "characters under NFKC normalization")

Normalize the netloc. If nothing changed, fine. If something changed and the result contains any of /?#@:, refuse to return a parse at all:

ValueError: netloc 'example.com#@bing.com' contains invalid characters under NFKC normalization

Look at the rpartition line, though. It rebinds the local netloc, and then the loop below tests netloc2. That is still the normalized form of the whole thing. The stated intention was to skip the userinfo, on the grounds that it cannot affect routing. It has no effect on the check. This version was strict everywhere, including in places its author did not think it needed to be. That accident is what kept it safe.

Shipped in 3.5.7 on 18 March 2019 and 3.7.3 on 25 March, as CVE-2019-9636.

The false positive#

On 27 April, Chihiro Ito filed a straightforward bug report. This raised:

>>> urlsplit('http://プ:80')
ValueError: netloc 'プ:80' contains invalid characters under NFKC normalization

Nothing malicious there. プ is フ followed by a combining handakuten, which NFKC composes into the single character プ. You can read that off the error message itself, since it prints the normalized form. The netloc changed under normalisation, so the check moved on to the loop, and the loop found a :. But that colon was the port separator. It was already in the URL, honestly, before any normalisation happened.

The check was asking "does the normalized string contain a delimiter". The question it wanted answered was "does normalisation introduce a delimiter that wasn't there before".

Dower diagnosed exactly that two days later. The fix was merged on 30 April:

    n = netloc.rpartition('@')[2]  # ignore anything to the left of '@'
    n = n.replace(':', '')         # ignore characters already included
    n = n.replace('#', '')         # but not the surrounding text
    n = n.replace('?', '')
    netloc2 = unicodedata.normalize('NFKC', n)
    if n == netloc2:
        return

Strip out the delimiters that are legitimately present, normalize what is left, and check that. The false positive is gone and the original attack still gets caught. The fix is right about the thing it was written to fix.

It also promoted that dead rpartition into working code, at the top, on a variable the check actually uses. Everything to the left of the last @ is now unexamined.

The regression#

Go back to https://attacker.example＃@bank.example/. The ＃ sits before the @. Python calls that region the username. Under the new check, n is bank.example, which is ASCII and unchanged by NFKC. It passes immediately. urlsplit returns a hostname of bank.example with no complaint, and the client that normalizes before resolving goes to attacker.example.

The comment on that line reads ignore anything to the left of '@'. For a URL that is already canonical it holds: a username can't change which host you connect to. But this string isn't canonical yet. A character in the userinfo that decomposes into # or / doesn't stay in the userinfo. It creates a boundary further left than the one Python found, and the whole host disappears into a fragment.

Riccardo Schirone at Red Hat caught the regression. The corrective patch went up on 4 June 2019 and was merged twenty-four minutes later. It was titled "bpo-36742: Corrects fix to handle decomposition in usernames". The rpartition line became:

    n = netloc.replace('@', '')   # ignore characters already included

The @ is treated like the other delimiters now. Remove the ones already there, wherever they are. Normalize everything that remains, and reject the URL if the result grew a delimiter. Nothing gets excluded from the check on the grounds that it looks harmless. Harmless is a property of the normalized string, and you are holding the other one.

CVE-2019-10160 covers the window between those two commits: 2.7, 3.5, 3.6, 3.7, and 3.8.0a4 through 3.8.0b1. Fixed releases followed in 2019: 3.6.9 on 2 July, 3.7.4 on 8 July, 3.8.0 on 14 October, 2.7.17 on 19 October, and 3.5.8 on 29 October.

NVD scores it 9.8, critical, under CVSS v3.1: network attack vector, high confidentiality, integrity and availability impact. The same NVD entry also carries a CVSS v2.0 base score of 5.0. That one rates the confidentiality impact as partial and the other two as none. The 9.8 is the number that ends up in your scanner output, and it's hard to defend. Nothing here is compromised by parsing a URL. What you get is a parser that disagrees with the resolver. Whether that costs you anything depends entirely on what your code does with the answer.