Smuggling requests past HAProxy with a vertical tab in Transfer-Encoding

A reverse proxy and the server behind it are supposed to agree about where each HTTP request ends. Almost always they do. A request normally has one obvious length. Either a Content-Length header says "the body is this many bytes," or a Transfer-Encoding: chunked header says "read chunks until you hit a zero-length one." Some requests carry both, by accident or on purpose. The HTTP spec has a rule for that case: chunked wins, and you throw the Content-Length away. Get the rule right on both machines and the two of them count to the same place.

Nathan Davison, working independently, found a way to make HAProxy and its backend count to different places. The trick was a single byte that neither machine printed and one of them refused to see.

The vertical tab#

Here is the shape of the request, with the odd character written out so you can spot it:

POST / HTTP/1.1
Host: 127.0.0.1:1080
Content-Length: 6
Transfer-Encoding:[\x0b]chunked

0

X

\x0b is a vertical tab. It sits between the colon and the word chunked, where you would normally put a space. It is whitespace in the loose sense that it moves the cursor and prints nothing. It is not one of the two whitespace bytes HTTP allows around a header value. Those are the space and the horizontal tab. The vertical tab and its cousin the form feed (\x0c) are not on the list.

So what is Transfer-Encoding: [\x0b]chunked? Read it strictly and the value of the header is the literal string \x0bchunked. That is not chunked, so this header asks for an encoding nobody implements. The spec is clear about what to do with that. A request whose Transfer-Encoding you can't make sense of is a request you reject. You do not quietly fall back to the Content-Length and forward the mess along.

HAProxy's older HTTP engine, the one that shipped as the legacy decoder, did exactly the thing you are not supposed to do. It looked at the malformed Transfer-Encoding, decided it wasn't valid chunked, and framed the request on Content-Length: 6 instead. Then it forwarded the request to the backend with both headers still attached. It had decided the Transfer-Encoding header was meaningless. It passed the header on intact anyway, to a machine that might read it differently.

The lenient backend#

Plenty of HTTP servers are more relaxed than the spec. A common shortcut is this: if I see a Transfer-Encoding header at all, I treat the body as chunked and ignore Content-Length. A server written that way is also not fussy about stray whitespace in the value. It looks at Transfer-Encoding:[\x0b]chunked, sees the word chunked in there, and switches to chunked framing.

Now the two machines disagree about one message. HAProxy read Content-Length: 6 and believes the body is six bytes. The backend read the Transfer-Encoding header and believes the body is a chunked stream that ends at the first zero-length chunk. That disagreement is the entire bug. Every detail after this is plumbing.

Line up what happens on a reused connection. HAProxy forwards six bytes of body, as it counted them. The backend chunk-decodes, reaches the 0 chunk terminator early, and decides the request is over before those six bytes are used up. Whatever HAProxy sent after the terminator, the backend treats as the start of the next request on that connection. An attacker picks those trailing bytes. They become a prefix stapled onto whoever's request comes down the pipe next, which on a busy proxy is some other user's. Their request line gets glued to the attacker's smuggled headers, and the backend answers a request assembled from two people.

Davison's proof of concept shows the poisoning directly. The backend echoes the headers it received. A victim's clean GET / comes back with an attacker-supplied X-Foo header welded to the front of the victim's own request line. One request's bytes leaked into another.

This is the class of bug people call HTTP request smuggling. Two parsers, rather than one bad byte, are what make it dangerous. Neither machine is broken on its own. HAProxy is arguably too lenient and the backend is definitely too lenient, but you can build each of them in good faith. What the attacker supplies is not a malfunction. It is a byte the two of them classify differently.

Connection reuse#

The desync only turns into smuggling if the connection between HAProxy and the backend is reused for more than one client's request. In Davison's setup that's the http-reuse always directive. It tells HAProxy to pool and reuse backend connections aggressively. Without connection reuse there's no next request sharing the pipe, so the attacker's leftover bytes have nobody to attach to.

Responsibility is split. The HAProxy bug is toothless without a buggy backend. The backend's leniency wouldn't amount to smuggling without HAProxy's bug handing it a Transfer-Encoding header it should never have seen. It takes both. Davison credits James Kettle for the Turbo Intruder tips that made the attack reproducible. One of them was pinning the tool to one request per connection, so a real smuggle couldn't be mistaken for ordinary pipelining noise.

Affected versions and the fix#

Davison tested the smuggle against a spread of releases: 1.7.9, 1.7.11, 1.8.19, 1.8.21, 1.9.10, and 2.0.5.

The 2.0.5 entry needs a caveat, and it's the interesting part of the version story. HAProxy was in the middle of replacing its HTTP internals. The old code path was the legacy decoder. The new one was called HTX, a native structured representation of HTTP messages. HTX did not have this bug, and by 2.0 HTX was the default. So a stock 2.0.5 was not vulnerable. You got the vulnerable behaviour back only by explicitly setting no option http-use-htx, which reverts to the legacy decoder. On 1.9, HTX existed but was off by default, so the way out there was to turn it on with option http-use-htx.

• 1.7 and 1.8 have the legacy decoder only. Vulnerable.
• 1.9 runs the legacy decoder by default, with HTX opt-in. Vulnerable by default, safe with option http-use-htx.
• 2.0 runs HTX by default. Safe by default, vulnerable with no option http-use-htx.
• 2.1 runs HTX only, the legacy decoder having been removed. Not vulnerable.

The straight code fix landed in HAProxy 2.0.6. The legacy decoder was taught to reject a Transfer-Encoding header whose value isn't actually chunked, vertical tab and all, instead of falling through to Content-Length. That mattered for people who still needed the legacy path through no option http-use-htx. It was backported to the maintained branches down to 1.6. Version 2.1 deleted the legacy decoder outright, which retires the bug by retiring the code that had it.

This is CVE-2019-18277. Davison reported it on 7 September 2019. The fix shipped in 2.0.6 on 13 September, six days later, and the CVE was assigned that October.

What generalises#

Patch every whitespace byte HAProxy mishandled and you could still build this bug tomorrow. Some other disagreement would do: a header seen twice, a chunk size read in a different base, a case-folding rule one side applies and the other doesn't. The failure wasn't that HAProxy couldn't parse a byte. It was that HAProxy decided a header was meaningless and forwarded it anyway, to a machine entitled to its own opinion about what that header meant.