Last write wins: an ext_authz authorization bypass in Envoy

HTTP lets a header name appear as many times as it likes in one message. A protobuf map<string, string> holds one value per key. Envoy's external authorization API describes a request's headers to the policy engine with a map of exactly that kind. Somebody saw the problem coming, because this is the comment they left above the field:

// The HTTP request headers. If multiple headers share the same key, they
// must be merged according to the HTTP spec. All header keys must be
// lower-cased, because HTTP header keys are case-insensitive.
map<string, string> headers = 3;

The comment says what has to happen. The type underneath it can't. And the loop that filled the map in did what everybody's first loop does:

headers.iterate([mutable_headers](const Envoy::Http::HeaderEntry& e) {
  // Skip any client EnvoyAuthPartialBody header, which could interfere with internal use.
  if (e.key().getStringView() != Headers::get().EnvoyAuthPartialBody.get()) {
    (*mutable_headers)[std::string(e.key().getStringView())] =
        std::string(e.value().getStringView());
  }
  return Envoy::Http::HeaderMap::Iterate::Continue;
});

Send two headers with the same name and the second assignment overwrites the first. That's CVE-2021-32777, scored 8.6 in the Envoy release announcement that carried it, one of a batch of five. The credit line covers the whole batch:

Thank you to Chaoqin Li, Yangmin Zhu, Raul Gutierrez Segales and Nikolas Koutounidis for making this release happen.

It doesn't say who found which, so neither will I.

What ext_authz is for#

Envoy is a proxy that sits in front of your services, and in a service mesh like Istio it sits in front of every service, one instance per pod. ext_authz is one of its HTTP filters. Its job is to ask somebody else whether a request is allowed. Per the docs, it

calls an external gRPC or HTTP service to check whether an incoming HTTP request is authorized or not. If the request is deemed unauthorized, then the request will be denied normally with 403 (Forbidden) response.

The appeal of this design is that the policy lives in one place, written by whoever owns policy, in whatever language they like. The services behind the proxy stay ignorant of it. Envoy packages up a description of the request into a CheckRequest message: method, path, host, scheme, size, protocol, optionally the body, and the headers. The authorization service reads that, applies whatever rules it has, and answers yes or no. On yes, Envoy forwards the original request upstream.

Note the word description. The authorization service never sees the request. It sees a summary of the request built by the proxy, and the security of the whole arrangement depends on that summary being faithful about the parts anyone might write a rule against. Headers are the obvious such part. Rules about authorization, x-forwarded-for, cookie, or some internal x-tenant-id are the entire point of running the filter.

Repeated header names#

HTTP has always allowed a header name to appear more than once in a message. That's how Set-Cookie works, how a proxy chain builds up Via, how X-Forwarded-For accumulates hops. The spec says a recipient may join repeated fields into a single comma-separated value. That is why the proto comment above says the map must be filled in "according to the HTTP spec": one key, all the values, joined.

The filter didn't join them. It kept the last one and threw the rest away. So for a request like:

GET /admin HTTP/1.1
Host: internal.example
x-tenant-id: victim-corp
x-tenant-id: attacker-corp

the authorization service is told that x-tenant-id is attacker-corp, full stop, and it happily approves a request in the attacker's own tenant. Envoy then forwards the request upstream with both headers still attached, because it never removed anything. What the backend does next is a matter of which HTTP library it happens to use. Some return the first value for a repeated header, some the last, some the joined list. Only one of those three matches what the authorizer was shown, and none of them is a bug in the backend.

That's the general shape. The authorizer and the upstream disagree about what the request says, and the attacker gets to choose which of them is wrong about what. The advisory phrases it as "an HTTP request with multiple value headers may bypass authorization policies in ext_authz extension", affecting "Envoy through 1.19.1".

The patch is a few lines:

const std::string key(e.key().getStringView());
if (mutable_headers->find(key) == mutable_headers->end()) {
  (*mutable_headers)[key] = std::string(e.value().getStringView());
} else {
  // Merge duplicate headers.
  (*mutable_headers)[key].append(",").append(std::string(e.value().getStringView()));
}

The fix does what the comment said all along. The requirement was written down in the right place, directly above the field it applied to. The code that filled the field in ignored it for years, in the security-sensitive path of a proxy a large share of the industry's service meshes run on.

The fragment bug#

The same release fixed CVE-2021-32779. Envoy could match authorization rules against the URI path. A request for /user/profile#section1 was treated as a path of /user/profile#section1, which does not match a policy on /user/profile. The deny rule missed it, while the upstream saw the request as being for the profile page.

The same proto file already had the correct opinion about fragments. Field 8 of HttpRequest is documented like this:

// This field is always empty, and exists for compatibility reasons. The URL fragment is
// not submitted as part of HTTP requests; it is unknowable.
string fragment = 8;

A fragment isn't supposed to be on the wire at all. The browser keeps it. If one shows up in a request line, something unusual is going on. The fix in 1.19.1 reflects that: by default Envoy now rejects a path containing #. There is a runtime flag, envoy.reloadable_features.http_reject_path_with_fragment, for anyone who needs to turn it off, and a second temporary flag for the old behaviour of just truncating the path at the #.

If you go looking for this one under an Istio version number, mind the identifiers. Istio's bulletin, ISTIO-SECURITY-2021-008, dated 24 August 2021, lists the fragment bug as CVE-2021-39156 at CVSS 8.1, noted as merged with Envoy's CVE-2021-32779. The Envoy announcement scores CVE-2021-32779 at 8.6 like everything else in that batch. The ext_authz header bug keeps its number across both, CVE-2021-32777 at 8.6. Envoy fixed the batch in 1.19.1, 1.18.4, 1.17.4 and 1.16.5; Istio in 1.9.8, 1.10.4 and 1.11.1.