Half-blind: SSRF in Kubernetes dynamic provisioning via StorageClass parameters

Put a URL in a YAML file and the Kubernetes control plane will go and fetch it for you. Not from your network. From its own.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi.storage.svc:8080"
  clusterid: "630372ccdc720a92c681fb928f27b53f"

That is a StorageClass, the small object describing one kind of storage a cluster can hand out. This one is for GlusterFS. resturl is the address of a Heketi server, the REST service that actually creates Gluster volumes. When somebody creates a PersistentVolumeClaim that names this class, kube-controller-manager makes an HTTP request to that address on their behalf.

That's the bug. Everything after this is about why it took until 2020 to notice, and about the return path. The return path is the better half of the story.

Brice Augras of Groupe-Asten and Christophe Hauquiert of Nokia reported it. It's CVE-2020-8555, and the Kubernetes issue is titled "Half-Blind SSRF in kube-controller-manager". The advisory went out on oss-security on 1 June 2020, signed by Tim Allclair for the Kubernetes Product Security Committee. It describes the impact as allowing "certain authorized users to leak up to 500 bytes of arbitrary information from unprotected endpoints within the master's host network".

Dynamic provisioning#

Dynamic provisioning is the feature where a user asks for storage and gets it without an administrator doing anything. The user writes a PVC and the PVC names a StorageClass. A controller notices and goes off to create the volume. That controller lives in kube-controller-manager, which runs on the control plane node. Not in the pod's namespace, not on a worker, not behind whatever network policy applies to workloads. On the control plane, usually on the host network, next to the API server and next to etcd.

So the request goes out from a place with a much better view of the cluster than the person asking for it has. That is the classic setup for server-side request forgery, where a privileged component fetches a URL you chose. Kubernetes has a lot of that by design. The whole control plane is a machine for turning declarative objects into actions taken by privileged components.

The four in-tree volume plugins named in the issue all take an endpoint as a plain string parameter: GlusterFS, Quobyte, StorageOS and ScaleIO. Here's GlusterFS reading its parameters, from parseClassParameters in pkg/volume/glusterfs/glusterfs.go:

for k, v := range params {
    switch dstrings.ToLower(k) {
    case "resturl":
        cfg.url = v
    case "restuser":
        cfg.user = v
    ...

No parse, no scheme check, no host allowlist. The string goes to the Heketi client. The client builds a URL out of it, and off it goes. clusterid is a parameter too, appended into the request path for the ClusterInfo call, so the attacker gets to shape the path as well as the host.

There are two ways in. If you can write StorageClasses, you can point one anywhere. If you can't, creating a pod with one of the affected volume types is enough for some of them, because those volume sources carry their own endpoint in the pod spec. Quobyte's has a registry field, described in the API types as "a single or multiple Quobyte Registry services specified as a string as host:port pair". You put a host:port in your own pod spec and the control plane connects to it.

What you don't get is much control over the request itself. The GitHub issue is precise about this: the attacker can "cause kube-controller-manager to make GET requests or POST requests without an attacker controlled request body". You choose the destination. You don't choose the verb, the headers, or the body. That rules out a lot of the fun SSRF targets and leaves the ones that respond to a bare GET, which, on a control plane host, is still a decent list.

The return path#

An SSRF where you can't see the response is a lot less useful. You can port-scan by timing, you can hit something that has a side effect, and that's about it. This one gives you the response, and the way it does that is entirely accidental.

Kubernetes vendors the Heketi client. Its error handling looks like this, in pkg/utils/bodystring.go:

// Return the body from a response as an error
func GetErrorFromResponse(r *http.Response) error {
	s, err := GetStringFromResponse(r)
	if err != nil {
		return err
	}

	s = strings.TrimSpace(s)
	if len(s) == 0 {
		return fmt.Errorf("server did not provide a message (status %v: %v)", r.StatusCode, http.StatusText(r.StatusCode))
	}
	return errors.New(s)
}

The body of a non-OK response becomes the text of the error. That is a perfectly reasonable thing for a client library to do when it's talking to its own server. Heketi's error bodies are short JSON messages meant for humans.

That error travels up. The volume plugin wraps it:

volume, err := cli.VolumeCreate(volumeReq)
if err != nil {
    return nil, 0, "", fmt.Errorf("failed to create volume: %v", err)
}

And the persistent volume controller, in pv_controller.go, does what controllers do when something goes wrong. It tells the user, by writing an event against the object they created.

strerr := fmt.Sprintf("Failed to provision volume with StorageClass %q: %v", storageClass.Name, err)
klog.V(2).Infof("failed to provision volume for claim %q with StorageClass %q: %v", claimToClaimKey(claim), storageClass.Name, err)
ctrl.eventRecorder.Event(claim, v1.EventTypeWarning, events.ProvisioningFailed, strerr)

An event on a PVC. The user who created that PVC can read it. Reading events on your own objects is the most ordinary thing in the world, and a failed provisioning attempt is exactly the situation where you'd go and look. kubectl describe pvc, and there in the Events table is the first few hundred bytes of whatever the control plane got back from the address you named.

Point it at something that returns a body to an unauthenticated GET, and the body comes back to you through the error message. That's the half-blind part: no control over what you send, and up to 500 bytes of sight of what comes back.

What the patch changes#

The patch is PR #89794, titled "Clean up event messages for errors". That is an unglamorous name for a security patch and a fair description of what it does. The same edit went into all four plugins:

volume, err := cli.VolumeCreate(volumeReq)
if err != nil {
    // don't log error details from client calls in events
    klog.V(4).Infof("failed to create volume: %v", err)
    return nil, 0, "", fmt.Errorf("failed to create volume: see kube-controller-manager.log for details")
}

The detail moves to a verbosity-4 log line on the control plane node, where a cluster administrator can read it and an ordinary user can't. The user-facing error becomes a pointer to that log. Roughly a hundred and twenty lines changed across the four plugins, all of it the same substitution.

It closes the exfiltration channel. The SSRF is still there. kube-controller-manager will still connect to whatever host you put in resturl; you just can't see what it found. The advisory lists mitigations for people who couldn't upgrade straight away, and they are about the request side rather than the response side. Block usage of the affected volume types with PodSecurityPolicy or an admission controller. Restrict who can write StorageClasses through RBAC. Put protections on endpoints reachable from the control plane host.

The CVSS is AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:N/A:N, which lands as medium. High attack complexity, low privileges required, and a scope change. The scope changes because you reach out of your box into the control plane's network. Fixed in v1.18.1, v1.17.5, v1.16.9 and v1.15.12.