Exploiting a use-after-free race in the Linux kernel's vivid driver

There is a Linux driver whose entire job is to pretend. vivid emulates video4linux hardware (capture devices, output devices, radio receivers and transmitters, a software-defined radio). You can develop and test an application against /dev/video0 without owning any of that gear. The kernel documentation treats it as a test fixture.

Needing no hardware is exactly the property that makes it interesting. Ubuntu, Debian, Arch Linux, SUSE Linux Enterprise and openSUSE all ship it as a module, CONFIG_VIDEO_VIVID=m.

Alexander Popov, working at Positive Technologies, found three race conditions in it at the end of 2019 and wrote the patch. He then wrote a local privilege escalation exploit for x86_64. It beats KASLR, SMEP and SMAP on Ubuntu Server 18.04. His write-up is one of the better pieces of kernel-exploitation writing on the public internet. This post is a summary of it, not a replacement.

He got there with syzkaller plus some custom changes to the kernel source. That produced a suspicious crash: KASAN reporting a use-after-free during linked list manipulation in vid_cap_buf_queue(). Following that backwards led a long way from the memory corruption itself.

Dropping the mutex#

Three functions in drivers/media/platform/vivid make the same mistake: vivid_stop_generating_vid_cap(), vivid_stop_generating_vid_out() and sdr_cap_stop_streaming(). All three are called with vivid_dev.mutex already held, because that's what the caller does when streaming is being stopped. All three then need to shut down a kernel thread. Here is the capture one:

/* shutdown control thread */
vivid_grab_controls(dev, false);
mutex_unlock(&dev->mutex);
kthread_stop(dev->kthread_vid_cap);
dev->kthread_vid_cap = NULL;
mutex_lock(&dev->mutex);

You can see the reasoning. kthread_stop() sets a flag and then blocks until the thread actually exits. The thread's main loop takes dev->mutex on every iteration. Call kthread_stop() while holding the mutex and the thread parks forever, waiting for a lock held by the task waiting for the thread. So: drop the lock, stop the thread, take the lock back. It reads like good hygiene.

The problem is what a mutex is for. Dropping it doesn't hand it to the kthread. It hands it to whoever asks next. On a busy machine that can easily be another userspace caller arriving through vb2_fop_read(). That caller walks in and manipulates the buffer queue, in a window where the driver believes nothing is happening.

Winning the race#

The proof-of-concept crasher is about as simple as kernel race code gets. Two pthreads, pinned to separate CPUs with sched_setaffinity() so they really run in parallel, and then:

for (loop = 0; loop < LOOP_N; loop++) {
	int fd = 0;

	fd = open("/dev/video0", O_RDWR);
	if (fd < 0)
		err_exit("[-] open /dev/video0");
	read(fd, buf, 0xfffded);
	close(fd);
}

That's the whole race. The first read on a freshly opened descriptor makes V4L2 call vb2_core_streamon(), which starts streaming. Releasing the last reference to the file makes it call __vb2_queue_cancel(), which stops streaming and runs the code above. If a read from the other thread wins the race for that briefly unlocked mutex, it calls vb2_core_qbuf(). That adds an extra vb2_buffer to vb2_queue.queued_list that nobody is expecting.

Then the close path continues. vb2_core_queue_release() runs, and __vb2_queue_free() frees the buffers, including the one that arrived late. The driver carries on holding a pointer to it. On the next iteration of the loop, streaming starts again and vivid writes through that pointer. KASAN catches it:

BUG: KASAN: use-after-free in vid_cap_buf_queue+0x188/0x1c0
Write of size 8 at addr ffff8880798223a0 by task v4l2-crasher/300
...
The buggy address belongs to the object at ffff888079822000
 which belongs to the cache kmalloc-1k of size 1024

kmalloc-1k is a quiet cache. Not much in the kernel allocates objects that size. An attacker spraying it has less competition, and better odds of landing controlled bytes in the freed slot. Good for exploitation, as Popov drily notes.

On Ubuntu you don't need to be anyone special to do this. The distro applies an RW ACL to /dev/video0 for the logged-in user, so getfacl shows a plain user with read-write on the device. The one thing holding the severity down is that Popov couldn't find a way to make the vulnerable driver autoload. It has to already be there. That limitation is why the kernel security team was happy for him to do full disclosure rather than a quiet fix.

Exploitation#

The short version, because the long version is his to tell. The freed object is overwritten by heap spraying with setxattr(), held in place by userfaultfd(). The technique comes from Vitaly Nikolenko. userfaultfd() lets you stall the kernel mid-copy, so the allocation stays alive as long as you decline to service the fault. The vulnerable buffer isn't the last thing freed, so one allocation isn't enough. The exploit runs a pool of 44 spraying pthreads that each call setxattr() and hang, spread across CPUs because slab caches are per-CPU.

The control-flow hijack goes through vb2_buffer.vb2_queue->mem_ops->vaddr, a function pointer. Its argument comes from vb2_buffer.planes[0].mem_priv, also attacker-controlled. Then a problem. Pointing vb2_queue at an mmap'ed userspace address produces "unable to handle page fault". The dereference happens in kernel thread context, where the exploit's userspace isn't mapped at all. Andrey Konovalov suggested the way out, which Popov credits as xairy's method. Put the payload on the kernel stack instead, using any syscall that does copy_from_user() into stack storage and freezing it there with userfaultfd(). adjtimex() fits.

That leaves the question of where the kernel stack is. It comes from the bug itself. Before the use-after-free, the exploit trips a WARN_ON in __vb2_queue_cancel(). That is the one whose comment politely tells driver authors they aren't cleaning up properly in stop_streaming(). The resulting register dump goes to the kernel log, which unprivileged users can read on Ubuntu Server. A parser thread pulls RSP out of it to compute the stack top, and R11 to compute the KASLR offset. With those known, the exploit places a vb2_queue and a vb2_mem_ops at predicted stack addresses. It sets vaddr to a push rdi; pop rsp gadget and pivots onto a ROP chain. The chain calls run_cmd() with a shell command, then do_task_dead() to stop the hijacked kthread from crashing the box afterwards. Fifty threads in five roles, synchronised on six pthread barriers.

Setting /proc/sys/vm/unprivileged_userfaultfd to 0 breaks the trick that holds the payload in place, and kernel.dmesg_restrict = 1 breaks the infoleak. Popov points out that on Ubuntu members of the adm group can read the same log out of /var/log/syslog anyway.

The fix#

The fix has two halves. First, stop unlocking the mutex on streaming stop:

 /* shutdown control thread */
 vivid_grab_controls(dev, false);
-mutex_unlock(&dev->mutex);
 kthread_stop(dev->kthread_vid_cap);
 dev->kthread_vid_cap = NULL;
-mutex_lock(&dev->mutex);

Second, make the kthread loop stop insisting on the lock. Then the deadlock the original code was avoiding never happens:

 for (;;) {
 	try_to_freeze();
 	if (kthread_should_stop())
 		break;
-	mutex_lock(&dev->mutex);
+	if (!mutex_trylock(&dev->mutex)) {
+		schedule_timeout_uninterruptible(1);
+		continue;
+	}

It took four versions of that patch to get right, which is the part I enjoyed most. Popov sent the first one to security@kernel.org, and Linus Torvalds replied in under two hours. Version one had no sleep at all, just continue. Linus pointed out it was a busy loop that could deadlock on a non-preemptible kernel. Popov built a kernel with CONFIG_PREEMPT_NONE=y, ran his own crasher, and watched it deadlock exactly as described. Version two used schedule_timeout_interruptible() because that's what the rest of vivid-kthread-cap.c used. Maintainers asked for plain schedule_timeout() instead, since kernel threads shouldn't be taking signals. Version three did that. Then Linus came back:

I just realized that this too is wrong. It _works_, but because it doesn't actually set the task state to anything particular before scheduling, it's basically pointless. It calls the scheduler, but it won't delay anything, because the task stays runnable.

So the third patch worked by luck. The final version uses schedule_timeout_uninterruptible(1), which actually sets TASK_UNINTERRUPTIBLE first. Popov then sent a patch adding a warning for that API misuse. Steven Rostedt told him the behaviour is known and intended, so he settled for improving the documentation instead.

The races are CVE-2019-18683, rated 7.0 by NVD with the vector AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H. Local, high attack complexity, low privileges required, total compromise if you land it. NVD describes the affected kernels as everything through 5.3.8, with the vulnerable code going back to 3.18. Five years of a driver that doesn't drive anything.