|
|
Subscribe / Log in / New account

Fighting Spectre with cache flushes

By Jonathan Corbet
October 15, 2018
One of the more difficult aspects of the Spectre hardware vulnerability is finding all of the locations in the code that might be exploitable. There are many locations that look vulnerable that aren't, and others that are exploitable without being obvious. It has long been clear that finding all of the exploitable spots is a long-term task, and keeping new ones from being introduced will not be easy. But there may be a simple technique that can block a large subset of the possible exploits with a minimal cost.

Speculative-execution vulnerabilities are only exploitable if they leave a sign somewhere else in the system. As a general rule, that "somewhere else" is the CPU's memory cache. Speculative execution can be used to load data into the cache (or not) depending on the value of the data the attacker is trying to exfiltrate; timing attacks can then be employed to query the state of the cache and complete the attack. This side channel is a necessary part of any speculative-execution exploit.

It has thus been clear from the beginning that one way of blocking these attacks is to flush the memory caches at well-chosen times, clearing out the exfiltrated information before the attacker can get to it. That is, unfortunately, an expensive thing to do. Flushing the cache after every system call would likely block a wide range of speculative attacks, but it would also slow the system to the point that users would be looking for ways to turn the mechanism off. Security is all-important — except when you have to get some work done.

Kristen Carlson Accardi recently posted a patch that is based on an interesting observation. Attacks using speculative execution involve convincing the processor to speculate down a path that non-speculative execution will not follow. For example, a kernel function may contain a bounds check that will prevent the code from accessing beyond the end of an array, causing an error to be returned instead. An attack using the Spectre vulnerability will bypass that check speculatively, accessing data that the code was specifically (and correctly) written not to access.

In other words, the attack is doing something speculatively that, when the speculation is unwound, results in an error return to the calling program — but, by then, the damage is done. The error return is a clue that there maybe something inappropriate going on. So Accardi's patch will, in the case of certain error returns from system calls, flush the L1 processor cache before returning to user space. In particular, the core of the change looks like this:

    __visible inline void l1_cache_flush(struct pt_regs *regs)
    {
	if (IS_ENABLED(CONFIG_SYSCALL_FLUSH) &&
	    static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
	    if (regs->ax == 0 || regs->ax == -EAGAIN ||
		regs->ax == -EEXIST || regs->ax == -ENOENT ||
		regs->ax == -EXDEV || regs->ax == -ETIMEDOUT ||
		regs->ax == -ENOTCONN || regs->ax == -EINPROGRESS)
			return;

	    wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
	}
    }

The code exempts some of the most common errors from the cache-flush policy, which makes sense. Errors like EAGAIN and ENOENT are common in normal program execution but are not the sort of errors that are likely to be generated by speculative attacks; one would expect an error like EINVAL in such cases. So exempting those errors should significantly reduce the cost of this mitigation without significantly reducing the protection that it provides.

(Of course, the code as written above doesn't quite work right, as was pointed out by Thomas Gleixner, but the fix is easy and the posted patch shows the desired result.)

Alan Cox argued for this patch, saying:

The current process of trying to find them all with smatch and the like is a game of whack-a-mole that will go on for a long long time. In the meantime (and until the tools get better) it's nice to have an option that takes a totally non-hot path (the fast path change is a single test for >= 0) and provides additional defence in depth.

Andy Lutomirski is not convinced, though. He argued that there are a number of possible ways around this protection. An attacker running on a hyperthreaded sibling could attempt to get the data out of the L1 cache between the speculative exploit and the cache flush, though Cox said that the time window available would be difficult to hit. Fancier techniques, such as loading the cache lines of interest onto a different CPU and watching to see when they are "stolen" by the CPU running the attack could be attempted. Or perhaps the data of interest is still in the L2 cache and could be probed for there. In the end, he said:

Adding ~1600 cycles plus the slowdown due to the fact that the cache got flushed to a code path that we hope isn't hot to mitigate one particular means of exploiting potential bugs seems a bit dubious to me.

Answering Lutomirski's criticisms is probably necessary to get this patch set merged. Doing so would require providing some numbers for what the overhead of this change really is; Cox claimed that it is "pretty much zero" but no hard numbers have been posted. The other useful piece would be to show some current exploits that would be blocked by this change. If that information can be provided, though (and the bug in the patch fixed), flushing the L1 cache could yet prove to be a relatively cheap and effective way to block Spectre exploits that have not yet been fixed by more direct means. As a way of hardening the system overall, it seems worthy of consideration.

Index entries for this article
KernelSecurity/Meltdown and Spectre
SecurityMeltdown and Spectre


to post comments

Fighting Spectre with cache flushes

Posted Oct 16, 2018 6:46 UTC (Tue) by blackwood (guest, #44174) [Link] (3 responses)

Instead of a global whitelist for all syscalls/ioctls, why not a special errno flag to indicate an expected failure? Expected here meaning that it's expected to hit this as part of normal uapi usage.

We use EINVAL in drm to iteratively discover an optimal configuration in the atomic display api. But it's only one specific source of EINVAL, all others should still be treated as possible exploits. So anything at the global level, or even just at the ioctl level, isn't a fine-grained enough filter. And I suspect there's lots of other places.

The flag would also serve as nice documentation for the fast-path error case.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 7:12 UTC (Tue) by josh (subscriber, #17465) [Link] (2 responses)

Because going through all kernel entry points (syscalls, ioctls, etc) and marking "expected failures" would defeat the purpose of catching the vast majority of cases in one place with one cheap test.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 8:07 UTC (Tue) by lkundrak (subscriber, #43452) [Link] (1 responses)

You wouldn't have to walk through all of them, merely the ones where the expected error would need to be whitelisted in order to avoid a performance hit.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 10:38 UTC (Tue) by blackwood (guest, #44174) [Link]

Yeah, that's what I had in mind. E.g. for EEXISTS I suspect the only case we really care about for performance is the dentry cache hit fastpath, and nothing else. That would greatly reduce the amount of code you have to audit for spectre hardening even more, much better than what the current patch already achieves.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 7:13 UTC (Tue) by josh (subscriber, #17465) [Link]

This seems like an absolutely brilliant idea; simple in hindsight, elegant, effective, and likely to catch the vast majority of issues.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 7:44 UTC (Tue) by johill (subscriber, #25196) [Link] (2 responses)

I guess the same should be added to netlink, and I'm missing -ERANGE in the list?

Fighting Spectre with cache flushes

Posted Oct 22, 2018 7:04 UTC (Mon) by cpitrat (subscriber, #116459) [Link] (1 responses)

You mean you would want to flush the cache for -ERANGE right ? The mentioned errors are the ones for which the cache is NOT flushed, so that's OK.

Fighting Spectre with cache flushes

Posted Oct 22, 2018 13:00 UTC (Mon) by johill (subscriber, #25196) [Link]

Oh, right, that's the negative list, so yeah ... I guess I wasn't paying attention.

Regarding netlink: "syscall" exit point is different there (it's a message reporting the error, not the syscall itself), so similar code would have to be added there

Instead of flushing, return on a different CPU?

Posted Oct 16, 2018 8:45 UTC (Tue) by epa (subscriber, #39769) [Link] (1 responses)

On multi-CPU systems (where the CPUs don't share a level 1 cache, that is, they are not just hyperthreads on the same core) you could kick the calling process off the current CPU when it hits a system call error, and let it continue executing on a different CPU at least for a short while.

Instead of flushing, return on a different CPU?

Posted Oct 22, 2018 7:05 UTC (Mon) by cpitrat (subscriber, #116459) [Link]

Which would be worse in terms of performances as on top of impacting this process, you would potentially kick another one (on the destination core)

Fighting Spectre with cache flushes

Posted Oct 16, 2018 8:46 UTC (Tue) by anton (subscriber, #25547) [Link] (5 responses)

Lutomirski is probably right that flushing the L1 is not enough; extracting information from the L2 or L3 may take longer because virtual-to-physical mapping distributes the accesses over more potential places, but it's probably still possible.

If the non-whitelisted error returns are as rare as Alan Cox suggests, flushing the L2 and the L3 should not impose a significant performance penalty, either.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 10:19 UTC (Tue) by epa (subscriber, #39769) [Link]

I think it's the nature of Spectre and similar hardware bugs that you can never defend against them completely. Making the attacker's life ten times harder seems like a very worthwhile improvement.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 11:50 UTC (Tue) by nhaehnle (subscriber, #114772) [Link]

Higher-level caches are shared, so flushing them would affect other threads as well. That'd be a rather big performance issue.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 20:11 UTC (Tue) by luto (subscriber, #39314) [Link] (2 responses)

L2 and L3 flushes are (I think) a per-package thing, not a per-core thing. WBINVD in particular is *incredibly* expensive. Letting an unprivileged program trigger WBINVD would be a major DoS problem.

Fighting Spectre with cache flushes

Posted Oct 17, 2018 8:30 UTC (Wed) by anton (subscriber, #25547) [Link] (1 responses)

L2 is per-core on mainstream Intel and AMD CPUs. L3 is per-package. Overall, with 20-30GB/s of memory bandwidth and 2-16MB L3 cache, the flush costs on the order of 1ms for refilling the L3; I doubt that the WBINVD itself makes that much worse. But if there is <1 such error per second, the performance impact will only be <0.1%.

Concerning DoS attacks, unprivileged programs can thrash the caches anyway (with ordinary memory accesses). Is it easier for a remote attacker to trigger such OS error returns than to induce the attacked program to perform thrashing memory accesses? It could be.

Another, cheaper, thing in the same vein that could be done is to CLFLUSH(OPT) the speculatively accessed memory. Problems: This would need compiler support to make it generally applicable with little programmer effort; CLFLUSH reports page faults, so probably has to be protected from that (how?); it would open a side channel that allows determining which cache lines in one process conflict with which cache lines in another process (can attackers do something with that that they cannot do now?).

Fighting Spectre with cache flushes

Posted Oct 17, 2018 8:59 UTC (Wed) by matthias (subscriber, #94967) [Link]

The cost for refilling L3 will be higher. The 1ms is calculated on the assumption that the cache is read as one piece. However, the many cache misses due to the cold cache will stall the CPU and prevent it from using the full memory bandwidth.

Triggering this kind of error is trivial, while the cache replacement strategies of the CPU should take care of not throwing away heavily used content in the cache. Also trashing the cache is very slow, as by the nature of trashing every memory access is a cache miss.

TLB

Posted Oct 16, 2018 12:05 UTC (Tue) by geert (subscriber, #98403) [Link]

That "somewhere else" is not just the CPU's memory cache, but also other caches, like the TLB.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 12:28 UTC (Tue) by roc (subscriber, #30627) [Link] (3 responses)

I think there's a real risk this will hurt performance in unexpected ways. Two examples I know of off the top of my head:

* Some applications close all open file descriptors before they exec a subprocess. There are various ways to do this but I've seen code that uses getrlimit() to get the maximum FD number and then simply calls close() on every possible FD value up to that limit. (Maybe they don't want to depend on /proc being mounted.) That could be tens of thousands or maybe even a million close() calls returning EBADF. I guess this patch would make each of those significantly more expensive.

* Some applications want to probe their address space to see if memory is mapped at an address. You can do this using various syscalls that return ENOMEM or EFAULT if the memory is not mapped ... maybe faster and definitely more conveniently than using a signal handler. I guess those operations would get significantly slower with this patch.

I don't think it's safe to assume that a particular error path is not a hot path for any application.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 14:10 UTC (Tue) by epa (subscriber, #39769) [Link] (1 responses)

I imagine that particular calls like close() could be whitelisted to say that they don't need the cache flushing -- after someone has checked they are not vulnerable to known speculation attacks. Doing it with a whitelist is a safer approach than playing whack-a-mole trying to find all the cases where there might be a vulnerability.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 21:48 UTC (Tue) by roc (subscriber, #30627) [Link]

How do you develop and maintain such a whitelist? "Make it slow and wait for users to figure what went wrong and file bug reports" is not ideal.

> Doing it with a whitelist is a safer approach than playing whack-a-mole trying to find all the cases where there might be a vulnerability.

Maybe so if this approach was a comprehensive bulletproof fix to a defined class of Spectre vulnerabilities. But it's really just a heuristic to make exploitation harder in some set of cases that aren't easily characterized.

Adding patches to the kernel to achieve not-very-well-understood security benefits, in exchange for not-very-well-understood performance costs, should make people nervous.

Fighting Spectre with cache flushes

Posted Oct 16, 2018 19:19 UTC (Tue) by xorbe (guest, #3165) [Link]

I agree, the cost is "pretty much zero" until someone unexpectedly trips over it, then blames Linux as being dog slow compared to Windows. Seems like a bad strategy since the effects can't concretely quantified wrt other changes down the line.

Flush+Flush

Posted Oct 16, 2018 12:56 UTC (Tue) by azilian (guest, #47340) [Link]

It seams to me, that such flushes, may actually help if we are using flush + flush (https://gruss.cc/files/flushflush.pdf) attack.
And as it was stated above, hyperthreads are still a real problem. On busy systems it is not so hard to run your code in the same time window, when your first thread is working.

rowhammer

Posted Oct 16, 2018 13:53 UTC (Tue) by abatters (✭ supporter ✭, #6932) [Link]

Rowhammer required cache flushes, and the mitigation was to disallow cache flushes from certain environments. Could this be used to make rowhammer exploits possible again?


Copyright © 2018, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy