The status of kernel hardening
One area of progress is in the integration of GCC plugins into the build system. The plugins in the kernel now are mostly examples, but there will be more interesting ones coming in the future. Plugins are currently supported for the x86, arm, and arm64 architectures; he would like to see that list grow, but he needs help from the architecture maintainers to validate the changes. Plugins are also not yet used for routine kernel compile testing, since it is hard to get the relevant sites to install the needed dependencies.
Linus asked how much plugins would slow the kernel build process; linux-next maintainer Stephen Rothwell also expressed interest in that question, noting that "some of us do compiles all day." Kees responded that there hadn't been a lot of benchmarking done, but that the cost was "not negligible." It is, though, an important part of protecting the kernel.
Probabilistic protections
The kernel has adopted a number of probabilistic protections over the last year. These protections only work if the attacker doesn't know something about the system. They include kernel address-space layout randomization (KASLR) and stack protection. Probabilistic protections can be defeated if the information leaks out, but they are still effective and worth doing.
One improvement is in the randomization of the kernel text base; it was added to arm64 in the 4.6 release and MIPS in 4.7. But the text base is only the beginning, more memory areas need to be randomized. One possibility is to randomize the kernel's link order at boot time. That would be a lot of work, but it would mean that an attacker would need more than a single information leak to defeat the whole thing.
Linus said that randomization can be a pain for debugging; it is not fun to
track down a problem that only happens in one boot out of every 300 or so. Al
Viro worried that changing the link order would also change the order in
which the kernel's initialization calls are made, with unpredictable
effects. Kees responded that this particular change isn't coming anytime
soon. Andi Kleen suggested just doing the link randomization and dropping
KASLR altogether; the kernel's addresses tend to leak via all kinds of paths
anyway. Linus responded that, while the address leaks are being plugged
over time, KASLR does indeed work poorly against local
attackers, but it is more useful against remote attackers.
Kees went on to say that the kernel got KASLR for its memory areas in 4.8 for the x86_64 architecture.
Work is being done on free-list randomization, which makes the layout of the heap less predictable. Perhaps more controversial is struct layout randomization. That cannot be done in a general way without causing all kinds of problems, but there is one place where it is especially useful: structs consisting of only function pointers. Such structs are one of the most prized targets for attackers, and the kernel has a lot of them. A GCC plugin can be used to detect these structures and randomize their order. In general, the kernel shouldn't care about that ordering, and changing it should not have performance effects.
Linus was not entirely convinced; he said that most people are running distributor kernels, so the specific ordering used will always be available to an attacker. The value, Kees responded, is forcing attackers to identify specific kernel builds; that is "excruciating" for them. It greatly expands the number of settings their exploit has to work in.
Deterministic protections
While probabilistic protections only work if some key data remains secret, Kees said, deterministic protections work all the time. These include things like read-only memory; if memory is read-only, it is always protected from being changed. Bounds checking to head off overflows is another form of deterministic protection.
One useful protection is the CONFIG_DEBUG_RODATA configuration option which, Kees said, is badly named. It ensures that executable memory is not writable anywhere in the kernel; it should be mandatory on all systems that support it. It is turned on by default on the x86 architecture as of 4.6, and will be for arm64 as of 4.9.
Another important protection is protection of user space against access by
the processor when it is running in a privileged mode. By far the most
common way
to exploit the kernel, he said, is to get the kernel to execute code
that has been placed somewhere in user-space memory. If the kernel cannot
access that memory, such exploits will not work. Processor vendors have
worked to provide such protections using technologies like SMAP and SMEP
(on x86) and PAN (on ARM), but there is a problem: such protections are not
widely available yet. There are no Xeon processors with
SMEP SMAP on the
market; PAN was added to the ARMv8.1 specification, but no hardware is
shipping yet.
So, he said, the kernel needs emulation of those features instead; it is, he said, a fundamental need. Linus replied, though, that he hates the emulation patches with a passion. And, he said, it is not necessary, in that the kernel's support for SMEP protects systems that lack SMEP too. That is because it forces all kernel paths that access user-space memory to be verified, preventing accidental accesses. So, he said, the emulation does not buy much. Kees disagreed, saying that the emulation can protect systems that will not have hardware protection for a few years yet.
Work is being done on hardened usercopy, which performs sanity checking on operations that copy data to and from user space. The current patch set contains about 1/3 of the PaX USERCOPY protections, which is a start. Next steps include segregating the slab caches; objects that are exposed to user space should be stored apart from those that are purely internal to the kernel. The problem here is to find a clean way to deal with exceptions. An inode object, for example, should not be copyable to or from user space, but there can be reasons to copy the file name stored within that structure. The PaX code does such copies by way of the stack, which is generally seen as being the wrong approach; Kees said that a more maintainable API for exceptions is needed. Linus added that this kind of problem is exactly why he has never seriously considered merging the grsecurity patch set; it's full of "this kind of craziness."
Memory wiping is useful, in that it can block information leaks and some types of use-after-free exploits. The slab allocator can do poisoning of memory, but not zeroing, which would be nice to add. After Linus asked, Kees said that the advantage of zeroing is that the kernel often needs to allocate zeroed pages; if freed memory has already been zeroed, those allocations can be optimized. A problem with zeroing is that some objects are allocated and freed so often that the performance hit becomes prohibitive, so there needs to be a way to make exceptions. There is a GCC plugin out there to do stack clearing, which is worth looking at.
"Constification" — making unchanging data constant — can protect against some types of exploits. The lowest-hanging fruit here is structs full of function pointers; the "constify" GCC plugin tries to make those const by default. As of 4.6, the kernel can make data read-only after initialization, but that feature is not yet widely used in the kernel. There would be value in identifying "write-rarely" data that would be read-only most of the time, and only made writable during explicit updates.
Kees's final topic was reference-count hardening. If an attacker is able to force a reference count to overflow, a use-after-free exploit is usually not far away. Most of these attacks can be blocked if atomic variables can be kept from overflowing. The hardening patches out there will kill the responsible process when an overflow is detected, and the counter involved is permanently blocked at a high value. In this way, an exploit is downgraded to a denial-of-service situation.
Kees's slides are
available for the curious.
Index entries for this article | |
---|---|
Kernel | Security/Security technologies |
Security | Linux kernel/Hardening |
Conference | Kernel Summit/2016 |
Posted Nov 2, 2016 22:24 UTC (Wed)
by kees (subscriber, #27264)
[Link] (2 responses)
Posted Nov 4, 2016 9:54 UTC (Fri)
by corsac (subscriber, #49696)
[Link] (1 responses)
- http://ark.intel.com/products/family/87722/Intel-Xeon-Pro...
I couldn't find SMEP/SMAP in E5 and E7 Xeons, even v4 (Broadwell) but it might be my fault.
Posted Nov 10, 2016 11:44 UTC (Thu)
by nim (subscriber, #102653)
[Link]
https://software.intel.com/en-us/articles/intel-xeon-proc...
You can also search the CPUID database at cpu-world.com and notice that the E5-1600/2600 v4 CPUs have SMAP as reported by CPUID.
Posted Nov 10, 2016 11:39 UTC (Thu)
by ajdlinux (subscriber, #82125)
[Link]
Posted Nov 10, 2016 16:14 UTC (Thu)
by l1k (subscriber, #112260)
[Link] (18 responses)
If reordering results in members not being aligned, the struct's memory footprint will grow. Does the gcc plugin account for that and try to optimize the order or is it just completely random? Depending on how frequently the struct is used and how much memory is available, an argument could be made that reordering *can* "have performance effects".
Posted Nov 10, 2016 20:24 UTC (Thu)
by joib (subscriber, #8541)
[Link] (17 responses)
>If reordering results in members not being aligned, the struct's memory footprint will grow. Does the gcc plugin account for that and try to optimize the order or is it just completely random? Depending on how frequently the struct is used and how much memory is available, an argument could be made that reordering *can* "have performance effects".
It says in the article that it only affects structs which contain only function pointers (which all have the same size, AFAICS). Furthermore, the slides say that it can be configured to randomize only within a cache line.
Posted Nov 10, 2016 23:21 UTC (Thu)
by spender (guest, #23067)
[Link] (16 responses)
Of course, given that it's a pure copy+paste lift from grsecurity, one has to wonder how upstream will be able to maintain it themselves given that no KSPP contributor has written a GCC plugin, let alone one of the level of complexity of what's present in grsecurity. What if during 4.10-rc there is some problem that needs to be fixed before a final release can be made? Our 4.10 release will come out after the 4.10 upstream release. Are they simply going to disable the security for all users until they can copy+paste a fix from us? I don't think this is how upstream kernel development is supposed to work -- it seems to be missing the actual "development" to me.
-Brad
Posted Nov 11, 2016 6:44 UTC (Fri)
by tao (subscriber, #17563)
[Link] (15 responses)
Posted Nov 11, 2016 11:03 UTC (Fri)
by PaXTeam (guest, #24616)
[Link] (14 responses)
Posted Nov 11, 2016 17:08 UTC (Fri)
by tao (subscriber, #17563)
[Link] (13 responses)
Posted Nov 11, 2016 17:24 UTC (Fri)
by edgewood (subscriber, #1123)
[Link] (11 responses)
Posted Nov 11, 2016 17:37 UTC (Fri)
by PaXTeam (guest, #24616)
[Link] (10 responses)
Posted Nov 12, 2016 1:28 UTC (Sat)
by edgewood (subscriber, #1123)
[Link] (9 responses)
If you think the code is valuable, but don't want to do the hard work of getting it integrated upstream (and I don't dispute that it could be hard), that's your choice, but that's exactly the part employers find valuable.
Posted Nov 12, 2016 10:56 UTC (Sat)
by PaXTeam (guest, #24616)
[Link] (8 responses)
Posted Nov 12, 2016 12:55 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link] (7 responses)
So a kernel patch not contributed doesn't make you a kernel contributor. Got it.
Less snarkily, there has to at least be some attempt. A potential employer can see that you have good skills at development, but if you show no aptitude at getting things merged, how can they know that giving you money would actually get things upstream rather than just the continuing status quo? There's a chicken and an egg here that is no different than having some kind of contribution record being at least some kind of evidence for backing up an "I am an effective FOSS contributor" claim during an interview.
Posted Nov 12, 2016 13:57 UTC (Sat)
by PaXTeam (guest, #24616)
[Link] (6 responses)
exactly which is why you look at stuff that got contributed, not stuff that didn't.
> [...]how can they know that giving you money would actually get things upstream[...]
look at past contributions and/or use probation time which they'd require regardless. not that it's a silver bullet mind you, having successfully upstreamed code is no guarantee for competence and future successes. a recent example is http://www.openwall.com/lists/kernel-hardening/2016/11/11/26 where Peter Zijlstra told Kees Cook in no uncertain words that "To me that shows you shouldn't be allowed near atomic_t" and basically shot down their current efforts of upstreaming my REFCOUNT implementation as is. all that's because, suprise surprise, my code wasn't written for upstreaming purposes and any competent developer should have recognized that (or should just have listened to me) and have done it differently. to his credit, Peter is slowly figuring it out and the crack team of the KSPP is beginning to realize just how much more work than mindlessly copy-pasting and renaming it will be. at least they'll be able to claim to actually have done some real work for a change.
> [...] having some kind of contribution record being at least some kind of evidence for backing up
i have plenty, do you? your insinuated ad hominem aside, you guys completely lost track of the problem here: it's about you making demands on our free time (PaX/grsecurity have been hobby projects) and then blame us that the universe didn't give us more than 24h of it in a day.
Posted Nov 12, 2016 15:03 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link]
I didn't say that it was.
> i have plenty, do you?
Kernel? 6 or so patches. Elsewhere, yeah.
> your insinuated ad hominem aside, you guys completely lost track of the problem here: it's about you making demands on our free time (PaX/grsecurity have been hobby projects) and then blame us that the universe didn't give us more than 24h of it in a day.
I'm not saying you have zero evidence of contribution, but the proportion may be what is lacking. But there's a negotiation that needs to happen. Your time for their money. You're seeing requests to upstream as demands on your time and you're, rightfully, demanding money for that time, but it seems that no one is willing to blink first. In your case, it seems, to me, to be stubbornness, and on theirs it can be anything from dislike of attitude, dislike of your anonymity, or ignorance of your requests. I assume you're actively seeking money for upstreaming your efforts, but I guess the question is how much do you want the kernel to have your changes versus new development. Not sure there's a right answer to that one.
Posted Nov 12, 2016 18:05 UTC (Sat)
by tao (subscriber, #17563)
[Link] (4 responses)
If you don't have time to submit your patches (yet you seem to have plenty of time to develop your patches and keep them up to date with upstream; usually upstreaming your patches saves time since it saves you the effort of keeping up to speed with the kernel), then either let someone else do it *and help them do so instead of whining whenever they mistakes* or accept that others get their solutions (which might be misunderstood versions of your code) merged instead. It's not really a complicated concept.
I'm sure that for instance Red Hat would be willing to sponsor you to merge the upstreamable parts of your code.
As it is now, however, *every* *single* kernel release (and not just the major ones, but the minor ones as well) you complain. You complain about every single thing Greg KH does. You complain about every single security related effort done by someone else than you. You complain if Greg releases fixes too often. You complain if he doesn't release often enough. You complain if people merge your patches. You complain if they implement something else instead. You complain if they don't merge them at all.
I'm sure that Kees would be more than willing to move over and yield the position of security guy to you. I mean it's not exactly news that you're competent in this area; far more competent than he is. But he's actually doing stuff that's getting upstreamed. You're standing on the side, shouting "you're doing it wrong, I know how to do it right, but I don't have the time to help".
Every time you get annoyed with the flawed implementation of your mitigation, instead of posting 5 posts about it on LWN, submit a revert of the flawed implementation + your correct version to LKML. The time you save on not posting should be more than plenty to fire off those e-mails; you already have the patches, after all, and you've already reviewed the flawed implementation.
I respect your work, I really do. But it's hard to take your concern for the kernel security seriously when you make no effort to actually mitigate the flaws in upstream. Because no matter how good your patch set is, most distros, most companies, even most users who build their own kernels, will use upstream kernels (though companies use upstream kernels from a gazillion years ago).
Posted Nov 12, 2016 20:09 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link]
Nobody has any obligation to do something that's going to result in them being insulted and abused, and that's pretty much a given for taking this kind of thing to LKML (even if there's a fairly significant level of irony involved there)
Posted Nov 13, 2016 13:03 UTC (Sun)
by spender (guest, #23067)
[Link] (2 responses)
1) I have not received any money from the Linux Foundation, have never sought any money from the Linux Foundation, and would refuse to accept their money even if it were offered
2) I am not *ever* going to perform any work for the Linux Foundation in any form, paid or unpaid
3) One of the Linux Foundation platinum members is one of the companies responsible for the removal of grsecurity's stable patches
4) Said company cost us thousands of dollars in legal fees alone and hasn't even had the decency to contact us to apologize for their lawyer's lies (while they were claiming the trademark use was nominative, archive.org shows they were purging their website's promotional materials of references to grsecurity).
5) After stable patches were removed and said company denied access to them, guess who became the largest contributor to the KSPP?
6) No one from KSPP has ever contributed anything to grsecurity
7) Not one of the upstream developers who have spouted BS about our work for over a decade (that even continues in articles like this), who are now allowing in verbatim copy+pastes, making no non-bikeshedding improvements to it at all, has ever contacted either of us to apologize
8) KSPP's bikeshedding is increasing the amount of time required to produce our free test patches, likely partially done to force free work out of us (auditing of their modifications, free fixes, etc). I am telling everyone now that my response to this attempt to force free labor out of me by BSD-style upstream churn will never be to contribute. I am not stupid -- the questions in private are posed as "we would like to minimize your integration work" but the intent of "the LF wants free work to know what we screwed up in blindly copy+pasting" is clear.
9) KSPP is merely a short-sighted reputation saving attempt for the Linux Foundation. Unlike the nonsense spouted by the CII, contributing to it or receiving funding from them is not in our best interests for the future of grsecurity or for our development of new security technologies (which the KSPP is doing none of). The only people who have the best interests of our long term stability is us.
10) Working with upstream offers us *nothing* except the promise of spending thousands of hours of my *free time* working with a bunch of jerks who don't understand security -- we have been doing kernel development for 16 years now just fine without them.
So please, next time you want to reply to a post of mine because you can't stand reading facts that no one else on this site is going to share, feel free to:
If you hate complaining so much, to me you seem like the worst kind of complainer, one who shoots the messenger to maintain the ridiculous status quo and the unethical liars who create it.
Thanks,
Posted Mar 2, 2017 16:34 UTC (Thu)
by stephenm (guest, #114284)
[Link] (1 responses)
Posted Mar 2, 2017 17:14 UTC (Thu)
by corbet (editor, #1)
[Link]
Posted Nov 11, 2016 17:35 UTC (Fri)
by PaXTeam (guest, #24616)
[Link]
Posted Nov 27, 2016 16:19 UTC (Sun)
by spender (guest, #23067)
[Link]
While looking at commits like:
All done in a rush for 4.9, where they'll brag about how fast CONFIG_VMAP_STACK is while ignoring all these performance-damaging changes where grsecurity would have handled them much faster (by just switching to a lowmem alias in the scatterlist API, a few instructions at most) and also not DoS users' machines on the many instances we discovered via a smart GCC IPA plugin where stack addresses are still used that upstream hasn't fixed. Unfortunately for us, with all this junk code being introduced in 4.9, our performance for these cases is lowered to that of the upstream kernel unless I wanted to revert hundreds of these patches and maintain them into the future.
I'll likely write a blog on this topic after 4.9 is released, because there's a lot more to discuss about this particular feature and how it compares to what we've had in grsecurity for years.
-Brad
The status of kernel hardening
The status of kernel hardening
- http://ark.intel.com/products/family/88210/Intel-Xeon-Pro...
The status of kernel hardening
The status of kernel hardening
struct layout randomization
struct layout randomization
struct layout randomization
struct layout randomization
struct layout randomization
struct layout randomization
Any kernel developer who shows the ability to work with the kernel community and get his or her changes accepted, that is.
struct layout randomization
struct layout randomization
Of course the code you didn't submit counts. "You miss 100% of the shots you don't take."
struct layout randomization
struct layout randomization
struct layout randomization
struct layout randomization
> an "I am an effective FOSS contributor" claim during an interview.
struct layout randomization
struct layout randomization
struct layout randomization
struct layout randomization
1) Keep your unwanted opinions to yourself
or
2) Place me on a killfile and save us both the trouble, as I will now do for you
-Brad
struct layout randomization
This sort of comment would be more helpful if you said what you think the Linux Foundation should be doing differently? The post you're responding to says that LF money won't be accepted, so that's not it.
struct layout randomization
struct layout randomization
The status of kernel hardening
> The PaX code does such copies by way of the stack, which is generally seen as being the wrong approach; Kees said that a more maintainable API for exceptions is needed. Linus added that this kind of problem is exactly why he has never seriously considered merging the grsecurity patch set; it's full of "this kind of craziness."
https://git.kernel.org/cgit/linux/kernel/git/torvalds/lin...
which "fix" stack scatterlist uses by performing atomic allocations, copying memory, and freeing the allocation; i.e. managing to effectively implement bounce buffers much slower than the "craziness" they identified in grsecurity.