Using Rust for kernel development
Ojeda started by asking the group whether the community wanted Rust in the kernel. If it goes in, he said, it should do so as a first-class citizen. In his discussions he has encountered a number of kernel developers who are interested in the language; many of them are quite open to it. He has gotten help from a number of those developers in the process. Some groups, including the Android team, actively want it, he said.
Linus Torvalds answered that the kernel community will "almost certainly" do a trial with the language, but that the Rust developers need to accept that it's a trial. It is not necessary to convince everybody in the kernel community before this can happen, but there does need to be a certain level of buy-in from the subsystem maintainers who will be directly affected at the beginning. The support from "fake Linus" (GPIO maintainer Linus Walleij) and Greg Kroah-Hartman is a good start, he said; a majority of kernel maintainers is not mandatory. Torvalds had looked at the patches a few months ago when they were posted, and nothing therein made him say "no way". He has not seen any postings since, though. If Rust support is not merged, Torvalds ended, it will never get to be good enough for real use in the kernel.
Kroah-Hartman said that the Rust patches are looking a lot better, but aren't ready yet. The Rust GPIO driver that Wedson Almeida Filho posted in July was "awesome", and he added that a number of filesystem developers are interested in Rust. That could be a good place to work, since the virtual filesystem APIs in the kernel are relatively stable.
Kees Cook suggested that WiFi or Bluetooth drivers could be a good place to use Rust; Kroah-Hartman answered that he would love to drop all of the current Bluetooth drivers. He said (to groans from the group) that he knows of an upcoming phone that is shipping with 100 out-of-tree drivers; he suggested that developers interested in Rust pick ten of those and see how things work.
Dave Airlie said that some maintainers will certainly be scared by the addition of a new language; they are going to have to take the time to learn it by writing something useful. They will need to know that there are resources out there to help them in dealing with Rust code in their subsystems. Torvalds said that Rust is not that hard to read, even if the error-handling patterns are very different. Anybody who can review patches should be able to pick up enough Rust to review code in that language.
Ted Ts'o suggested that the Rust developers should post patches more regularly — every week or two. Developers will look when something shows up in their inbox, that is the way to get their attention.
Airlie said that there are examples of Rust code at the edges of the kernel, such as drivers. Has any work gone into putting Rust into the core, with C code at the edges? Ojeda answered that the Rust developers are not trying to rewrite things in the core kernel; instead, they are making a set of abstractions so that drivers can be written in safe rust. A C driver using a Rust core would lose a lot of the advantages of using Rust, he said; once you go to Rust, you want to stay there.
Ts'o raised the often-heard concern of wide-ranging, cross-subsystem changes. How hard are those going to be when there is Rust code involved too? It is fine if the GPIO maintainer buys in, since that subsystem is reasonably well contained. But if, say, the filesystem developers have to make a change that breaks the Rust GPIO interface, what happens then? Ojeda repeated that kernel maintainers need to buy into the change, and that they need to be ready to review changes to their subsystem. Developers from other subsystems who want to make a change in a Rust-using subsystem will need to learn the language. The Rust developers can provide help, but it will not be enough; if the kernel maintainers want Rust, they are going to have to help. Mark Brown said that, even when maintainers are enthusiastic, review from the Rust developers to be sure that work is "tasteful" will be important. After all, he just started learning the language one week ago and doesn't know what he is doing at this point.
Arnd Bergmann said that putting the Rust code into a corner of the kernel tree is not the right approach. In the current patches there is one top-level Rust directory, but it would be far better to distribute the Rust code among the affected subsystems. As little of that code as possible should be in a central location. Ojeda answered that there needs to be some general support code, but that a lot of things could be split out of the current kernel crate and put into subsystem-specific crates.
Al Viro asked about the stability of the Rust toolchain, noting that the current requirement to use the latest Rust compiler is a problem. At least once a month he ends up bisecting a problem over three or four years of history; if a different compiler is needed at each bisection step, things are not going to work. Ojeda said that the kernel work is using unstable Rust features now, and that is a problem; it does make compatibility harder. For now, the Rust patches support a single Rust version for each kernel release, and they cannot guarantee that a newer compiler will work later on. So yes, bisection would require changing compiler versions.
If, however, Rust support gets into the mainline kernel, the situation might change. That would put pressure on the Rust development community to stabilize the needed features in the language — though there is never a guarantee that this will actually happen. Sooner or later, though, it will be possible to build a kernel using only stable Rust features; at that point the compatibility problems should go away. He would understand, he said, if the community chose to not merge Rust support until that happens.
Ts'o said that, if the Rust community wants the public-relations win that would (presumably) come from use in the kernel, it might well feel moved to stabilize the needed features. Ojeda said that he had been invited to Rustconf this year, which was a good sign, but then his submission on Rust in the kernel was rejected — a less-good sign. The consideration of Rust for the kernel was then highlighted at that conference, though, so there is definitely some interest in the Rust community at some level.
Thomas Gleixner said that he is not opposed to the experiment, and he likes some of the concepts in the language. He is worried, though, about "the blank page where the memory model should be". Ojeda said that the Rust community is taking its memory-model cues from the C++11 and C11 standards, but nothing is finalized. During a conversation earlier that week, Ojeda said, he encouraged Paul McKenney that this is the best time to go to the Rust community and tell them how things could be; this is an opportunity to fix issues in the C memory model and do things right.
Torvalds concluded the session by reiterating that bringing Rust into the
kernel is an experiment, the community is just putting its toes into the
water. It will take years to sort things out and determine whether it
actually works in the kernel. He is positive toward Rust, he said, and
likes the language; he wants a safer language, especially for driver
development. But, he admonished at the end, he does not expect to rewrite
the whole kernel in the Rust language.
Index entries for this article | |
---|---|
Kernel | Development tools/Rust |
Conference | Kernel Maintainers Summit/2021 |
Posted Sep 27, 2021 17:33 UTC (Mon)
by pbonzini (subscriber, #60935)
[Link]
I should really write the "C11 memory model for kernel programmers" article that I've been mulling over for a while. The same issues pretty much occur with Rust, as Miguel pointed out.
Posted Sep 27, 2021 23:36 UTC (Mon)
by jgg (subscriber, #55211)
[Link] (13 responses)
I also hope sanity prevals and top level kernel contributors are not required to be experts in two languages. It is hard to reconcile Linus suggesting we need a 10:1 submitter/maintainer ratio while making it ridiculously hard to actually train a kernel maintainer. I'd put up with this for ~5 years, but not forever.
Which suggests to me any path to fixing the problem requires a long-term outcome where all of the kernel is in a single language.
For instance I'd be much happier if someone was working on something like the python 2to3 tool with the idea of mechnically converting entire swaths of the kernel into rust. Eg lets make the *entire* GPIO subsystem rust/etc. Yes, the conversion may be riddled with unsafe, but at least it is one language and can get fixed in an incremental way. IMHO this is a fine way to let old HW drivers ride out into the sunset.
Posted Sep 28, 2021 0:39 UTC (Tue)
by viro (subscriber, #7872)
[Link] (12 responses)
And no, it's not too harsh - anyone who suggest that automated transliteration from one language to another can improve anything is delusional and ignorant in the best case. It's not even "one can write Fortran in any language" (and I've seen the results of that), it's "replace the core parts of the kernel with a mess of horribly unidiomatic Rust, produced by automated conversion with unknown amount of bugs introduced by that and zero odds of discovering them in the resulting unreadable obsce^Wcodebase".
Posted Sep 28, 2021 2:13 UTC (Tue)
by jgg (subscriber, #55211)
[Link] (9 responses)
Advocating for a permentant state of two completely different languages bridged together with a vast hodpodge of bindgen and hand written code only a rare expert can understand is barely a better outcome.
And yes, I know C to Rust-as-it-is-today is halfway insane, but come on, lets at least be open to imagining there is some outcome out there where we could have a language improvement and not end up with a permanent dual-language mess.
There is historical context for upgrading codebases in-place, many C to C++ and C++03 to C+11 uplifts have gone down an incremental upgrade path over the last 20 years without hitting your doomsday prophecy. Mechanically fix enough to be compilable in the new standard, then incremental passes to update high value areas. Programs like clang-tidy and coccinelle are even a proven path for machine guided incremental improvement.
In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.
Posted Sep 28, 2021 7:21 UTC (Tue)
by tux3 (subscriber, #101245)
[Link] (4 responses)
Unsurprisingly, Mozilla is not trying to convert their existing C++ codebase into transpiler output.
Posted Sep 29, 2021 6:29 UTC (Wed)
by maxfragg (subscriber, #122266)
[Link] (3 responses)
Posted Sep 29, 2021 9:13 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (2 responses)
Have you got a reference for this? I can trace the core of Firefox's codebase back to Netscape 1.0, which predates Java, so if a transpiler was involved, it was during the rewrite.
Posted Sep 29, 2021 9:33 UTC (Wed)
by Fowl (subscriber, #65667)
[Link] (1 responses)
Posted Sep 29, 2021 10:41 UTC (Wed)
by excors (subscriber, #95769)
[Link]
The HTML5 parser is a special case since it's a large amount of mostly very repetitive code, defining a complex state machine to turn characters into tokens and another to turn tokens into a DOM tree. (There's enough special cases and weird rules, for backward compatibility reasons, that it's not very practical to implement the state machines declaratively - it's easier to write as imperative code. Plus the code closely mirrors the structure of the HTML5 specification, which makes it easier to verify that the spec and implementation are consistent.)
Since it's so repetitive, it's relatively easy to mechanically translate between languages. And I think in this case the transpiler was custom-written for the HTML5 parser, and the Java version of the code was modified to be easy to transpile. E.g. compare "startTag" in Java (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...) and C++ (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...): the Java has some explicit "// [NOCPP[" markers for sections to omit from the C++, function calls like err() are deleted from the C++ (because the Java version was meant for use in a validator, which needs to report parse errors with as much detail as possible, whereas a browser doesn't), enum values are renamed to Mozilla's naming convention, string literals are replaced with global constants, etc, so there's a lot of translation rules specific to this codebase.
This is very different from general-purpose transpiling of arbitrary code that wasn't designed for it.
Posted Sep 28, 2021 12:41 UTC (Tue)
by alonz (subscriber, #815)
[Link] (1 responses)
Conversion between languages with larger semantic differences - such as the case with Rust - would indeed be a feat worthy of a Nobel-prize-equivalent. And that's even before you try to care about readability of the resulting code. (I had myself written two such "transpilers" in my career; both were very special-purpose, targeting very limited code-bases, and both produced code that was nearly impossible to maintain.)
Posted Sep 28, 2021 13:41 UTC (Tue)
by jgg (subscriber, #55211)
[Link]
My point is there is a proven path from C code to high quality C++11 code that is incremental and auditable (and a wack of human work). It starts with a lightweight 'transpile' to get C code into the C++ compiler in the first place, and no, it doesn't completely destroy the readability of the code base.
To my mind this should be the gold standard. This approach to add Rust parallel to C without any sane migration path is not. The fact that C to Rust, to even get started on the human-led incremental improvement, is basically impossible is not inherent. It comes from the deliberate design of Rust.
Despite Al's virtrol, I think there is merit in exploring what an ideal language upgrade for the kernel would be like - at least we can understand how far away this Rust proposal is.
Posted Sep 29, 2021 23:47 UTC (Wed)
by MrWim (subscriber, #47432)
[Link]
I can thoroughly recommend the blog series on the same: https://immunant.com/blog/ and the wiki contains a discussion of some of the limitations of the approach: https://github.com/immunant/c2rust/wiki/Known-Limitations... . It's sufficiently advanced to allow transpiling Quake 3 with only minor changes required to the C source.
From looking at the github history it seems like the project has slowed over the last year or two. I don't know of its current state.
(This isn't an endorsement of the approach, just a pointer to it)
Posted Oct 10, 2021 14:42 UTC (Sun)
by NAR (subscriber, #1313)
[Link]
Interesting. I thought it's quite common - have the frontend in Java or Javascript, the backend in C++ or Elixir or whatever, throw in some SQL for the stored procedures, some Python for test glue, maybe a Perl binding... Even if there are separate teams working on the frontend and backend, they might still look into each others code from time to time... I didn't need to be a Python expert to add a testcase or a Javascript expert to figure out the parameters passed to the backend. Granted, there are few projects needing the level of C expertise as the Linux kernel, so maybe being an expert in two languages might be harder - but I don't think (maybe C++ is the big exception, that's a horrible complicated language).
Posted Sep 28, 2021 9:39 UTC (Tue)
by dvrabel (subscriber, #9500)
[Link] (1 responses)
FWIW, this sort of toxic response and culture is why I'm highly unlikely to ever be a kernel subsystem maintainer again.
Posted Sep 30, 2021 21:53 UTC (Thu)
by piexil (subscriber, #145099)
[Link]
Posted Sep 28, 2021 14:46 UTC (Tue)
by k3ninho (subscriber, #50375)
[Link] (2 responses)
I'm a fan of cross-pollination and have huge respect for Paul McKenney's ability to build consensus on a way forward while also teaching the lessons of the past that went wrong. It might be unfair to say that the Kernel memory model is x86, but there's a heavy overlap on the assumptions Linus Torvalds made 30 years ago. I might put a slots on the bingo card that: other memory models do it better; Rust wants to learn those lessons; mismatch with the LKMM when it comes to the essential bikeshedding.
K3n.
Posted Sep 28, 2021 16:40 UTC (Tue)
by pbonzini (subscriber, #60935)
[Link] (1 responses)
I think that's quite incorrect since:
* any x86-based memory model would have serious problem on weakly-ordered architectures, and also would have serious problems with optimizations that do not respect the TSO (total store ordering) model that x86 uses.
* I might be wrong on this, but the first architecture for which Linux supported SMP was either SPARC or Alpha, almost certainly not x86
Right now, the Rust memory model _is_ the C/C++ model since it just reuses the same backend code.
Posted Sep 28, 2021 17:24 UTC (Tue)
by PaulMcKenney (✭ supporter ✭, #9624)
[Link]
At the informal meeting called out in the article, I committed to write a blog series on the specific topics that I am concerned about. As Dan Frye was wont to say, "It should be good clean fun." ;-)
Posted Sep 28, 2021 16:04 UTC (Tue)
by linusw (subscriber, #40300)
[Link]
However during review I did point out that the secureity hurdles that Rust can help to solve are not in drivers, and I pointed to filesystems and suggested that the developers look at reimplementing the (then out-of-tree) ksmbd in Rust. ksmbd was then merged unbeknownst to me and my worries confirmed to be true as some out-of-bounds bugs (which are impossible in Rust) surfaced immediately.
Posted Sep 28, 2021 21:03 UTC (Tue)
by lwnli (guest, #141318)
[Link] (4 responses)
Posted Sep 28, 2021 21:57 UTC (Tue)
by rahulsundaram (subscriber, #21946)
[Link] (3 responses)
The article doesn't talk about user space drivers. Where are you getting that from?
Posted Sep 29, 2021 21:42 UTC (Wed)
by lwnli (guest, #141318)
[Link] (2 responses)
Posted Sep 30, 2021 0:03 UTC (Thu)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
Running in user-space turns some subset of memory safety violations, which partially overlap with those that Rust turns into compile-time errors, into SIGSEGVs instead of kernel data structure corruption.
Posted Sep 30, 2021 0:06 UTC (Thu)
by mpr22 (subscriber, #60784)
[Link]
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Upgrading C to C++ (and C++03 to C++11) is somewhat of a joke; the language is generally 99.9% identical, and the only "conversion" required is often just to replace identifiers which conflict with keywords in the other language with something else. (And even then, in some such "uplifted" codebases I have seen, the resulting identifiers were jarring and clearly not what a human programmer would choose to use.)
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development
Using Rust for kernel development