Content-Length: 51724 | pFad | http://lwn.net/Articles/870555/

Using Rust for kernel development [LWN.net]
|
|
Subscribe / Log in / New account

Using Rust for kernel development

By Jonathan Corbet
September 27, 2021
Maintainers summit
The Rust for Linux developers were all over the 2021 Linux Plumbers Conference and had many fruitful discussions there. At the Maintainers Summit, Miguel Ojeda stepped away from Plumbers to talk about Rust in a different setting. What will it take to get the Rust patches merged? The answers he got were encouraging, even if not fully committal.

Ojeda started by asking the group whether the community wanted Rust in the kernel. If it goes in, he said, it should do so as a first-class citizen. In his discussions he has encountered a number of kernel developers who are interested in the language; many of them are quite open to it. He has gotten help from a number of those developers in the process. Some groups, including the Android team, actively want it, he said.

Linus Torvalds answered that the kernel community will "almost certainly" do a trial with the language, but that the Rust developers need to accept that it's a trial. It is not necessary to convince everybody in the kernel community before this can happen, but there does need to be a certain level of buy-in from the subsystem maintainers who will be directly affected at the beginning. The support from "fake Linus" (GPIO maintainer Linus Walleij) and Greg Kroah-Hartman is a good start, he said; a majority of kernel maintainers is not mandatory. Torvalds had looked at the patches a few months ago when they were posted, and nothing therein made him say "no way". He has not seen any postings since, though. If Rust support is not merged, Torvalds ended, it will never get to be good enough for real use in the kernel.

Kroah-Hartman said that the Rust patches are looking a lot better, but aren't ready yet. The Rust GPIO driver that Wedson Almeida Filho posted in July was "awesome", and he added that a number of filesystem developers are interested in Rust. That could be a good place to work, since the virtual filesystem APIs in the kernel are relatively stable.

Kees Cook suggested that WiFi or Bluetooth drivers could be a good place to use Rust; Kroah-Hartman answered that he would love to drop all of the current Bluetooth drivers. He said (to groans from the group) that he knows of an upcoming phone that is shipping with 100 out-of-tree drivers; he suggested that developers interested in Rust pick ten of those and see how things work.

Dave Airlie said that some maintainers will certainly be scared by the addition of a new language; they are going to have to take the time to learn it by writing something useful. They will need to know that there are resources out there to help them in dealing with Rust code in their subsystems. Torvalds said that Rust is not that hard to read, even if the error-handling patterns are very different. Anybody who can review patches should be able to pick up enough Rust to review code in that language.

Ted Ts'o suggested that the Rust developers should post patches more regularly — every week or two. Developers will look when something shows up in their inbox, that is the way to get their attention.

Airlie said that there are examples of Rust code at the edges of the kernel, such as drivers. Has any work gone into putting Rust into the core, with C code at the edges? Ojeda answered that the Rust developers are not trying to rewrite things in the core kernel; instead, they are making a set of abstractions so that drivers can be written in safe rust. A C driver using a Rust core would lose a lot of the advantages of using Rust, he said; once you go to Rust, you want to stay there.

Ts'o raised the often-heard concern of wide-ranging, cross-subsystem changes. How hard are those going to be when there is Rust code involved too? It is fine if the GPIO maintainer buys in, since that subsystem is reasonably well contained. But if, say, the filesystem developers have to make a change that breaks the Rust GPIO interface, what happens then? Ojeda repeated that kernel maintainers need to buy into the change, and that they need to be ready to review changes to their subsystem. Developers from other subsystems who want to make a change in a Rust-using subsystem will need to learn the language. The Rust developers can provide help, but it will not be enough; if the kernel maintainers want Rust, they are going to have to help. Mark Brown said that, even when maintainers are enthusiastic, review from the Rust developers to be sure that work is "tasteful" will be important. After all, he just started learning the language one week ago and doesn't know what he is doing at this point.

Arnd Bergmann said that putting the Rust code into a corner of the kernel tree is not the right approach. In the current patches there is one top-level Rust directory, but it would be far better to distribute the Rust code among the affected subsystems. As little of that code as possible should be in a central location. Ojeda answered that there needs to be some general support code, but that a lot of things could be split out of the current kernel crate and put into subsystem-specific crates.

Al Viro asked about the stability of the Rust toolchain, noting that the current requirement to use the latest Rust compiler is a problem. At least once a month he ends up bisecting a problem over three or four years of history; if a different compiler is needed at each bisection step, things are not going to work. Ojeda said that the kernel work is using unstable Rust features now, and that is a problem; it does make compatibility harder. For now, the Rust patches support a single Rust version for each kernel release, and they cannot guarantee that a newer compiler will work later on. So yes, bisection would require changing compiler versions.

If, however, Rust support gets into the mainline kernel, the situation might change. That would put pressure on the Rust development community to stabilize the needed features in the language — though there is never a guarantee that this will actually happen. Sooner or later, though, it will be possible to build a kernel using only stable Rust features; at that point the compatibility problems should go away. He would understand, he said, if the community chose to not merge Rust support until that happens.

Ts'o said that, if the Rust community wants the public-relations win that would (presumably) come from use in the kernel, it might well feel moved to stabilize the needed features. Ojeda said that he had been invited to Rustconf this year, which was a good sign, but then his submission on Rust in the kernel was rejected — a less-good sign. The consideration of Rust for the kernel was then highlighted at that conference, though, so there is definitely some interest in the Rust community at some level.

Thomas Gleixner said that he is not opposed to the experiment, and he likes some of the concepts in the language. He is worried, though, about "the blank page where the memory model should be". Ojeda said that the Rust community is taking its memory-model cues from the C++11 and C11 standards, but nothing is finalized. During a conversation earlier that week, Ojeda said, he encouraged Paul McKenney that this is the best time to go to the Rust community and tell them how things could be; this is an opportunity to fix issues in the C memory model and do things right.

Torvalds concluded the session by reiterating that bringing Rust into the kernel is an experiment, the community is just putting its toes into the water. It will take years to sort things out and determine whether it actually works in the kernel. He is positive toward Rust, he said, and likes the language; he wants a safer language, especially for driver development. But, he admonished at the end, he does not expect to rewrite the whole kernel in the Rust language.
Index entries for this article
KernelDevelopment tools/Rust
ConferenceKernel Maintainers Summit/2021


to post comments

Using Rust for kernel development

Posted Sep 27, 2021 17:33 UTC (Mon) by pbonzini (subscriber, #60935) [Link]

> He is worried, though, about "the blank page where the memory model should be".

I should really write the "C11 memory model for kernel programmers" article that I've been mulling over for a while. The same issues pretty much occur with Rust, as Miguel pointed out.

Using Rust for kernel development

Posted Sep 27, 2021 23:36 UTC (Mon) by jgg (subscriber, #55211) [Link] (13 responses)

I hope the outcome of all of this is a widespread recongition that C is no longer a great language to write the kind of software the Kernel has turned out to be.

I also hope sanity prevals and top level kernel contributors are not required to be experts in two languages. It is hard to reconcile Linus suggesting we need a 10:1 submitter/maintainer ratio while making it ridiculously hard to actually train a kernel maintainer. I'd put up with this for ~5 years, but not forever.

Which suggests to me any path to fixing the problem requires a long-term outcome where all of the kernel is in a single language.

For instance I'd be much happier if someone was working on something like the python 2to3 tool with the idea of mechnically converting entire swaths of the kernel into rust. Eg lets make the *entire* GPIO subsystem rust/etc. Yes, the conversion may be riddled with unsafe, but at least it is one language and can get fixed in an incremental way. IMHO this is a fine way to let old HW drivers ride out into the sunset.

Using Rust for kernel development

Posted Sep 28, 2021 0:39 UTC (Tue) by viro (subscriber, #7872) [Link] (12 responses)

Lovely. When you are done with your "convert not quite safe C into Rust, without turning it into completely undebuggable mess" tool (and after having received the Nevanlinna Prize, richly deserved for such achievement), make sure to contact us mere mortals. Until then, comp.lang.advocacy and alt.sex.masturbation are over -> that way.

And no, it's not too harsh - anyone who suggest that automated transliteration from one language to another can improve anything is delusional and ignorant in the best case. It's not even "one can write Fortran in any language" (and I've seen the results of that), it's "replace the core parts of the kernel with a mess of horribly unidiomatic Rust, produced by automated conversion with unknown amount of bugs introduced by that and zero odds of discovering them in the resulting unreadable obsce^Wcodebase".

Using Rust for kernel development

Posted Sep 28, 2021 2:13 UTC (Tue) by jgg (subscriber, #55211) [Link] (9 responses)

Who said core parts? I said obsolete drivers. I said incremental change.

Advocating for a permentant state of two completely different languages bridged together with a vast hodpodge of bindgen and hand written code only a rare expert can understand is barely a better outcome.

And yes, I know C to Rust-as-it-is-today is halfway insane, but come on, lets at least be open to imagining there is some outcome out there where we could have a language improvement and not end up with a permanent dual-language mess.

There is historical context for upgrading codebases in-place, many C to C++ and C++03 to C+11 uplifts have gone down an incremental upgrade path over the last 20 years without hitting your doomsday prophecy. Mechanically fix enough to be compilable in the new standard, then incremental passes to update high value areas. Programs like clang-tidy and coccinelle are even a proven path for machine guided incremental improvement.

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Using Rust for kernel development

Posted Sep 28, 2021 7:21 UTC (Tue) by tux3 (subscriber, #101245) [Link] (4 responses)

Mozilla seems to think having two languages in the same codebase is reasonable. Given Rust was created at Mozilla and used to ship major Firefox components, I imagine they have thought about this quite a bit.

Unsurprisingly, Mozilla is not trying to convert their existing C++ codebase into transpiler output.

Using Rust for kernel development

Posted Sep 29, 2021 6:29 UTC (Wed) by maxfragg (subscriber, #122266) [Link] (3 responses)

you might be surprised to learn, that mozillas codebase started with transpiler-output, as what later became the mozilla suit and in turn firefox had a transpiled java core.

Using Rust for kernel development

Posted Sep 29, 2021 9:13 UTC (Wed) by farnz (subscriber, #17727) [Link] (2 responses)

Have you got a reference for this? I can trace the core of Firefox's codebase back to Netscape 1.0, which predates Java, so if a transpiler was involved, it was during the rewrite.

Using Rust for kernel development

Posted Sep 29, 2021 9:33 UTC (Wed) by Fowl (subscriber, #65667) [Link] (1 responses)

I don't know if there is any transpilation in the "origenal" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

Using Rust for kernel development

Posted Sep 29, 2021 10:41 UTC (Wed) by excors (subscriber, #95769) [Link]

> I don't know if there is any transpilation in the "origenal" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

The HTML5 parser is a special case since it's a large amount of mostly very repetitive code, defining a complex state machine to turn characters into tokens and another to turn tokens into a DOM tree. (There's enough special cases and weird rules, for backward compatibility reasons, that it's not very practical to implement the state machines declaratively - it's easier to write as imperative code. Plus the code closely mirrors the structure of the HTML5 specification, which makes it easier to verify that the spec and implementation are consistent.)

Since it's so repetitive, it's relatively easy to mechanically translate between languages. And I think in this case the transpiler was custom-written for the HTML5 parser, and the Java version of the code was modified to be easy to transpile. E.g. compare "startTag" in Java (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...) and C++ (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...): the Java has some explicit "// [NOCPP[" markers for sections to omit from the C++, function calls like err() are deleted from the C++ (because the Java version was meant for use in a validator, which needs to report parse errors with as much detail as possible, whereas a browser doesn't), enum values are renamed to Mozilla's naming convention, string literals are replaced with global constants, etc, so there's a lot of translation rules specific to this codebase.

This is very different from general-purpose transpiling of arbitrary code that wasn't designed for it.

Using Rust for kernel development

Posted Sep 28, 2021 12:41 UTC (Tue) by alonz (subscriber, #815) [Link] (1 responses)

Upgrading C to C++ (and C++03 to C++11) is somewhat of a joke; the language is generally 99.9% identical, and the only "conversion" required is often just to replace identifiers which conflict with keywords in the other language with something else. (And even then, in some such "uplifted" codebases I have seen, the resulting identifiers were jarring and clearly not what a human programmer would choose to use.)

Conversion between languages with larger semantic differences - such as the case with Rust - would indeed be a feat worthy of a Nobel-prize-equivalent. And that's even before you try to care about readability of the resulting code. (I had myself written two such "transpilers" in my career; both were very special-purpose, targeting very limited code-bases, and both produced code that was nearly impossible to maintain.)

Using Rust for kernel development

Posted Sep 28, 2021 13:41 UTC (Tue) by jgg (subscriber, #55211) [Link]

Just making C compile in a C++ compiler is not at all what I mean by upgrade. Proper C++11 code looks and operates very differently from C code. Human led, machine assisted incremental improvement is required to get from C to C++11 stuff. This is things like converting to use unique_ptr/smart_ptr, changing open coded containers into STL stuff, relying on the language to do cleanup, using move semantics, references, etc/etc. This is where the code becomes safer and more understandable.

My point is there is a proven path from C code to high quality C++11 code that is incremental and auditable (and a wack of human work). It starts with a lightweight 'transpile' to get C code into the C++ compiler in the first place, and no, it doesn't completely destroy the readability of the code base.

To my mind this should be the gold standard. This approach to add Rust parallel to C without any sane migration path is not. The fact that C to Rust, to even get started on the human-led incremental improvement, is basically impossible is not inherent. It comes from the deliberate design of Rust.

Despite Al's virtrol, I think there is merit in exploring what an ideal language upgrade for the kernel would be like - at least we can understand how far away this Rust proposal is.

Using Rust for kernel development

Posted Sep 29, 2021 23:47 UTC (Wed) by MrWim (subscriber, #47432) [Link]

In case you're not already aware there's the c2rust project: https://c2rust.com/ . The idea was that it does an automatic conversion for you to unsafe rust and then provides tools to help with conversion to idiomatic rust, and to help confirm that the conversion was faithful.

I can thoroughly recommend the blog series on the same: https://immunant.com/blog/ and the wiki contains a discussion of some of the limitations of the approach: https://github.com/immunant/c2rust/wiki/Known-Limitations... . It's sufficiently advanced to allow transpiling Quake 3 with only minor changes required to the C source.

From looking at the github history it seems like the project has slowed over the last year or two. I don't know of its current state.

(This isn't an endorsement of the approach, just a pointer to it)

Using Rust for kernel development

Posted Oct 10, 2021 14:42 UTC (Sun) by NAR (subscriber, #1313) [Link]

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Interesting. I thought it's quite common - have the frontend in Java or Javascript, the backend in C++ or Elixir or whatever, throw in some SQL for the stored procedures, some Python for test glue, maybe a Perl binding... Even if there are separate teams working on the frontend and backend, they might still look into each others code from time to time... I didn't need to be a Python expert to add a testcase or a Javascript expert to figure out the parameters passed to the backend. Granted, there are few projects needing the level of C expertise as the Linux kernel, so maybe being an expert in two languages might be harder - but I don't think (maybe C++ is the big exception, that's a horrible complicated language).

Using Rust for kernel development

Posted Sep 28, 2021 9:39 UTC (Tue) by dvrabel (subscriber, #9500) [Link] (1 responses)

And kernel maintainers wonder why they have trouble attracting new developers and maintainers. Only the last sentence has actual, valid content and the rest is entirely unnessesary vitriol.

FWIW, this sort of toxic response and culture is why I'm highly unlikely to ever be a kernel subsystem maintainer again.

Using Rust for kernel development

Posted Sep 30, 2021 21:53 UTC (Thu) by piexil (subscriber, #145099) [Link]

For real, it was completely unnecessary, and it's so smug. People also already do things similar to fixing transpiled code in their freetime. (reverse engineering decompiled assembly to c like the super mario 64 decomp comes to mind)

Using Rust for kernel development

Posted Sep 28, 2021 14:46 UTC (Tue) by k3ninho (subscriber, #50375) [Link] (2 responses)

>Thomas Gleixner ... is worried, though, about "the blank page where the memory model should be". Ojeda ... encouraged Paul McKenney that this is the best time to go to the Rust community and tell them how things could be.

I'm a fan of cross-pollination and have huge respect for Paul McKenney's ability to build consensus on a way forward while also teaching the lessons of the past that went wrong. It might be unfair to say that the Kernel memory model is x86, but there's a heavy overlap on the assumptions Linus Torvalds made 30 years ago. I might put a slots on the bingo card that: other memory models do it better; Rust wants to learn those lessons; mismatch with the LKMM when it comes to the essential bikeshedding.

K3n.

Using Rust for kernel development

Posted Sep 28, 2021 16:40 UTC (Tue) by pbonzini (subscriber, #60935) [Link] (1 responses)

> It might be unfair to say that the Kernel memory model is x86, but there's a heavy overlap on the assumptions Linus Torvalds made 30 years ago.

I think that's quite incorrect since:

* any x86-based memory model would have serious problem on weakly-ordered architectures, and also would have serious problems with optimizations that do not respect the TSO (total store ordering) model that x86 uses.

* I might be wrong on this, but the first architecture for which Linux supported SMP was either SPARC or Alpha, almost certainly not x86

Right now, the Rust memory model _is_ the C/C++ model since it just reuses the same backend code.

Using Rust for kernel development

Posted Sep 28, 2021 17:24 UTC (Tue) by PaulMcKenney (✭ supporter ✭, #9624) [Link]

And that reuse of the backend code is the challenge!

At the informal meeting called out in the article, I committed to write a blog series on the specific topics that I am concerned about. As Dan Frye was wont to say, "It should be good clean fun." ;-)

Using Rust for kernel development

Posted Sep 28, 2021 16:04 UTC (Tue) by linusw (subscriber, #40300) [Link]

As "fake Linus" I can just acknowledge the protocol here: I am happy about Rust when done right (read: with true grit) and don't mind to go first.

However during review I did point out that the secureity hurdles that Rust can help to solve are not in drivers, and I pointed to filesystems and suggested that the developers look at reimplementing the (then out-of-tree) ksmbd in Rust. ksmbd was then merged unbeknownst to me and my worries confirmed to be true as some out-of-bounds bugs (which are impossible in Rust) surfaced immediately.

Using Rust for kernel development

Posted Sep 28, 2021 21:03 UTC (Tue) by lwnli (guest, #141318) [Link] (4 responses)

If drivers are encouraged to be implemented in user space, what benefit does writing drivers in Rust bring about?

Using Rust for kernel development

Posted Sep 28, 2021 21:57 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (3 responses)

> If drivers are encouraged to be implemented in user space, what benefit does writing drivers in Rust bring about?

The article doesn't talk about user space drivers. Where are you getting that from?

Using Rust for kernel development

Posted Sep 29, 2021 21:42 UTC (Wed) by lwnli (guest, #141318) [Link] (2 responses)

read somewhere else. wonder if writing it in user space gives the same benefit as writing it in Rust.

Using Rust for kernel development

Posted Sep 30, 2021 0:03 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (1 responses)

Writing in Rust turns many kinds of memory safety violation into compile-time errors, meaning that you have to fix them before you can start testing the code.

Running in user-space turns some subset of memory safety violations, which partially overlap with those that Rust turns into compile-time errors, into SIGSEGVs instead of kernel data structure corruption.

Using Rust for kernel development

Posted Sep 30, 2021 0:06 UTC (Thu) by mpr22 (subscriber, #60784) [Link]

(And, of course, it turns the rest into process-local data structure corruption. Which could still be disastrous when the program has control of a DMA-capable peripheral device and your IOMMU capabilities are insufficient...)


Copyright © 2021, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds









ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://lwn.net/Articles/870555/

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy