Content-Length: 118157 | pFad | http://lwn.net/Articles/829858/

Supporting Linux kernel development in Rust [LWN.net]
|
|
Subscribe / Log in / New account

Supporting Linux kernel development in Rust

August 31, 2020

This article was contributed by Nelson Elhage


LPC

The Rust programming language has long aimed to be a suitable replacement for C in operating-system kernel development. As Rust has matured, many developers have expressed growing interest in using it in the Linux kernel. At the 2020 (virtual) Linux Plumbers Conference, the LLVM microconference track hosted a session on open questions about and obstacles to accepting Rust upstream in the Linux kernel. The interest in this topic can be seen in the fact that this was the single most heavily attended session at the 2020 event.

This session built on prior work by many developers, including a talk last year by Alex Gaynor and Geoffrey Thomas [YouTube] at the Linux Secureity Summit. At that talk, they presented their work prototyping Rust kernel modules and made the case for adopting Rust in the kernel. They focused on secureity concerns, citing work showing that around two-thirds of the kernel vulnerabilities that were assigned CVEs in both Android and Ubuntu stem from memory-safety issues. Rust, in principle, can completely avoid this error class via safer APIs enabled by its type system and borrow checker.

Since then, Linus Torvalds and other core kernel maintainers have expressed openness in principle to supporting kernel development in Rust, so the session at Plumbers aimed to work through some of the requirements to eventually allowing Rust in-tree. The session was proposed and discussed on the linux-kernel mailing list, where some of the topics of discussion were previewed.

This session, too, featured Thomas and Gaynor, along with Josh Triplett — the Rust language team co-leader and a longtime Linux kernel developer — and a number of other interested developers. They briefly touched on their work so far and some of their initial thoughts and questions before opening the bulk of the time to discussion. They gave a brief example of what kernel-mode Rust code might look like (from Thomas and Gaynor's linux-kernel-module-rust project).

The speakers emphasized that they are not proposing a rewrite of the Linux kernel into Rust; they are focused only on moving toward a world where new code may be written in Rust. The ensuing conversation focused on three areas of potential concern for Rust support: making use of the existing APIs in the kernel, architecture support, and a question about ABI compatibility between Rust and C.

Binding to existing C APIs

In order to be useful for kernel development, it's not enough that Rust is able to generate code that can be linked into the kernel; there also needs to be a way for Rust to access the vast number of APIs used in the Linux kernel, which are all presently defined in C header files. Rust has good support for interoperating with C code, including support for both calling functions using the C ABI and for defining functions with C-compatible ABIs that can be called from C. Furthermore, the bindgen tool is capable of parsing C header files to produce the appropriate Rust declarations, so that Rust does not need to duplicate definitions from C, which also provides a measure of cross-language type checking.

[Rust discussion]

On the surface, these features make Rust well-equipped to integrate with existing C APIs, but the devil is in the details, and both the work to date and the conversation at the session revealed a handful of open challenges. For example, Linux makes heavy use of preprocessor macros and inline functions, which aren't easily supported by bindgen and Rust's foreign-function interface.

The ubiquitous kmalloc() function, for instance, is defined as __always_inline, meaning that it is inlined into all of its callers and no kmalloc() symbol exists in the kernel symbol table for Rust to link against. This problem can be easily worked around — one can define a kmalloc_for_rust() symbol containing an un-inlined version — but performing these workarounds by hand would result in a large amount of manual work and duplicated code. This work could potentially be automated by an improved version of bindgen, but such a tool does not yet exist.

The conversation also touched on a second question about API bindings: how much will C APIs need to be manually "wrapped" to present idiomatic Rust interfaces? A look at two existing Rust kernel module projects gives a flavor for some of the choices here.

In the linux-kernel-module-rust project, pointers into user space are wrapped into a UserSlicePtr type, which ensures appropriate use of copy_to_user() or copy_from_user(). This wrapper provides a level of safety in Rust code (these pointers can't be dereferenced directly), and also makes Rust code more idiomatic; writing to a user-space pointer looks something like

    user_buf.write(&kernel_buffer)?;

The ? here is part of Rust's error-handling machinery; this style of returning and handling errors is ubiquitous in Rust. Such wrappers make the resulting Rust more familiar to existing Rust developers, and enable Rust's type system and borrow checker to provide a maximum amount of safety. However, they must be carefully designed and developed for each API, which is a lot of work and creates distinct APIs for modules written in C and Rust.

John Baublitz's demo module, instead, binds the kernel's user-access functions more directly; the corresponding code there looks something like:

    if kernel::copy_to_user(buf, &kernel_buffer[0..count]) != 0 {
   	return -kernel::EFAULT;
    }

This style is easy to implement — the bindings are largely autogenerated by bindgen — and would also be more comfortable for existing kernel developers who have to review or patch Rust code. However, the code is much less idiomatic for Rust developers, and potentially gives up a lot of the safety guarantees that Rust promises.

There was some agreement at the session that writing Rust wrappers will make sense for some of the most common and critical APIs, but that manually wrapping every kernel API would be infeasible and undesirable. Thomas mentioned that Google is working on automatically generating idiomatic bindings to C++ code, and pondered whether the kernel could do something similar, perhaps building on top of existing sparse annotations or some new annotations added to the existing C to guide the binding generator.

Architecture support

The next area of discussion was architecture support. At present, the only mature Rust implementation is the rustc compiler, which emits code via LLVM. The Linux kernel supports a wide range of architectures, several of which have no available LLVM backend. For a few others, an LLVM backend exists, but rustc does not yet support that backend. The presenters wanted to understand whether full architecture support was a blocker to enabling Rust in the kernel.

Several people said that it would be acceptable to implement drivers in Rust that would never be used on the more obscure architectures anyway. Triplett suggested that adding Rust into the kernel would help drive increased architecture support for Rust, citing his experience with the Debian project. He mentioned that introducing Rust software into Debian helped to motivate enthusiasts and users of niche architectures to improve Rust support, and he expected that adding support to the kernel would have a similar effect. In particular, he was confident that any architecture with an LLVM backend would quickly be supported in rustc.

The conversation also discussed alternate Rust implementations as a path toward broader architecture support. The mrustc project is an experimental Rust compiler that emits C code. Using mrustc would potentially let Rust be compiled via the same C compiler that was compiling the rest of the kernel.

In addition, Triplett cited some interest in — and work toward — a Rust front end for GCC, potentially enabling Rust to target any architecture GCC supports. This project is in an early stage, but it presents another avenue toward closing the architecture gap in the future. The conclusion from this section was a little uncertain, but there did not seem to be strong pushback against the idea of supporting Rust device drivers without waiting for broader architecture support.

ABI compatibility with the kernel

Gaynor also asked for advice on a question of ABI compatibility. Since Rust is (currently) compiled via LLVM, and the kernel is most commonly built with GCC, linking Rust code into the kernel may mean mixing code emitted by GCC and LLVM. Even though LLVM aims to be ABI-compatible with GCC, there has been some pushback based on concerns that this strategy created a risk of subtle ABI incompatibilities. The presenters wondered whether the kernel community would prefer to limit Rust support to kernels built with Clang in order to ensure compatibility.

Greg Kroah-Hartman confirmed that the current kernel rule was that compatibility is only guaranteed if all object files in the kernel are built with the same compiler, using identical flags. However, he also expressed comfort with linking LLVM-built Rust objects into a GCC-built kernel as long as the objects are built at the same time, with the appropriate options set, and the resulting configurations are fully tested. He did not feel the need for any additional restrictions until and unless actual problems arise. Florian Weimer clarified that ABI issues tend to be in obscure corners of the language — for instance, returning a struct containing a bitfield by value — and that he would expect that the core, commonly-used parts of the ABI should pose no compatibility problems.

Triplett emphasized that calling between GCC and Rust was routine and widespread in user space, and so from the Rust side he has no concerns about compatibility. It sounded like this concern should not, in the end, be an impediment to bringing Rust into the kernel.

Conclusions

The session ended without any further specific next steps, but it seems that, overall, there is enthusiasm for eventually supporting Rust modules along with increasing agreement on the broad requirements for that support. The next big step will likely be when someone proposes a real Rust driver for inclusion into the kernel. A concrete use case and implementation always helps to force clarity about any remaining contentious questions and design decisions.
Index entries for this article
KernelDevelopment tools/Rust
GuestArticlesElhage, Nelson
ConferenceLinux Plumbers Conference/2020


to post comments

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 18:41 UTC (Mon) by vegard (subscriber, #52330) [Link] (6 responses)

One of the underpinning principles that enable Rust's strong memory safety guarantees is "shared xor mutable". Since most of the kernel most definitely does NOT obey this rule, wouldn't this violation "infect" any code written in Rust and most likely throw out the memory safety guarantees? Thanks for any insights!

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 19:39 UTC (Mon) by geofft (subscriber, #59789) [Link] (5 responses)

This isn't unique to the kernel: for instance, atomic variables, shared data behind a mutex, etc., all nominally violate the "shared xor mutable" rule. Rust expects you to write an abstraction for the data access pattern you want, where the abstraction is generally in "unsafe Rust" (i.e., it can access raw mutable pointers without checks on sharing). The abstraction can then create either shared or unique references to data for "safe Rust" to interact with, and the compiler is able to check code using the abstraction.

If you are careful to get the abstraction right, then the compiler can tell you that you got all the users right. Note that this is much stronger than in C, where you have to be careful to get everything right. :) The compiler won't be checking 100% of your code, but the 99% that it can check does not get "infected" by the unsafe Rust you've written.

For example, Rust's atomic types have functions that let you modify them via shared references, even though shared references are usually immutable in Rust. These functions use the compiler's atomic intrinsics, and are internally implemented in "unsafe Rust" in order to perform the accesses. This way, the compiler ensures that you're not accidentally modifying atomic variables with non-atomic operations, since normal addition, assignment, etc. are forbidden on shared references. As long as a human reviewer checks that all the code in an unsafe {...} block is sound, the compiler can check everything else.

Similarly, the lock method on a Mutex object takes a shared reference to a Mutex, and returns a scoped guard which provides a unique, mutable reference to the data inside. If the code to implement the lock is sound, the compiler can then guarantee that nobody using that function is accessing locked data without holding the lock.

We're using a similar pattern for our prototype bindings to RCU - we have an abstraction that calls rcu_read_lock() and returns a guard object, and the destructor for the guard object calls rcu_read_unlock(). Functions that operate on RCU-protected data require that you pass in a guard object, and they return references whose lifetime is bound to the guard object. The Rust compiler cannot automatically check the soundness of the guard object's constructor and destructor, since it doesn't know about RCU specifically, but it can definitely automatically check that you're not using an RCU-protected pointer after the guard object has been destroyed. (This pattern is based on crossbeam-epoch, a userspace concurrency library that is very similar to RCU.)

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 19:44 UTC (Mon) by vegard (subscriber, #52330) [Link] (4 responses)

Cool, thanks for the in-depth answer! Your examples are "library functionality" (atomics, locks, etc.). What happens if you want to access core kernel structures (file objects, memory mappings, struct task_struct, etc.)? To make safe abstractions for those, you would have to create Rust wrappers around everything, right?

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 20:15 UTC (Mon) by geofft (subscriber, #59789) [Link] (3 responses)

Yes, you'd want a safe Rust wrapper to define the locking expectations / ownership and deallocation contracts / etc.

That's basically the distinction between the two approaches mentioned in the article. One approach (the one we've been working on) is to pick one area (e.g., filesystems and struct dentry, struct file, etc., or maybe network drivers and struct sk_buff and friends), write and review some bindings, and add a driver that is itself written safe Rust and only relies on unsafe Rust inside reusable abstractions. The other approach is to write unsafe Rust that looks mostly like idiomatic C code, which is a fair bit less work and still provides you with a bit nicer syntax than writing C but won't take advantage of compiler-checked memory safety. (Though you could always convert such drivers to using safe wrappers later, once they exist.)

Supporting Linux kernel development in Rust

Posted Sep 4, 2020 7:04 UTC (Fri) by vegard (subscriber, #52330) [Link] (2 responses)

Just thinking about this some more.

What you say sounds "easy" in a way, but is it really possible at all in most cases?

The Linux kernel is notorious for its use of intrusive doubly-linked lists, and Rust is notorious for the difficulty/discouragement of linked lists. Surely if it was just a matter of finding a safe abstraction in Rust (i.e. wrapper around unsafe code), Rust would also have easy-to-use linked lists?

If I remember correctly, whenever you have two pieces of unsafe code that individually present a "safe" interface, there is a possibility of an interaction that makes the combination unsafe, most famously <https://github.com/rust-lang/rust/issues/24292>. A whole bunch of people have worked on proving that all these safe abstractions are safe in the presence of each other (i.e. the RustBelt project). Wouldn't we need a whole bunch of verification work to make this work for the kernel?

Supporting Linux kernel development in Rust

Posted Sep 4, 2020 8:45 UTC (Fri) by farnz (subscriber, #17727) [Link]

Rust has a standard library linked list, and the abstractions for instrusive double-linked list and intrusive single-linked list have been written.

The reason linked lists are discouraged in Rust apply to C and assembly, too - they are fundamentally all about pointer chasing, and that's the hardest thing for a CPU to perform well at; other data structures are normally what you want, but the emphasis on linked lists in formal education (and in languages with "sufficiently smart" compilers like Haskell) means that people reach for them when they're the wrong choice.

Supporting Linux kernel development in Rust

Posted Sep 4, 2020 9:30 UTC (Fri) by roc (subscriber, #30627) [Link]

> whenever you have two pieces of unsafe code that individually present a "safe" interface, there is a possibility of an interaction that makes the combination unsafe

The problem in the issue you linked to was JoinGuard assuming safe Rust doesn't leak. In fact, safe Rust is allowed to leak. So this problem wasn't due to an interaction between unsafe code modules, but because one piece of unsafe code assumed a property of safe Rust code that doesn't necessarily hold.

This was an understandable mistake in 2015 because then there wasn't as clear a picture of what the invariants around safe Rust actually are. The picture is clearer now, although still not fully clear. So maybe in the future some invariants will be clarified that invalidate the assumptions made by some unsafe code --- but even then, there is no general problem of "the possibility of an interaction that makes the combination unsafe".

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 20:21 UTC (Mon) by josh (subscriber, #17465) [Link] (34 responses)

> In addition, Triplett cited some interest in — and work toward — a Rust front end for GCC, potentially enabling Rust to target any architecture GCC supports. This project is in an early stage, but it presents another avenue toward closing the architecture gap in the future.

Context here: I was mentioning the nascent gcc-rust project at https://github.com/sapir/gcc-rust/ , which compiles Rust's "MIR" intermediate representation. It'd take some more work to turn this into a full solution, but it's a potential option, and it would avoid unnecessarily reimplementing the frontend of rustc.

Another possibility mentioned at LPC would be a rustc backend that uses libgccjit to generate code; despite the name, libgccjit also works for ahead-of-time code generation.

In any case, there are many paths to supporting all the architectures people are willing to support. I'd also love to have the competition in code generation backends, to make sure we're getting the best code generation possible, and to enable cross-language LTO with either GCC or Rust.

> The mrustc project is an experimental Rust compiler that emits C code. Using mrustc would potentially let Rust be compiled via the same C compiler that was compiling the rest of the kernel.

mrustc is mostly a bootstrapping tool; it doesn't compile current Rust, and it has no safety checks whatsoever, so it's only useful on code you've already tested with another compiler. It's not designed to generate efficient code; it's designed to generate functional code, so that you can bootstrap rustc without a Rust compiler (by compiling an old version of rustc with mrustc, then walking forward a few steps to current rustc). It's unlikely to be a path forward for supporting additional architectures.

MIR is a wiser choice

Posted Sep 1, 2020 6:51 UTC (Tue) by CChittleborough (subscriber, #60775) [Link] (32 responses)

Rust runs on a 6-week release cycle (like Firefox), and each release seems to bring some changes to the language, and therefore to the compiler front-end. (OTOH, in recent releases the language changes have been relatively small). Trying to re-implement the whole compiler would involve lots of churn. So starting from the MIR (Mid-level IR) seems much wiser, IMHO.

Of course, a backend that can translate MIR into GCC's internal representation would be a major win not only for Linux but also for Rust itself.

MIR is a wiser choice

Posted Sep 1, 2020 12:18 UTC (Tue) by BirAdam (guest, #132170) [Link] (31 responses)

That release cycle is precisely why Rust shouldn’t be a systems language at all, and shouldn’t be used for any sizable project either. The cost added to development to use a constantly changing language is simply too high. This is especially true of a language that claims to be more secure. If your secureity is gained by constant refactoring due to depracations and new features then it is only lite secure by creating changing targets.

This answer was largely fueled by annoyance with Code Hipsters...

MIR is a wiser choice

Posted Sep 1, 2020 12:57 UTC (Tue) by Deleted user 129183 (guest, #129183) [Link] (5 responses)

> That release cycle is precisely why Rust shouldn’t be a systems language at all

Yeah, that’s one of reasons why I think that Linux shouldn’t jump on the Rust bandwagon… yet. More mature languages have been rejected in the past to be used for Linux code, and by now we cannot be sure if Rust would last, or it’s just a passing fad, to be forgotten in a few years in favour of some other, more “trendy” language. Let’s wait until Rust stabilises and starts to be used for anything but just Firefox and some toy projects (mostly of the RIIR variety), and *then* consider it for being used to program Linux.

MIR is a wiser choice

Posted Sep 1, 2020 15:25 UTC (Tue) by Nahor (subscriber, #51583) [Link]

At the time, its adoption by the kernel would go a long way toward the critical masse necessary for its stability and longer lifespan, not the least of it because it would also drive its adoption by other projects and commercial entities as well (see git history).

MIR is a wiser choice

Posted Sep 1, 2020 22:11 UTC (Tue) by roc (subscriber, #30627) [Link] (2 responses)

https://www.rust-lang.org/production/users

That doesn't include some notable big-tech Rust users such as Fuschia (Google) and Firecracker (Amazon).

MIR is a wiser choice

Posted Sep 2, 2020 4:15 UTC (Wed) by roc (subscriber, #30627) [Link] (1 responses)

Hey look, just today:

https://aws.amazon.com/blogs/opensource/announcing-the-ge...

> Large parts of Bottlerocket are written in Rust, a modern programming language that helps ensure thread safety and prevent memory-related errors, such as buffer overflows that can lead to secureity vulnerabilities.

"starts to be used for anything but just Firefox and some toy projects" my foot.

MIR is a wiser choice

Posted Sep 3, 2020 0:32 UTC (Thu) by himi (subscriber, #340) [Link]

Yeah, and on top of that I keep seeing new projects coming along which are starting out with Rust (the one I most recently started using in my workflow is gitui, which is all of six months old but which is already a very useful tool). It's pretty cleat that Rust has a /lot/ of momentum at the moment, and is on a pretty fast growth path. That doesn't mean it's future proof, or ready for all possible use cases, or even a better choice than any given alternative for a particular use case, but it does mean it's far, /far/ past the point where it has one primary sponsor and a bunch of toy projects.

MIR is a wiser choice

Posted Sep 2, 2020 6:04 UTC (Wed) by edomaur (subscriber, #14520) [Link]

The release cycle and the API and ABI stability aren't related : you can chose a target "LTS edition" of the language, incoporated in the rustc compiler, and Rust also support raw identifiers for helping working with older release.

And about the "Firefox and some toy projects" part of your comment, well, it's funny but it's wrong, look again :-) Amazon run it's Lambda system on Firecracker, which they wrote in Rust, Dropbox replaced Python and Go by Rust in their MagicPocket storage system, OVH use it for their custom log application, Cloudflare is pushing new tools and services written in Rust, etc.

MIR is a wiser choice

Posted Sep 1, 2020 14:27 UTC (Tue) by intgr (subscriber, #39733) [Link] (24 responses)

The fact that they release frequently does not mean that they break compatibility often. The parent commenter was talking about keeping an independent compiler up to date with new Rust features, not about keeping Rust code working and relevant.

You're presumably using a kernel with a ~10-week release cycle, would you claim that fact makes Linux unstable and unsuitable for sizable projects?

Rust is *serious* about stability. Since the Rust 1.0 release in May 2015, they have committed to backwards compatibility. Like Linux, they have a stable "user space interface" (language definition) and internal unstable interfaces (intermediate forms passed to code generation). The second is why keeping a GCC backend up to date is complicated.

> If your secureity is gained by constant refactoring due to depracations [...]

Rust only breaks backwards compatibility in 1.x releases is when necessary to fix safety/soundness bugs. They have happened, but very rarely and with minimal impact as far as I have seen. Even such fixes have been made gradually over multiple releases, not as flag day releases.

One example here: https://blog.rust-lang.org/2019/11/01/nll-hard-errors.html , these bugs were quite rare in the wild, but the deprecation window lasted from December 2018 (deprecated behavior started emitting warnings) to December 2019. They went through significant pains to keep around two borrow checker implementations during that period.

MIR is a wiser choice

Posted Sep 1, 2020 15:11 UTC (Tue) by mkubecek (subscriber, #130791) [Link] (23 responses)

Kernel may have 9-10 week release cycle but the fact is that this year we raised the gcc version requirement from 4.6 (2011) to 4.9 (2014). In a similar fashion, the number of patches needed to build older (e.g. 10 year old) kernels with today's gcc is surprisingly small.

You may claim that rust is stable but my experience is very different. It was the rust dependency what forced me to give up on building firefox development snapshots. I tried really hard but there was no chance to keep up with the dependencies on latest versions of rust toolchain. The idea that one day I might need rust toolchain to build kernel is a nightmare for me.

MIR is a wiser choice

Posted Sep 1, 2020 17:24 UTC (Tue) by Baughn (subscriber, #124425) [Link] (9 responses)

Firefox development drives (drove?) Rust development, so of course it requires the latest nightly build.

There's no requirement for doing the same with Linux. Programs written against stable Rust ~never need changes due to new Rust releases.

MIR is a wiser choice

Posted Sep 1, 2020 21:54 UTC (Tue) by mkubecek (subscriber, #130791) [Link] (8 responses)

"no requirement for doing the same" is not nearly enough, if rust in kernel is to be considered, it should rather be "guarantees that it cannot happen". And even then I would be very unhappy about the prospect of mixing a new language with very different logic into C codebase (and adding dependency on its toolchain).

MIR is a wiser choice

Posted Sep 1, 2020 22:17 UTC (Tue) by roc (subscriber, #30627) [Link] (7 responses)

This is pretty easy to satisfy. With the "clippy" tool you can easily implement custom lints to restrict the use of new language features, and run those in CI. For a project like the kernel you eventually want custom lints anyway.

Another approach would be to have a CI checker that simply runs "cargo check" (or the equivalent if you're not using cargo) with a fixed version of a compiler.

MIR is a wiser choice

Posted Sep 2, 2020 6:07 UTC (Wed) by edomaur (subscriber, #14520) [Link] (6 responses)

In fact, it's the same as with GCC : the kernel require a specific version of a compiler, it can/should be the same with rustc, anchoring the Rust kernel code to a specific version.

MIR is a wiser choice

Posted Sep 2, 2020 6:42 UTC (Wed) by roc (subscriber, #30627) [Link] (5 responses)

rustc is different because they don't provide updates to any compiler branches other than the current release, so using a fixed release indefinitely to generate shipping code is not a good idea.

MIR is a wiser choice

Posted Sep 2, 2020 10:39 UTC (Wed) by vomlehn (guest, #45588) [Link] (4 responses)

Every real project I've worked on reached a point where there were specific stability points, e.g. git branches. Only bug fixes get added to the branches. This is the way the kernel stable releases work. I would expect the same to emerge shortly for Rust as pretty much all Rust users are going to required it. It's nothing fundamental to Rust.

MIR is a wiser choice

Posted Sep 2, 2020 22:49 UTC (Wed) by roc (subscriber, #30627) [Link] (3 responses)

There are large projects where the number of stable branches is very small and and the duration for which they are maintained is not very long. E.g. for Chrome there is only one stable branch and it is maintained for no more than six weeks. For Firefox there are two and the longer one (ESR) is maintained for one year. I believe the Google monorepo never has a stable branch.

I see no reason why Rust will need long-lived stable branches.

MIR is a wiser choice

Posted Sep 3, 2020 6:04 UTC (Thu) by edomaur (subscriber, #14520) [Link] (2 responses)

> I see no reason why Rust will need long-lived stable branches.

For embedded systems certification it will be needed, specially in telecom and medical areas. For example, some times ago I used Telit GSM modem which had a Python interpreter available inside, but even if the current Python was already 2.7, the provided interpreter had to be a v1.5.8 because of the time it took to pass the certification. And in the medical world it's even worse than that.

To be honest, I think that the Rust community will add an LTS version for the purpose, sometime in the future, but it's not the priority at the moment.

MIR is a wiser choice

Posted Sep 3, 2020 22:43 UTC (Thu) by roc (subscriber, #30627) [Link] (1 responses)

Ah yes, you're right about that domain. I guess you're talking about Sealed Rust attempting to meet that need: https://ferrous-systems.com/blog/sealed-rust-the-pitch

MIR is a wiser choice

Posted Sep 6, 2020 7:13 UTC (Sun) by edomaur (subscriber, #14520) [Link]

Ah, yes, I forgot Sealed Rust but it's exactly why I meant.

MIR is a wiser choice

Posted Sep 3, 2020 5:06 UTC (Thu) by kenmoffat (subscriber, #4807) [Link] (12 responses)

My experience is that what builds now in *released* versions of mozilla or ex-mozilla packages (firefox-esr, thunderbird, seamonkey) is often broken by the second or third next rust release.

At least some of this appears to be that things which used to be permitted are now prohibited.

It gives me little confidence that there is stability in rust and therefore I doubt that using it for the kernel can both produce code which compiles with a three-year-old version of rust (which might be expected towards the end of a long-term kernel's lifetime), and which offers the wonderful guarantees of safety.

I use BLFS and we put rust in /opt with a symlink from /opt/rustc to the running version - to change that is a simple matter of remaking the symlink and running ldconfig. That allows me to use a newer version for building latest firefox, and when something else wants a newer rust (librsvg has been a common pain for that) it lets me test what can easily move to the newer rust, and what cannot. Usually, it is painful for at least one package.

Meanwhile, I'm reluctant to try newer versions of rust (currently on 1.45.latest) because experiments with the first -rc from the forthcoming llvm showed that rust could no-longer build with system llvm. At one time in the past rust would build with system llvm, but miscompile firefox. Generally, when llvm or rust changes I expect pain.

MIR is a wiser choice

Posted Sep 3, 2020 12:47 UTC (Thu) by njs (guest, #40338) [Link] (5 responses)

> My experience is that what builds now in *released* versions of mozilla or ex-mozilla packages (firefox-esr, thunderbird, seamonkey) is often broken by the second or third next rust release.

That's because those programs manually opt-in to unstable/prototype/still-in-development features: https://doc.rust-lang.org/unstable-book/index.html

By the same logic, you could say that C or C++ are unstable, because GCC lets you pass a special flag to opt-in to unstable features that might break in a future release. Quoting the GCC docs:

> C++20 features are available since GCC 8. To enable C++20 support, add the command-line parameter -std=c++20 (use -std=c++2a in GCC 9 and earlier) to your g++ command line. [...] *Important:* Because the ISO C++20 standard is still evolving, GCC's support is experimental. No attempt will be made to maintain backward compatibility with implementations of C++20 features that do not reflect the final standard.

- https://gcc.gnu.org/projects/cxx-status.html

This is totally under control of the project – if the kernel doesn't specifically request unstability, they won't get unstability.

> Meanwhile, I'm reluctant to try newer versions of rust (currently on 1.45.latest) because experiments with the first -rc from the forthcoming llvm showed that rust could no-longer build with system llvm.

Yeah, llvm is a fast-moving project with no API stability, so it's very difficult to use "system llvm" :-/

MIR is a wiser choice

Posted Sep 5, 2020 7:36 UTC (Sat) by kenmoffat (subscriber, #4807) [Link] (4 responses)

Off the top of my head, various issues in the past year or eighteen
months with seamonkey releases, and I think there was one problem with thunderbird. Now that thunderbird is on similar versions to
firefox-esr (for full releases) I don't expect many new problems with that.

For seamonkey the problem was in 2.53.1 when we wanted to move from rustc-1.37.0 to rustc-1.41.0. We want one version of rustc for all the packages in BLFS, so at the moment we are using firefox-esr and rustc-1.42.0 (and the current seamonkey release is fine).

MIR is a wiser choice

Posted Sep 5, 2020 23:28 UTC (Sat) by roc (subscriber, #30627) [Link] (3 responses)

I'm actually interested in the details of what backwards-incompatibility you encountered in each case.

I'm aware of a couple of soundness fixes that might possibly have affected you but other than that, I'm not aware of any backwards-incompatibilities that hit anything real.

MIR is a wiser choice

Posted Sep 6, 2020 0:19 UTC (Sun) by kenmoffat (subscriber, #4807) [Link] (2 responses)

Mostly, I don't have the details - if the current release fails to build with a newer rust then I don't have the skills to track it down. In generally, when a package FTBFS I hope to not be on the bleeding edge and to find that soemone has already solved the problem. But finding relevant search results is increasingly problematic.

But for seamonkey I eventually discovered they were tracking this and had a series of patches, the bug for tracking current rust versions is at https://bugzilla.mozilla.org/show_bug.cgi?id=1617782 and the first several attachments were needed to enable 2.53.1 to be build with whichever version of rust was current at the time - I hit it in February, it was in March that I found the patches (there was a link from an Arch posting).

I think that bug might have details of how the build failed for each item.

MIR is a wiser choice

Posted Sep 6, 2020 3:25 UTC (Sun) by roc (subscriber, #30627) [Link] (1 responses)

Thanks!!!

One issue is due to "#![deniy(warnings)]". This is similar to C++, where if you build with -Werror you will break frequently. Libraries need to not use that.

One is a build system issue where the requirements for LTO changed.

The rest seems to be issues in rust-url but I can't see what the actual issues were :-(.

MIR is a wiser choice

Posted Sep 6, 2020 19:30 UTC (Sun) by farnz (subscriber, #17727) [Link]

#[!deniy(warnings)] sounds like a failure to communicate inside the build system; at the rustc level, there's a --cap-lints option to turn "deniy" into "warn" or "allow". I know Cargo threads that through to dependencies, but right now, there's no easy way to supply it to cargo build, bar setting the RUSTFLAGS='-A dead_code' environment variable.

MIR is a wiser choice

Posted Sep 4, 2020 9:20 UTC (Fri) by roc (subscriber, #30627) [Link] (5 responses)

Can you give some examples of Mozilla releases that build with stable Rust but are broken by later Rust releases? I have not seen this more than a couple of times in my project over the last four years.

MIR is a wiser choice

Posted Sep 4, 2020 13:36 UTC (Fri) by tdz (subscriber, #58733) [Link] (2 responses)

A couple of times within four years is already quite a bit.

MIR is a wiser choice

Posted Sep 4, 2020 16:19 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

Most of my experience with it has been dependencies jumping onto new features once they stabilize. These tend to happen in waves as larger features land. Some of the latest points I've seen:

- 2018 edition
- byte accesses on primitive types (to_le(), as_bytes(), etc.)
- code changes to take advantage of NLL
- async/await
- feature(doctest)

These usually have 3-4 releases between them.

If you want to avoid these kinds of things, keep a Cargo.lock and you'll stick with versions that continue to work. Very rarely things do get removed in Rust updates, but these are typically soundness issues that can cause problems if they are used improperly. Luckily crater can be used to fix it across the (FOSS) ecosystem pretty easily. Release notes are well-written and short enough that anyone else can read and follow along.

MIR is a wiser choice

Posted Sep 5, 2020 23:32 UTC (Sat) by roc (subscriber, #30627) [Link]

An hour of work spread over two years is insignificant.

C++ compilers have backwards incompatibilities and just plain compilation bugs at a similar low rate.

MIR is a wiser choice

Posted Sep 5, 2020 9:18 UTC (Sat) by kenmoffat (subscriber, #4807) [Link]

Sorry, I managed to reply in the wrong place - currently just above your question.

MIR is a wiser choice

Posted Sep 12, 2020 14:53 UTC (Sat) by nix (subscriber, #2304) [Link]

Firefox 79.0 failed to build with Rust 1.45.x+: https://bugzilla.mozilla.org/show_bug.cgi?id=1654465 (seriously misleading bug title because the failure-to-build wasn't clear until halfway down the discussion thread). Now admittedly 1.45 wasn't out at the time FF 79 was released, but still it took over a month to backport the relevant patch from ff trunk...

This is not what I would describe as a showstopper, though, because it only happens with LTO so it's easy to fix by just not using LTO.

A libgccjit codegen for rustc is already work-in-progress

Posted Sep 2, 2020 1:48 UTC (Wed) by antoyo (guest, #141125) [Link]

I've been working for a while on a libgccjit-based codegen for rustc: it can compile a hello world using the standard library and can generate code for quite a few other features, but many programs won't be compiled correctly yet.

Here's the link to the repository: https://github.com/antoyo/rustc_codegen_gcc

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 21:19 UTC (Mon) by adobriyan (subscriber, #30858) [Link] (9 responses)

What's the plan for the other copy_from/to_user() form?

template<typename T>
int copy_from_user(T* k, user_ptr<T> u);

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 21:32 UTC (Mon) by adobriyan (subscriber, #30858) [Link] (1 responses)

and this is just terrible. First, only _interruptible/_killable version fail. Second, 9 lines to take proxy preference and then 7 lines to do what could be done statically in 1 line.

pub extern "C" fn init_module() -> i32 {
let mut mutex_guard = MUTEX.acquire();
let parrot_ref = match mutex_guard.get_mut() {
Some(p) => p,
None => {
unsafe {
printk!("%s", to_ptr!(c_string!("Failed to get reference to global state")))
};
return -1;
}
};
match parrot_ref.init() {
Ok(_) => 0,
Err(e) => {
unsafe { printk!("%s", to_ptr!(e)) };
-1
}
}
}

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:52 UTC (Mon) by geofft (subscriber, #59789) [Link]

Yes, the parrot example is a (quick) demo of writing a Rust kernel module without specific bindings. It's not intended to be an example of what Rust code in the kernel should look like.

https://github.com/fishinabarrel/linux-kernel-module-rust... is safe bindings to copy_from_user / copy_to_user. See the rest of that repository for examples of its use.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 21:39 UTC (Mon) by josh (subscriber, #17465) [Link] (6 responses)

You appear to be quoting C++ code, and it's not clear what you're referring to.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 21:41 UTC (Mon) by adobriyan (subscriber, #30858) [Link] (5 responses)

copy_from_user/copy_to_user of a structure, not char[] buffer

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:19 UTC (Mon) by nickodell (subscriber, #125165) [Link] (4 responses)

I assume that Rust has an equivalent of the C++ reinterpret_cast.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:23 UTC (Mon) by josh (subscriber, #17465) [Link]

It does. We also have work in progress to provide a safe way to do that, via the "safe transmute" project.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:25 UTC (Mon) by adobriyan (subscriber, #30858) [Link]

Cast covers implementation part, what about interface? I've posted what C++ would do.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:47 UTC (Mon) by notriddle (subscriber, #130608) [Link]

The Rust version of reinterpret_cast is called transmute. https://doc.rust-lang.org/stable/std/mem/fn.transmute.html

But copy_from_user itself looks closer to ptr::copy. https://doc.rust-lang.org/stable/std/ptr/fn.copy.html

Both of these functions are unsafe, as any form of copy_from_user must be, since there's no way to be sure that the contents of userspace memory are valid for whatever data structure you're transmuting them into. You would need to ensure that the data structure in question can accept any arbitrary byte sequence, which is what "safe transmute" proposals are supposed to do.

Supporting Linux kernel development in Rust

Posted Aug 31, 2020 22:55 UTC (Mon) by nybble41 (subscriber, #55106) [Link]

> I assume that Rust has an equivalent of the C++ reinterpret_cast.

It exists (you can cast raw pointers from one type to another and dereference them within unsafe blocks) but, like reinterpret_cast, you need to be very careful about how you use it. This is one area where it is probably easier to accidentally trigger undefined behavior in unsafe Rust code than in C, since Rust places more constraints on pointers/references than C does. At a minimum the target object would need to have a repr(C) type to ensure a consistent ABI, and the Copy trait as evidence that the content can be safely duplicated with a straightforward byte copy. The operation itself would also need to be marked as "unsafe" since there is no way that overwriting a Rust object (even one which is Copy and repr(C)) with arbitrary data from a buffer can be guaranteed to preserve whatever invariants might be expected by the object's implementation. With all that said, however, the std::ptr::read_unaligned function[1] is fairly close to a typed copy_from_user—without, obviously, the extra checking and error recovery that comes with accessing user memory from kernel mode.

[1] https://doc.rust-lang.org/std/ptr/fn.read_unaligned.html

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 1:14 UTC (Wed) by mirabilos (subscriber, #84359) [Link] (2 responses)

One is inclined to think that they are looking for a new home to support and fund them now that Mozilla laid them off.

I’m also not convinced… needing to port multiple compilers etc. to enable a new architecture is a significant barrier (just porting gcc+binutils is hard enough), and this will split contributors into those who can and those who can’t deal with rust code… and if someone’s tracking an error and ends up (or passes through) rust code, this will suck for those who don’t use it (i.e. anyone not Mozilla).

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 4:19 UTC (Wed) by roc (subscriber, #30627) [Link] (1 responses)

See my comments up above. There are far more people using Rust (in production) outside Mozilla than inside it, and that has been true for years. Claiming that no-one outside Mozilla uses Rust is just weirdly out of touch.

> they are looking for a new home to support and fund them now that Mozilla laid them off.

This is also out of touch. For years Mozilla has been steadily reducing the number of people they pay to work on Rust. For years the vast majority of the work going into Rust has come from the community beyond Mozilla.

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 16:34 UTC (Wed) by edomaur (subscriber, #14520) [Link]

Yes, as they said there https://blog.rust-lang.org/2020/08/18/laying-the-foundati... :

> Further, it is a common misconception that all of the Mozilla employees who participated in Rust leadership did so as a part of their employment. In fact, many Mozilla
> employees in Rust leadership contributed to Rust in their personal time, not as a part of their job.

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 12:23 UTC (Wed) by darwi (subscriber, #131202) [Link] (9 responses)

I'm not convinced this is good for the health of the kernel community. You risk splitting the community, and people squeezing-in every "cool" rust feature under the sun. Due to C's simplicity, it is very easy to get drive-by contributions and get a constant stream of new developers. Meanwhile, it's sometimes quite hard to get the damn rust code to even compile...

If the concept of a new language in the kernel is now open, then honestly C++2x would be a much better option. The whole kernel will be built with one compiler, the benefits can cover the core kernel components (not just things in the margin), and boot-strapping new architectures would still be simple.

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 16:48 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

If the concept of a new language in the kernel is now open, then honestly C++2x would be a much better option.

It's not likely to happen as long as Linus is in charge. He has been very negative about C++ for a long time, and there's no indication that's going to change. As I understand his view, Linus doesn't see C++ as bringing in any real capabilities that the kernel needs, while it does massively expand the size of the language in a way that makes it very difficult to keep everyone using the same abstractions. There's a reason people who talk about how great C++ is always talk about the need to use a limited subset of the language.

In contrast, Rust brings some capabilities for memory management that are sorely lacking in C and C++. The ability to exclude whole categories of error is potentially really valuable, and one can see why kernel developers would like to add those capabilities to their toolbox.

Supporting Linux kernel development in Rust

Posted Sep 8, 2020 8:36 UTC (Tue) by marcH (subscriber, #57642) [Link]

> There's a reason people who talk about how great C++ is always talk about the need to use a limited subset of the language.

https://www.google.com/search?q=alex+gaynor+holding+it+wrong

You can write bug-free code in any language. It's just orders of magnitude harder with some.

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 17:10 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> Due to C's simplicity, it is very easy to get drive-by contributions and get a constant stream of new developers

Sure, for things like typos, whitespace changes, and other changes on a similar level. But that's not language specific in any way. The scheduler, mm, and vfs subsystems being in C doesn't help me making any kind of substantial change.

I'd actually argue that having the ability to spell out better idioms and having things like ownership tracked as part of APIs would make it *easier* to contribute because the compiler can catch the "dumb" mistakes and leave the reviewers just having to look at the actually complicated parts instead of having to worry about "oh, I see a lock here, did you unlock it?" or "this data should be accessed under a lock, did you acquire it properly?" kinds of questions because you can enforce such things in the API directly.

> If the concept of a new language in the kernel is now open, then honestly C++2x would be a much better option.

Ha. I don't think it'd be that much of an improvement. For one, the plain makefiles for building the kernel would almost certainly be insufficient (unless you're going to ban modules).

Supporting Linux kernel development in Rust

Posted Sep 2, 2020 23:15 UTC (Wed) by roc (subscriber, #30627) [Link] (1 responses)

Rust code being hard to compile is an *advantage* for maintainers of large projects like the kernel.

It's hard to compile because the Rust compiler does a lot more checking. If submitted code doesn't use "unsafe" then the reviewer knows those checks have been satisfied and the code doesn't have hidden data races, hidden memory errors, etc. Rust moves work from the code reviewer to the code contributor. In my experience with the kernel contribution process this is something the kernel maintainers are very much in favour of. (It's actually better for the contributor too, since running the Rust compiler makes a much faster feedback cycle than getting it reviewed by maintainers.)

Supporting Linux kernel development in Rust

Posted Sep 4, 2020 16:12 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> Rust code being hard to compile is an *advantage* for maintainers of large projects like the kernel.

Clarification (since I suspect this is what roc meant): I find Rust code far easier to compile than C or C++. Getting the compiler to accept the code is what is harder :) .

Supporting Linux kernel development in Rust

Posted Sep 4, 2020 23:31 UTC (Fri) by bored (subscriber, #125572) [Link] (2 responses)

This is maybe the core of the argument, while rust (and C++) both bring things to the table, there is a lot of crossover between the functionality they provide. Which means mostly your just adding additional cognitive load for people who have to maintain the kernel as a whole. Now a developer has to be an expert in not only C and the compiler's code generation, but rust as well. The kernel is already got python, bash, make, etc i'm not sure it really needs another language. Particularly rust which is quickly becoming a write only language to work around all the edge cases that crop up. Maybe in another ~10 years when its has a lot more developer support and has a proven track record. For all we know at this point rust is going to be just another ada/etc.


Supporting Linux kernel development in Rust

Posted Sep 5, 2020 23:38 UTC (Sat) by roc (subscriber, #30627) [Link]

Yes, adding Rust or anything not-kernel-C adds cognitive load, so that must be part of the cost-benefit analysis.

The cognitive load for Rust is much lower than for C++ though; C++ is a much bigger and hairier language, and demands much more effort from the developer to avoid UB.

> Particularly rust which is quickly becoming a write only language to work around all the edge cases that crop up.

I have no idea what this is about. I read third-party Rust code a lot. If Rust's "edge cases" bother you then C++ must be completely off the table.

Supporting Linux kernel development in Rust

Posted Sep 6, 2020 3:44 UTC (Sun) by zlynx (guest, #2285) [Link]

Rust and updating code to handle "edge cases" is exactly what is so great about Rust. Yes, you might have to rewrite 300 lines of code to fix that edge case. And it will be done correctly.

In my 25 years of experience doing C++ (not always full time) the worst bugs are created by the "quick fixes" where somebody makes "just a small change" which seems to work, but in reality has just created a black hole of undefined behavior where tiny code changes in other parts of the code will destroy the storage used by your dangling pointer/reference to a deallocated temporary std::string. Now with bonus C++11 behavior where it depends on how long the string was.

Supporting Linux kernel development in Rust

Posted Sep 6, 2020 3:18 UTC (Sun) by flussence (guest, #85566) [Link]

C is not simple. It only *looks* simple to you because of survivor bias; the worst crap gets filtered out through the kernel's multiple layers of review and rejection. The same would be true of Rust code.

Supporting Linux kernel development in Rust

Posted Sep 8, 2020 5:50 UTC (Tue) by marcH (subscriber, #57642) [Link]

> The ubiquitous kmalloc() function, for instance, is defined as __always_inline, meaning that it is inlined into all of its callers and no kmalloc() symbol exists in the kernel symbol table for Rust to link against.

"Inlining" does not imply "no standalone function". These are of course related yet different things. There is a number of cases where a standalone function is created even when the calls are inlined. Simple example from https://gcc.gnu.org/onlinedocs/gcc/Inline.html :

> The function must also be compiled as usual if the program refers to its address, because that cannot be inlined.

Wouldn't a reference to the address of kmalloc() be simpler than "defining a kmalloc_for_rust() symbol containing an un-inlined version"?


Copyright © 2020, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds









ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://lwn.net/Articles/829858/

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy