Well-maintained free-software projects usually make a point of quickly fixing known security problems, and the Samba project, which provides interoperability between Windows and Unix systems, is no exception. So it is natural to wonder why the fix for CVE-2021-20316, a symbolic-link vulnerability, was well over two years in coming. Sometimes, a security bug can be fixed with a simple tweak to the code. Other times, the fix requires a massive rewrite of much of a projects's internal code. This particular vulnerability fell firmly into the latter category, necessitating a public rewrite of Samba's virtual filesystem (VFS) layer to address a non-disclosed vulnerability.

The story starts with a bug report from Michael Hanselmann in May 2019. When an SMB client instructs the server to create a new directory, the server must carry out a number of checks to ensure that the client is entitled to do that. Among other things, the server makes sure that the requested directory actually lies within the exported SMB share rather than being at some arbitrary location elsewhere in the server's filesystem. Unfortunately, there is inevitably a window between when the server performs the check and when it actually creates the directory. If a malicious user is able to replace a component in the path for the new directory with a symbolic link during that window, Samba will happily follow the link and make the directory in the wrong place, with results that are generally seen as distasteful by anybody but an attacker.

This is a classic time-of-check/time-of-use (TOCTOU) vulnerability, of the sort that symbolic links have become notorious for. It is also a hard one to fix, especially for a system like Samba, where portability is an important concern. There is no easy, cross-platform way to query the attributes of a path in the filesystem and safely act on the result, secure in the knowledge that a malicious actor cannot change things in the middle. Still, something clearly needed to be done, so Samba developer Jeremy Allison jumped in to write a fix. The CVE number CVE-2019-10151 was duly assigned to this problem.

The real problem

The hope was to come up with a quick solution but, from the outset, Allison identified the real problem: the use of path names to interact with the server-side filesystem. Every time that a path is passed to the kernel, the process of walking through it must be carried out anew; any user who can make that process arrive in different places at different times (through carefully timed use of symbolic links, for example) could use that ability to confuse the server. The good news is that there is another way that doesn't rely on unchanging paths.

Over the years, the kernel has gained a set of system calls that operate on file handles (open file descriptors) rather than path names. A carefully written server can, for example, use openat2() to create a file descriptor for a directory of interest, do its checks to ensure that the directory is what is expected, then use mkdirat() to create a subdirectory that cannot be redirected to the wrong place. Properly used, these system calls remove the TOCTOU race from this kind of operation, but they only work if they are being used — and Samba's use of them in 2019 was limited. At the time, Allison remarked: "Ultimately we need to modify the VFS to use the syscallAT() variants of all the system calls, but that's a VFS rewrite we will have to schedule for another day".

Over a month or so of attempts to close the vulnerability (and other symbolic-link issues that arose once people started looking for them), it became increasingly clear that "another day" was coming sooner than anybody had thought. By mid-July 2019, Allison seemed resigned to the big rewrite: "This is going to be a long slog, re-writing the pathname processing in the server". There was a complication, though: while this slog was underway, the vulnerability would remain unfixed and undisclosed. So how would all of this work be explained to anybody else who was watching the Samba project's work? According to Allison:

We need to rewrite the fileserver to [make] arbitrary symlink race change safe on all pathname operations. This is too large to do in private, so I'm doing this in public under the guise of "modernising the VFS to use handle-based operations" (without saying explicitly *why* I'm doing so).

There were a few other aspects of this project — beyond the need to hide its real purpose — that made it hard. One of those is that version 1 of the SMB protocol (SMB1) is path-name-based at its core, making it almost impossible for the server to use anything else. The abrupt deprecation of SMB1 in the Samba 4.11 release in September 2019 was partially driven by this problem.

The SMB2 protocol, instead, is based heavily on file handles, which should make it easier for the server implementation to work the same way. But Samba is an old program with a lot of history, so many of the internal interfaces were still using path names, even when a file handle was readily available. This includes the VFS interface that is used to talk with modules for specific host filesystems, add functionality like virus scanning, and more. Changing all of those internal APIs was a large job that would touch much of the code in the Samba server.

Thousands of changes

During the next two years, Allison would contribute 1,638 commits to the Samba repository — 17% of the total over that period. Not all of those were aimed at the VFS rewrite, but most of them were. And Allison was not alone; Ralph Böhme (1,261 commits), Noel Power (438) and Samuel Cabrero (251) also contributed heavily to this project. "Modernizing" the Samba VFS took up much of the project's attention while remaining mostly under the radar for anybody who was unaware of the real problem.

Böhme presented this work at the 2021 SambaXP event (video, slides) without ever mentioning the (still urgent) security problem that was driving it. The talk gets into a lot of the details of what needed to be done and how various problems were solved on Linux; it is recommended viewing for anybody who wants to dig deeper. There is also some information on the Samba wiki.

In July 2021, Allison declared victory:

With the master commit e168a95c1bb1928cf206baf6d2db851c85f65fa9, I believe all race conditions on meta-data are now fixed in the default paths. The async DOS attribute read still uses path-based getxattr, and some of the VFS modules are not symlink safe, but out of the box Samba I believe will no longer be vulnerable to this in 4.15.0.

Since then, the remaining path-based extended-attribute calls have been fixed as well. Of course, there were a few details to deal with yet, including the fact that the original CVE number had expired due to not being updated for too long. That necessitated the assignment of a new number, which is why this vulnerability is known as CVE-2021-43566. The work did show up as expected in the Samba 4.15.0 release in September 2021 — more than two years after the initial vulnerability report.

In the vulnerability disclosure, the Samba project described the situation this way:

A two and a half year effort was undertaken to completely re-write the Samba VFS layer to stop use of pathname-based calls in all cases involving reading and writing of metadata returned to the client. This work has finally been completed in Samba 4.15.0. [...]

As all operations are now done on an open handle we believe that any further symlink race conditions have been completely eliminated in Samba 4.15.0 and all future versions of Samba.

The disclosure also notes that, due to the massive nature of the rewrite, it will not be possible to fix this vulnerability in earlier Samba releases.

In the end, the Samba project was able to get this vulnerability fixed before word of the problem spread, and before any known exploits took place. But it was a bit of a gamble; attackers tend to keep an eye on the repositories of interesting projects in the hope of noticing patches addressing undisclosed vulnerabilities. It is hard not to draw comparisons with the events leading up to the disclosure of Meltdown and Spectre, both of which also required massive changes to address an undisclosed vulnerability. But, unlike the developers working on Spectre, the Samba developers found a way to do their work in public, ensuring that all patches were properly reviewed and minimizing the problems that had to be addressed after disclosure of the problem.

The gamble appears to have paid off, though things could have gone differently. Since then, Allison has been making the point that symbolic links are dangerous in general, and that other projects almost certainly have similar problems. He has a talk planned for SambaXP later this year that, presumably, will be more forthcoming than Böhme's 2021 presentation. Samba users (those who have updated, at least) are hopefully immune to symbolic-link attacks, but that is probably not true of many other systems that we depend on.

(Thanks to Jeremy Allison for answering questions and performing technical review on a draft of this article.)

Index entries for this article
Security	Samba

---

to post comments

[/tmp:128248] ls -l foo
-rw-r--r-- 1 root root 4 Feb 23 12:56 foo
[/tmp:128249] ln foo bar
ln: failed to create hard link 'bar' => 'foo': Operation not permitted
[/tmp:128251] cp --reflink foo bar
[/tmp:128252] ls -l bar
-rw-r--r-- 1 anton users 4 Feb 23 12:57 bar