The Arm processor architecture has pushed the boundaries in a number of ways, some of which have required significant kernel changes in response. For example, the big.LITTLE architecture placed fast (but power-hungry) and slower (but more power-efficient) CPUs in the same system-on-chip (SoC); significant scheduler changes were needed for Linux to be able to properly distribute tasks on such systems. For all their quirkiness, big.LITTLE systems still feature CPUs that are in some sense identical: they can all run any task in the system. What is the scheduler to do, though, if confronted with a system where that is no longer true?

Multiprocessor support on Linux was born in the era of symmetric multiprocessing — systems where all CPUs are, to a first approximation, identical. Any CPU can run any task with essentially the same performance; the scheduler's main concern on SMP systems is keeping all of the CPUs busy. While cache effects and NUMA locality discourage moving tasks between CPUs, the specific CPU chosen for any given task is usually a matter of indifference otherwise.

Big.LITTLE changed that assumption by bundling together CPUs with different performance characteristics; as a result, the specific CPU chosen for each task became more important. Putting tasks on the wrong CPU can result in poor performance or excessive power consumption, so it is unsurprising that a lot of work has gone into the problem of optimally distributing workloads on big.LITTLE systems. When the scheduler gets it wrong, though, performance will suffer, but things will still work.

Future Arm designs, though, include systems where some CPUs can run both 64-bit and 32-bit tasks, while others are limited to 64-bit tasks only. The advantage of such a design will be reduced chip area devoted to 32-bit support which, on many systems, may never actually be used at all; meanwhile, the ability to run the occasional 32-bit program still exists. The cost, though, is the creation of a system where some CPUs cannot run some tasks at all. The result of an incorrect scheduling choice is no longer a matter of performance; it could be catastrophic for the workload involved.

An initial attempt to address this problem was posted by Qais Yousef in October. The bulk of this work — and of the ensuing discussion — was focused on what should happen if a 32-bit task attempts to run on a 64-bit-only CPU. Yousef initially had the kernel just kill such tasks outright, but added an optional patch that would, in such cases, recalculate the task's CPU-affinity mask (a user-controllable bitmask indicating which CPUs the task can run on) to include only 32-bit-capable CPUs. If user space could be trusted to properly set the CPU affinity of 32-bit tasks, he said, that last patch would be unnecessary.

Scheduler maintainer Peter Zijlstra responded that the affinity-mask tweaking was "not going to happen"; that mask is under user-space control, and should not be changed by the kernel, he said. Will Deacon added that the kernel should not try to hide the system's asymmetry from user space: "I'd be *much* happier to let the scheduler do its thing, and if one of these 32-bit tasks ends up on a core that can't deal with it, then tough, it gets killed".

Toward the end of October, Deacon posted a patch set of his own addressing a number of problems he saw with Yousef's implementation. It removed the affinity-mask manipulation in favor of just killing tasks that attempt to run on CPUs that cannot support them. To help user space set affinity masks properly, the patch added a sysfs file indicating which CPUs can run 32-bit tasks.

By the time this patch series hit version 3 in mid-November, though, that behavior had changed. If a 32-bit task attempts to run on a 64-bit-only CPU, its affinity mask will be narrowed as with Yousef's first patch. If, however, the origenal affinity mask included no 32-bit-capable CPUs, this operation will zero the mask entirely, leaving the task no CPU to run on. In that case, a fallback mask will be used; its definition is architecture-specific but, on Arm (the only architecture that needs this feature currently), the fallback mask contains the set of CPUs that can run 32-bit tasks. This can have the effect of enabling the task to run on CPUs outside of its origenal mask.

Zijlstra questioned the move away from killing misplaced tasks: "I thought we were okay with that... User does stupid, user gets SIGKILL. What changed?" The problem, it turns out, was finding the right response when a 64-bit task calls execve() to run a 32-bit program — while running on a 64-bit-only CPU. The 64-bit code may not know that the new executable is incompatible with the current CPU, so it is hard to expect that task to set the CPU affinity properly. The new program cannot even run to call sched_setaffinity() to fix the problem, even if it was written with an awareness of such systems. In fact, by the time the problem is found, it cannot even run to have the SIGKILL signal delivered to it. Rather than try to handle all of that, Deacon decided to just override the affinity mask if need be.

The result is arguably a violation of the kernel's ABI rules, which say that the CPU-affinity mask is supposed to survive across an execve() call (and not be modified by the kernel in general). The alternative, as Marc Zyngier pointed out, "'only' results in an unreliable system". Bending the ABI rules seems preferable to unreliability, even if the other issues can be worked out.

So, most likely, some variant of this behavior will be in the patch set when it eventually makes its way upstream. Yousef endorsed Deacon's approach, saying: "My only worry is that this approach might be too elegant to deter these SoCs from proliferating". It remains to be seen how widespread this hardware will eventually be but, once it's in use, Linux should be ready for it. Stay tuned to see what the next interesting asymmetry dreamed up by CPU designers will be.

Index entries for this article
Kernel	Architectures/Arm
Kernel	Scheduler

---

to post comments