“Remove some entries due to various compliance requirements. They can come back in the future if sufficient documentation is provided.”
That two-line comment, submitted by major Linux kernel maintainer Greg Kroah-Hartman, accompanied a patch that removed about a dozen names from the kernle’s MAINTAINERS file. “Some entries” notably had either Russian names or .ru email addresses. “Various compliance requirements” was, in this case, sanctions against Russia and Russian companies, stemming from that country’s invasion of Ukraine.
This merge did not go unnoticed. Replies on the kernel mailing list asked about this “very vague” patch. Kernel developer James Bottomley wrote that “we” (seemingly speaking for Linux maintainers) had “actual advice” from Linux Foundation counsel. Employees of companies on the Treasury Department’s Office of Foreign Assets Control list of Specially Designated Nationals and Blocked Persons (OFAC SDN), or connected to them, will have their collaborations “subject to restrictions,” and “cannot be in the MAINTAINERS file.” “Sufficient documentation” would mean evidence that someone does not work for an OFAC SDN entity, Bottomley wrote.
There followed a number of messages questioning the legitimacy, suddenness, potentially US-forced, and non-reviewed nature of the commit, along with broader questions about the separation of open source code from international politics. Linux creator Linus Torvalds entered the thread with, “Ok, lots of Russian trolls out and about.” He wrote: “It’s entirely clear why the change was done” and noted that “Russian troll factories” will not revert it and that “the ‘various compliance requirements’ are not just a US thing.
Enlarge/ Cutting metal with lasers is hard, but even harder when you don’t know the worst-case timings of your code.
Getty Images
As is so often the case, a notable change in an upcoming Linux kernel is both historic and no big deal.
If you wanted to use “Real-Time Linux” for your audio gear, your industrial welding laser, or your Mars rover, you have had that option for a long time (presuming you didn’t want to use QNX or other alternatives). Universities started making their own real-time kernels in the late 1990s. A patch set, PREEMPT_RT, has existed since at least 2005. And some aspects of the real-time work, like NO_HZ, were long ago moved into the mainline kernel, enabling its use in data centers, cloud computing, or anything with a lot of CPUs.
But officialness still matters, and in the 6.12 kernel, PREEMPT_RT will likely be merged into the mainline. As noted by Steven Vaughan-Nichols at ZDNet, the final sign-off by Linus Torvalds occurred while he was attending Open Source Summit Europe. Torvalds wrote the original code for printk, a debugging tool that can pinpoint exact moments where a process crashes, but also introduces latency that runs counter to real-time computing. The Phoronix blog has tracked the progress of PREEMPT_RT into the kernel, along with the printk changes that allowed for threaded/atomic console support crucial to real-time mainlining.
What does this mean for desktop Linux? Not much. Beyond high-end audio production or replication (and even that is debatable), a real-time kernel won’t likely make windows snappier or programs zippier. But the guaranteed execution and worst-case latency timings a real-time Linux provides are quite useful to, say, the systems that monitor car brakes, guide CNC machines, and regulate fiendishly complex multi-CPU systems. Having PREEMPT-RT in the mainline kernel makes it easier to maintain a real-time system, rather than tend to out-of-tree patches.
It will likely change things for what had been, until now, specialty providers of real-time OS solutions for mission-critical systems. Ubuntu, for example, started offering a real-time version of its distribution in 2023 but required an Ubuntu Pro subscription for access. Ubuntu pitched its release at robotics, automation, embedded Linux, and other real-time needs, with the fixes, patches, module integration, and testing provided by Ubuntu.
“Controlling a laster with Linux is crazy,” Torvalds said at the Kernel Summit of 2006, “but everyone in this room is crazy in his own way. So if you want to use Linux to control an industrial welding laser, I have no problem with your using PREEMPT_RT.” Roughly 18 years later, Torvalds and the kernel team, including longtime maintainer and champion-of-real-time Steven Rostedt, have made it even easier to do that kind of thing.
Enlarge/ Rust never sleeps. But Rust, the programming language, can be held at bay if enough kernel programmers aren’t interested in seeing it implemented.
Getty Images
The Linux kernel is not a place to work if you’re not ready for some, shall we say, spirited argument. Still, one key developer in the project to expand Rust’s place inside the largely C-based kernel feels the “nontechnical nonsense” is too much, so he’s retiring.
Wedson Almeida Filho, a leader in the Rust for Linux project, wrote to the Linux kernel mailing list last week to remove himself as the project’s maintainer. “After almost 4 years, I find myself lacking the energy and enthusiasm I once had to respond to some of the nontechnical nonsense, so it’s best to leave it up to those who still have it in them,” Filho wrote. While thanking his teammates, he noted that he believed the future of kernels “is with memory-safe languages,” such as Rust. “I am no visionary but if Linux doesn’t internalize this, I’m afraid some other kernel will do to it what it did to Unix,” Filho wrote.
Filho also left a “sample for context,” a link to a moment during a Linux conference talk in which an off-camera voice, identified by Filho in a Register interview as kernel maintainer Ted Ts’o, emphatically interjects: “Here’s the thing: you’re not going to force all of us to learn Rust.” In the context of Filho’s request that Linux’s file system implement Rust bindings, Ts’o says that while he knows he must fix all the C code for any change he makes, he cannot or will not fix the Rust bindings that may be affected.
“They just want to keep their C code”
Asahi Lina, developer on the Asahi Linux project, posted on Mastodon late last week: “I regretfully completely understand Wedson’s frustrations.” Noting that “a subset of C kernel developers just seem determined to make the lives of Rust maintainers as difficult as possible,” Lina detailed the memory safety issues they ran into writing Direct Rendering Manager (DRM) scheduler abstractions. Lina tried to push small fixes that would make the C code “more robust and the lifetime requirements sensible,” but was blocked by the maintainer. Bugs in that DRM scheduler’s C code are the only causes of kernel panics in Lina’s Apple GPU driver, she wrote—”Because I wrote it in Rust.”
“But I get the feeling that some Linux kernel maintainers just don’t care about future code quality, or about stability or security any more,” Lina wrote. “They just want to keep their C code and wish us Rust folks would go away. And that’s really sad… and isn’t helping make Linux better.”
Drew DeVault, founder of SourceHut, blogged about Rust’s attempts to find a place inside the Kernel. In theory the kernel should welcome enthusiastic input from motivated newcomers. “In practice, the Linux community is the wild wild west, and sweeping changes are infamously difficult to achieve consensus on, and this is by far the broadest sweeping change ever proposed for the project,” DeVault writes. “Every subsystem is a private fiefdom, subject to the whims of each one of Linux’s 1,700+ maintainers, almost all of whom have a dog in this race. It’s herding cats: introducing Rust effectively is one part coding work and ninety-nine parts political work – and it’s a lot of coding work.”
Rather than test their patience with the kernel’s politics, DeVault suggests Rust developers build a Linux-compatible kernel from scratch. “Freeing yourselves of the [Linux Kernel Mailing List] political battles would probably be a big win for the ambitions of bringing Rust into kernel space,” DeVault writes.
Torvalds understands why Rust uptake is slow
You might be wondering what lead maintainer Linus Torvalds thinks about all this. He took a “wait and see” approach in 2021, hoping Rust would first make itself known in relatively isolated device drivers. At an appearance late last month, Torvalds… essentially agreed with the Rust-minded developer complaints, albeit from a much greater remove.
“I was expecting [Rust] updates to be faster, but part of the problem is that old-time kernel developers are used to C and don’t know Rust,” Torvalds said. “They’re not exactly excited about having to learn a new language that is, in some respects, very different. So there’s been some pushback on Rust.” Torvalds added, however, that “another reason has been the Rust infrastructure itself has not been super stable.”
The Linux kernel is a high-stakes project in which hundreds or thousands of developers have a stake; conflict is perhaps inevitable. Time will tell how long C will remain the primary way of coding for, and thinking about, such a large yet always-moving, codebase.
Ars has reached out to both Filho and Ts’o for comment and will update this post with response.
Enlarge/ Cans of Tab diet soda on display in 2011. Tab was discontinued in 2020. There has never been a soda named “Spaces” that had a cult following.
Getty Images
Anybody can contribute to the Linux kernel, but any person’s commit suggestion can become the subject of the kernel’s master and namesake, Linus Torvalds. Torvalds is famously not overly committed to niceness, though he has been working on it since 2018. You can see glimpses of this newer, less curse-laden approach in how Torvalds recently addressed a commit with which he vehemently disagreed. It involves tabs.
The commit last week changed exactly one thing on one line, replacing a tab character with a space: “It helps Kconfig parsers to read file without error.” Torvalds responded with a commit of his own, as spotted by The Register, which would “add some hidden tabs on purpose.” Trying to smooth over a tabs-versus-spaces matter seemed to awaken Torvalds to the need to have tab-detecting failures be “more obvious.” Torvalds would have added more, he wrote, but didn’t “want to make things uglier than necessary. But it *mightbe necessary if it turns out we see more of this kind of silly tooling.”
If you’ve read this far and don’t understand what’s happening, please allow me, a failed CS minor, to offer a quick explanation: Tabs Versus Spaces will never be truly resolved, codified, or set right by standards, and the energy spent on the issue over time could, if harnessed, likely power one or more small nations. Still, the Linux kernel has its own coding style, and it directly cites “K&R,” or Kernighan & Ritchie, the authors of the coding bible The C Programming Language, which is a tabs book. If you are submitting kernel code, it had better use tabs (eight-character tabs, ideally, though that is tied in part to teletype and line-printer history).
By attempting to smooth over one tiny part of the kernel so that a parsing tool could see a space character as a delineating whitespace, Prasad Pandit inadvertently spurred a robust rebuttal:
It wasn’t clear what tool it was, but let’s make sure it gets fixed. Because if you can’t parse tabs as whitespace, you should not be parsing the kernel Kconfig files.
In fact, let’s make such breakage more obvious than some esoteric ftrace record size option. If you can’t parse tabs, you can’t have page sizes.
Yes, tab-vs-space confusion is sadly a traditional Unix thing, and ‘make’ is famous for being broken in this regard. But no, that does not mean that it’s ok.
Torvalds’ hidden tabs appear in the fourth release candidate for Linux kernel 6.9, which Torvlads wrote had “nothing particularly unusual going on” the week of its release.
Disclosure: The author is a tab person insofar as he has any idea what he’s doing.
This post was updated at 6: 33 pm Eastern to fix some line-break issues in the Torvalds blockquote. The irony was duly noted. A better link regarding the Tabs Vs. Spaces debate was also swapped in.
