“Remove some entries due to various compliance requirements. They can come back in the future if sufficient documentation is provided.”
That two-line comment, submitted by major Linux kernel maintainer Greg Kroah-Hartman, accompanied a patch that removed about a dozen names from the kernle’s MAINTAINERS file. “Some entries” notably had either Russian names or .ru email addresses. “Various compliance requirements” was, in this case, sanctions against Russia and Russian companies, stemming from that country’s invasion of Ukraine.
This merge did not go unnoticed. Replies on the kernel mailing list asked about this “very vague” patch. Kernel developer James Bottomley wrote that “we” (seemingly speaking for Linux maintainers) had “actual advice” from Linux Foundation counsel. Employees of companies on the Treasury Department’s Office of Foreign Assets Control list of Specially Designated Nationals and Blocked Persons (OFAC SDN), or connected to them, will have their collaborations “subject to restrictions,” and “cannot be in the MAINTAINERS file.” “Sufficient documentation” would mean evidence that someone does not work for an OFAC SDN entity, Bottomley wrote.
There followed a number of messages questioning the legitimacy, suddenness, potentially US-forced, and non-reviewed nature of the commit, along with broader questions about the separation of open source code from international politics. Linux creator Linus Torvalds entered the thread with, “Ok, lots of Russian trolls out and about.” He wrote: “It’s entirely clear why the change was done” and noted that “Russian troll factories” will not revert it and that “the ‘various compliance requirements’ are not just a US thing.
Enlarge/ Rust never sleeps. But Rust, the programming language, can be held at bay if enough kernel programmers aren’t interested in seeing it implemented.
Getty Images
The Linux kernel is not a place to work if you’re not ready for some, shall we say, spirited argument. Still, one key developer in the project to expand Rust’s place inside the largely C-based kernel feels the “nontechnical nonsense” is too much, so he’s retiring.
Wedson Almeida Filho, a leader in the Rust for Linux project, wrote to the Linux kernel mailing list last week to remove himself as the project’s maintainer. “After almost 4 years, I find myself lacking the energy and enthusiasm I once had to respond to some of the nontechnical nonsense, so it’s best to leave it up to those who still have it in them,” Filho wrote. While thanking his teammates, he noted that he believed the future of kernels “is with memory-safe languages,” such as Rust. “I am no visionary but if Linux doesn’t internalize this, I’m afraid some other kernel will do to it what it did to Unix,” Filho wrote.
Filho also left a “sample for context,” a link to a moment during a Linux conference talk in which an off-camera voice, identified by Filho in a Register interview as kernel maintainer Ted Ts’o, emphatically interjects: “Here’s the thing: you’re not going to force all of us to learn Rust.” In the context of Filho’s request that Linux’s file system implement Rust bindings, Ts’o says that while he knows he must fix all the C code for any change he makes, he cannot or will not fix the Rust bindings that may be affected.
“They just want to keep their C code”
Asahi Lina, developer on the Asahi Linux project, posted on Mastodon late last week: “I regretfully completely understand Wedson’s frustrations.” Noting that “a subset of C kernel developers just seem determined to make the lives of Rust maintainers as difficult as possible,” Lina detailed the memory safety issues they ran into writing Direct Rendering Manager (DRM) scheduler abstractions. Lina tried to push small fixes that would make the C code “more robust and the lifetime requirements sensible,” but was blocked by the maintainer. Bugs in that DRM scheduler’s C code are the only causes of kernel panics in Lina’s Apple GPU driver, she wrote—”Because I wrote it in Rust.”
“But I get the feeling that some Linux kernel maintainers just don’t care about future code quality, or about stability or security any more,” Lina wrote. “They just want to keep their C code and wish us Rust folks would go away. And that’s really sad… and isn’t helping make Linux better.”
Drew DeVault, founder of SourceHut, blogged about Rust’s attempts to find a place inside the Kernel. In theory the kernel should welcome enthusiastic input from motivated newcomers. “In practice, the Linux community is the wild wild west, and sweeping changes are infamously difficult to achieve consensus on, and this is by far the broadest sweeping change ever proposed for the project,” DeVault writes. “Every subsystem is a private fiefdom, subject to the whims of each one of Linux’s 1,700+ maintainers, almost all of whom have a dog in this race. It’s herding cats: introducing Rust effectively is one part coding work and ninety-nine parts political work – and it’s a lot of coding work.”
Rather than test their patience with the kernel’s politics, DeVault suggests Rust developers build a Linux-compatible kernel from scratch. “Freeing yourselves of the [Linux Kernel Mailing List] political battles would probably be a big win for the ambitions of bringing Rust into kernel space,” DeVault writes.
Torvalds understands why Rust uptake is slow
You might be wondering what lead maintainer Linus Torvalds thinks about all this. He took a “wait and see” approach in 2021, hoping Rust would first make itself known in relatively isolated device drivers. At an appearance late last month, Torvalds… essentially agreed with the Rust-minded developer complaints, albeit from a much greater remove.
“I was expecting [Rust] updates to be faster, but part of the problem is that old-time kernel developers are used to C and don’t know Rust,” Torvalds said. “They’re not exactly excited about having to learn a new language that is, in some respects, very different. So there’s been some pushback on Rust.” Torvalds added, however, that “another reason has been the Rust infrastructure itself has not been super stable.”
The Linux kernel is a high-stakes project in which hundreds or thousands of developers have a stake; conflict is perhaps inevitable. Time will tell how long C will remain the primary way of coding for, and thinking about, such a large yet always-moving, codebase.
Ars has reached out to both Filho and Ts’o for comment and will update this post with response.
“Nvidia transitions fully” sounds like real commitment, a burn-the-boats call. “Towards open-source GPU,” yes, evoking the company’s “first step” announcement a little over two years ago, so this must be progress, right? But, back up a word here, then finish: “GPU kernel modules.”
So, Nvidia has “achieved equivalent or better application performance with our open-source GPU kernel modules,” and added some new capabilities to them. And now most of Nvidia’s modern GPUs will default to using open source GPU kernel modules, starting with driver release R560, with dual GPL and MIT licensing. But Nvidia has moved most of its proprietary functions into a proprietary, closed-source firmware blob. The parts of Nvidia’s GPUs that interact with the broader Linux system are open, but the user-space drivers and firmware are none of your or the OSS community’s business.
Is it better than what existed before? Certainly. AMD and Intel have maintained open source GPU drivers, in both the kernel and user space, for years, though also with proprietary firmware. This brings Nvidia a bit closer to the Linux community and allows for community debugging and contribution. There’s no indication that Nvidia aims to go further with its open source moves, however, and its modules remain outside the main kernel, packaged up for users to install themselves.
Not all GPUs will be able to use the open source drivers: a number of chips from the Maxwell, Pascal, and Volta lines; GPUs from the Turing, Ampere, Ada Lovelace, and Hopper architectures are recommended to switch to the open bits; and Grace Hopper and Blackwell units must do so.
As noted by Hector Martin, a developer on the Asahi Linux distribution, at the time of the first announcement, this shift makes it easier to sandbox closed-source code while using Nvidia hardware. But the net amount of closed-off code is about the same as before.
Nvidia’s blog post has details on how to integrate its open kernel modules onto various systems, including CUDA setups.
Linux and its code are made by people, and people are not with us forever. Over the weekend, a brief message on the Linux kernel mailing list reminded people of just how much one person can mean to a seemingly gargantuan project like Linux, and how quickly they can disappear:
Denise Finger, wife of the deceased, wrote to the Linux Wireless list on Friday evening:
This is to notify you that Larry Finger, one of your developers, passed away on June 21st.
LWN.net reckons that Finger, 84, contributed to 94 Linux kernel releases, or 1,464 commits total, at least since kernel 2.6.16 in 2006 (and when the kernel started using git to track changes). Given the sometimes precarious nature of contributing to the kernel, this is on its own an impressive achievement—especially for someone with no formal computer training, and who considered himself a scientist.
The deepest of trenches: Linux Wi-Fi in the 2000s
That kind of effort is worth celebrating, regardless. But it’s the space that Finger devoted himself to that makes him a notably patient, productive contributor.
Getting Wi-Fi to work on a device running Linux back when Finger started contributing was awful. The chances of your hardware being recognized, activated, and working properly right after install was akin to getting a straight flush in poker. If nobody had gotten around to your wireless chipset yet, you used NDISwrapper, a Windows-interfacing kludge tool that simultaneously made your Linux install less open and yet still painful to install and maintain.
Finger started fixing this with work on Broadcom’s BCM43XX drivers. Broadcom provided no code for its gear, so Finger helped reverse-engineer the necessary specs by manually dumping and reading hardware registers. Along with Broadcom drivers, Finger also provided Realtek drivers. Many commenters across blogs and message boards are noting that their systems are still using pieces of Finger’s code today.
Fixing mainframes, science gear, and RV resorts
Larry Finger, and fish, from his Quora profile.
Quora
Finger doesn’t have a large footprint on the web, outside of his hundreds of kernel commits. He has a page for DRAWxtl, for producing crystal-structure drawings, on his personal domain, but not a general personal page. He sometimes answered Quora questions. He had a GitHub profile, showing more than 100 contributions to projects in 2024.
Perhaps the biggest insight into Finger found in one place is a three-part series for Linux Journal, “Linux in a Windows Workstation Environment,” written in 2005, when he was roughly 65. He summarizes his background: Fortran programmer in 1963, PDP-11 interfaces to scientific instruments in the 1970s, VAX-11/780 work in the early 1980s, and then Unix/Linux systems, until retiring from the Carnegie Institution for Science in Washington, DC, in 1999. The mineral Fingerite is named for Finger, whose work in crystallography took him on a fellowship to northern Bavaria, as noted in one Quora answer about the Autobahn.
“At that time, I became a full-time RV resident, dedicated to the avoidance of cold weather,” Finger writes. He and his wife Denise arrived that year at a 55-plus RV community in Mesa, Arizona. He joined the computer club, which had a growing number of Windows PCs sharing a DSL connection through one of the systems running WinGate. A new RV resort owner wanted to expand to 22 workstations, but WinGate licenses for that many would have been expensive for the club. Finger, who was “highly distrustful of using Windows 98 in a mission-critical role,” set to work.
Finger goes on across the series to describe the various ways he upgraded the routing and server capacity of the network, which grew to 38 user stations, Samba shares, a membership database, VPN tunnels, several free RJ-45 ports, and “free Wi-Fi access… throughout the park.”
Lots of people have commented on the broad work Finger did to make Linux usable for more people. A few mention that Finger also mentored people, the kind of work that has exponential effects. “MB” wrote on LWN.net that Finger “mentored other people to get the Broadcom Open Source code into kernel. And I think it was a huge success. And that was only a small part of Larry’s success story.”
In a 2023 Quora response to someone asking if someone without “any formal training in computer science” can “contribute something substantial” to Linux, Finger writes, “I think that I have.” Finger links to the stats for the 6.4 kernel, showing 172,346 lines of his code in it, roughly 0.5% of the total.
I have never taken any courses in Computer Science; however, I have considerable experience in coding, much of which happened when computers were a lot less powerful than today, and it was critical to write code that ran efficiently.
Finger suggests in his response small patches, deep reading of the guidelines, and always using git’s send-email to send patches: “Nothing will get shot down more quickly than a patch submitted from a mailer such as Thunderbird.” Finding typos and errors in comments and text strings can help, especially after translation. Finger advises being patient, expecting criticism about following rules and formats, and to keep plugging away at it.
In another Quora response about kernel driver development, Finger says, “This activity can be highly rewarding, and also equally frustrating!” You should learn C, Finger suggested, and maybe start with analyzing USB drivers, and take your time learning about DMA.
“Do not lose hope,” Finger wrote. “It took me about 2 years before I could do anything more than tell the experts where my system was generating a fault.”
Enlarge/ Cans of Tab diet soda on display in 2011. Tab was discontinued in 2020. There has never been a soda named “Spaces” that had a cult following.
Getty Images
Anybody can contribute to the Linux kernel, but any person’s commit suggestion can become the subject of the kernel’s master and namesake, Linus Torvalds. Torvalds is famously not overly committed to niceness, though he has been working on it since 2018. You can see glimpses of this newer, less curse-laden approach in how Torvalds recently addressed a commit with which he vehemently disagreed. It involves tabs.
The commit last week changed exactly one thing on one line, replacing a tab character with a space: “It helps Kconfig parsers to read file without error.” Torvalds responded with a commit of his own, as spotted by The Register, which would “add some hidden tabs on purpose.” Trying to smooth over a tabs-versus-spaces matter seemed to awaken Torvalds to the need to have tab-detecting failures be “more obvious.” Torvalds would have added more, he wrote, but didn’t “want to make things uglier than necessary. But it *mightbe necessary if it turns out we see more of this kind of silly tooling.”
If you’ve read this far and don’t understand what’s happening, please allow me, a failed CS minor, to offer a quick explanation: Tabs Versus Spaces will never be truly resolved, codified, or set right by standards, and the energy spent on the issue over time could, if harnessed, likely power one or more small nations. Still, the Linux kernel has its own coding style, and it directly cites “K&R,” or Kernighan & Ritchie, the authors of the coding bible The C Programming Language, which is a tabs book. If you are submitting kernel code, it had better use tabs (eight-character tabs, ideally, though that is tied in part to teletype and line-printer history).
By attempting to smooth over one tiny part of the kernel so that a parsing tool could see a space character as a delineating whitespace, Prasad Pandit inadvertently spurred a robust rebuttal:
It wasn’t clear what tool it was, but let’s make sure it gets fixed. Because if you can’t parse tabs as whitespace, you should not be parsing the kernel Kconfig files.
In fact, let’s make such breakage more obvious than some esoteric ftrace record size option. If you can’t parse tabs, you can’t have page sizes.
Yes, tab-vs-space confusion is sadly a traditional Unix thing, and ‘make’ is famous for being broken in this regard. But no, that does not mean that it’s ok.
Torvalds’ hidden tabs appear in the fourth release candidate for Linux kernel 6.9, which Torvlads wrote had “nothing particularly unusual going on” the week of its release.
Disclosure: The author is a tab person insofar as he has any idea what he’s doing.
This post was updated at 6: 33 pm Eastern to fix some line-break issues in the Torvalds blockquote. The irony was duly noted. A better link regarding the Tabs Vs. Spaces debate was also swapped in.
Reiser, 59, serving a potential life sentence in a California prison for the 2006 murder of his estranged wife, Nina Reiser, wrote back with more than 6,500 words, which Brennan then forwarded to the LKML. It’s not often you see somebody apologize for killing their wife, explain their coding decisions around balanced trees versus extensible hashing, and suggest that elementary schools offer the same kinds of emotional intelligence curriculum that they’ve worked through in prison, in a software mailing list. It’s quite a document.
What follows is a relative summary of Reiser’s letter, dated November 26, 2023, which we first saw on the Phoronix blog, and which, by all appearances, is authentic (or would otherwise be an epic bit of minutely detailed fraud for no particular reason). It covers, broadly, why Reiser believes his system failed to gain mindshare among Linux users, beyond the most obvious reason. This leads Reiser to detail the technical possibilities, his interpersonal and leadership failings and development, some lingering regrets about dealings with SUSE and Oracle and the Linux community at large, and other topics, including modern Russian geopolitics.
“LKML and Slashdot.org seem like reasonable places to send it (as of 2006)”
In a cover letter, Reiser tells Brennan that he hopes he can use OCR to import his lengthy letter and asks him to use his best judgment in where to send his reply. He also asks, if he has time, Brennan might send him information on “Reiser5, or any interesting papers on other Filesystems, compression (especially Deep Learning based compression), etc.”
Then Reiser addresses the kernel mailing list directly—very directly:
I was asked by a kind Fredrick Brennan for my comments that I might offer on the discussion of removing ReiserFS V3 from the kernel. I don’t post directly because I am in prison for killing my wife Nina in 2006.
I am very sorry for my crime–a proper apology would be off topic for this forum, but available to any who ask.
A detailed apology for how I interacted with the Linux kernel community, and some history of V3 and V4, are included, along with descriptions of what the technical issues were. I have been attending prison workshops, and working hard on improving my social skills to aid my becoming less of a danger to society. The man I am now would do things very differently from how I did things then.
ReiserFS V3 was “our first filesystem, and in doing it we made mistakes, because we didn’t know what we were doing,” Reiser writes. He worked through “years of dark depression” to get V3 up to the performance speeds of ext2, but regrets how he celebrated that milestone. “The man I was then presented papers with benchmarks showing that ReiserFS was faster than ext2. The man I am now would stat his papers … crediting them for being faster than the filesystems of other operating systems, and thanking them for the years we used their filesystem to write ours.” It was “my first serious social mistake in the Linux community, and it was completely unnecessary.”
Reiser asks that a number of people who worked on ReiserFS be included in “one last release” of the README, and to “delete anything in there I might have said about why they were not credited.” He says prison has changed him in conflict resolution and with his “tendency to see people in extremes.”
Reiser extensively praises Mikhail Gilula, the “brightest mind in his generation of computer scientists,” for his work on ReiserFS from Russia and for his ideas on rewriting everything the field knew about data structures. With their ideas on filesystems and namespaces combined, it would be “the most important refactoring of code ever.” His analogy at the time, Reiser wrote, was Adam Smith’s ideas of how roads, waterways, and free trade affected civilization development; ReiserFS’ ideas could similarly change “the expressive power of the operating system.”
