What it takes to keep an enterprise 'Frankenkernel' alive
- Reference: 1688119684
- News link: https://www.theregister.co.uk/2023/06/30/enterprise_distro_feature_devconf/
- Source link:
A [1]talk by Red Hat Principal Kernel Engineer Jiří Benc at this year's [2]DevConf.cz event covered some of the inherent contradictions in keeping an enterprise distro's kernel on its feet. Or at least on somebody – or something's – feet, as its title hinted: "CentOS Frankenkernel: Append Your Limb."
He focused on the kernel of CentOS Stream, which in time will be the kernel of the next point-release of RHEL 9 – at the time of writing, that will be RHEL 9.3, but like the other versions of RHEL 9, this will have kernel 5.14 – released way back on August 29 2021. How do they achieve this?
[3]
The goals of any kernel update are simple: stability, obviously. No regressions, and that also means no performance regressions. No API changes, and no internal ABI changes either: in fact, no changes in behaviour. But, at the same time, customers want new features, and support for new hardware, including new drivers; they want updates, at a minimum any outstanding security updates. All without breaking whatever they are currently using, because that's what they are paying for.
[4]
[5]
This is a big ask, and the result must inevitably be a compromise. The team tries to deliver no functional regressions, and to limit performance regressions to the important stuff. To make no backwards-incompatible microAPI changes, and to avoid kernel ABI changes for important stuff. The problem is that people want new features… and new or updated drivers.
So, what the team are working on is a Frankenstein's monster, sewn together from different codebases. Although the base kernel is still version 5.14, it is full of backports from upstream. It has the XFS filesystem code from kernel 6.0, the USB subsystem – complete with drivers – and BPF subsystem from kernel 6.2, the wireless stack and all drivers from kernel 6.3, and the multipath TCP/IP code from kernel 6.4 – which at the time of the talk hadn't even been released upstream yet. (It was [6]released last weekend .)
[7]It's 2023 and memory overwrite bugs are not just a thing, they're still number one
[8]Rocky Linux claims to have found 'path forward' from CentOS source purge
[9]Inclusive Naming Initiative limps towards release of dangerous digital dictionary
[10]A (cautionary) tale of two patched bugs, both exploited in the wild
It works because of a lot of testing and a very cautious release process. Of course, the developer themselves tests it, but it also undergoes continuous-integration testing thanks to tools from the [11]CKI project , as well as network-stack testing using the [12]LNST tools. Then, it undergoes preverification, meaning that a human – someone other than the author – manually checks the change. Only then is the change merged into the CentOS kernel tree, after which it undergoes integration testing: checks against another 150 or so work-in-progress changes. Then, once it's passed all those, it undergoes normal QA testing with the rest of the OS.
The results can be seen on the CentOS Stream [13]Gitlab – Benc was keen to stress this all happens in public, and it's all [14]documented . Indeed, anyone can open a request for such a change, by filing a bug on Bugzilla, or opening a JIRA issue, according to a prescribed format: Product … Version … Component … Subcomponent … Benefit … Tests . Similarly, there's also a very strict [15]format for merge requests (which are Gitlab's equivalent of Github pull requests), and for commit messages – and it must be followed exactly, because the messages are parsed by machines as well as by humans.
[16]
So long as the format is followed precisely, then the automation kicks in. It adds lots of labels, checks for subsequent fixes and patches from upstream, tags various people who must inspect and check the change, and more. All the discussion is handled in the comments on MR itself on Gitlab – except for dependencies, such as drivers, because Gitlab can't currently handle that.
If you listen very carefully to the [17]Youtube stream of the talk, the first question was from the Reg FOSS desk, asking if this didn't overlap with the work of the long-term-support [18]releases from the upstream kernel developers. Benc told us that he feels that Red Hat's level of testing and quality control exceeds that of the upstream LTS kernels, and that they don't deliver the level of stability that an enterprise distro needs.
That was quite surprising to us, but this is an undeniably impressive amount of work and level of attention to detail. In the light of the [19]continuing furore that has followed [20]Red Hat's withdrawal of the RHEL source code from publication, the talk emphasized the sheer amount of work that goes into maintaining a distro, complete with a single version of the kernel, for a life cycle of a whole decade. RHEL 9.10, for instance, is not [21]planned to go out of support until 2032.
[22]
This is the work that Red Hat wants to get paid for, and the reason that it is still trying to find ways to exclude the downstream rebuilds – as it [23]has been doing for a dozen years . ®
[24]Youtube Video
Get our [25]Tech Resources
[1] https://devconfcz2023.sched.com/event/1MYlC/centos-frankenkernel-append-your-limb
[2] https://www.devconf.info/cz/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZJ78IuA9UKt1AOsBa9Ai2AAAAIc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJ78IuA9UKt1AOsBa9Ai2AAAAIc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZJ78IuA9UKt1AOsBa9Ai2AAAAIc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/06/26/linux_kernel_6_4_released/
[7] https://www.theregister.com/2023/06/29/cwe_top_25_2023/
[8] https://www.theregister.com/2023/06/28/rocky_linux_rhel_ripples/
[9] https://www.theregister.com/2023/06/22/inclusive_naming_initiative_word_list/
[10] https://www.theregister.com/2023/06/21/vmware_bug_under_exploit/
[11] https://cki-project.org/
[12] https://github.com/LNST-project/lnst
[13] https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9
[14] https://redhat.gitlab.io/centos-stream/src/kernel/documentation/docs/rhel_kernel_workflow.html
[15] https://redhat.gitlab.io/centos-stream/src/kernel/documentation/docs/commitrules.html
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJ78IuA9UKt1AOsBa9Ai2AAAAIc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[17] https://www.youtube.com/live/nS0Z1OilOas?feature=share
[18] https://kernel.org/releases.html
[19] https://www.theregister.com/2023/06/28/rocky_linux_rhel_ripples/
[20] https://www.theregister.com/2023/06/23/red_hat_centos_move/
[21] https://access.redhat.com/support/policy/updates/errata#RHEL8_and_9_Life_Cycle
[22] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZJ78IuA9UKt1AOsBa9Ai2AAAAIc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[23] https://www.theregister.com/2011/03/04/red_hat_twarts_oracle_and_novell_with_change_to_source_code_packaging/
[24] https://www.youtube.com/watch?v=nS0Z1OilOas&t=22050s
[25] https://whitepapers.theregister.com/
Re: Just because it's hard doesn't mean you're in the right
I do sympathise to some extent, but you can't have your cake and eat it. Want to do as you wish? Rewrite/with permission relicense all necessary code.
This.... Red Hat has built it's business atop of a GPL-2 codebase (in the case of the Kernel), it has been successful too.
According to Wikipedia it was founded some 30 years ago and has annual revenue in the billions, yet suddenly it seems like the opinions of others who licence the code it builds on top of means nothing because Red Hat engineers work oh so very hard or something??
If you can grow a company to total assets over 5 billion dollars, whilst complying with your licence requirements, then it doesn't really seem that the model it's built on needs to change.
If it does need to change then fork FreeBSD (used by many servers around the world), and start selling Red Had Enterprise BSD. You could keep the source code completely closed, whilst still taking the work of others from upstream as this is precisely what the BSD licence allows. (See Apple).
But whilst you build atop of the work licenced by people who would like the improvements you make to be freely distributable in source form then it's probably good form to listen to them.
Re: Just because it's hard doesn't mean you're in the right
[Author here]
> Rewrite/with permission relicense all necessary code.
But the Hat *does*. It is all out there.
The kernel tree is right there on Gitlab.
Want the latest bleeding-edge dev code? Fedora Rawhide. Free of charge, no licensing restrictions.
https://docs.fedoraproject.org/en-US/releases/rawhide/
Want semi-stabilised semi-annual releases, with everything fresh and current, but at least tested and assembled into a coherent whole? Just like Ubuntu, but you can develop your skills on the RH tools, RH commands, RH config files in RH locations in RH syntax?
Fedora. 100% FOSS, no proprietary code, free open licence, use it for whatever.
https://fedoraproject.org/
Want the stable subset, branched off Fedora every couple of years or so, which allows you to contribute code, fixes, etc.? CentOS Stream.
https://www.centos.org/centos-stream/
There is nothing in RHEL you can't get from them.
What you pay for is tonnes of testing and integration work, rapid fixes, and enterprise support.
Only want to test? Only want a few boxes? Gratis to you. Free of charge. Free as in beer. Includes source code access.
https://developers.redhat.com/products/rhel/download
RH is not holding back code. It is not making anything proprietary. It just wants money for the oldest slowest-moving slowest-changing distro there is in the world.
It pays for a huge amount of FOSS development and this is how it funds it.
Nothing secret, nothing proprietary, nothing relicensed. All 100% GPL 1/2/3/AGPL compliant.
What IMHO it really wants to do is stop Oracle making money from RHEL. That seems fair to me TBH.
Re: Just because it's hard doesn't mean you're in the right
The linked stream is over 8 hours. Where does the talk come in that?
Re: Just because it's hard doesn't mean you're in the right
[Author here]
> Where does the talk come in that?
Sorry about that!
Somehow the time code got lost. The talk begins at 6h 07m 30s. In theory, this is a direct link:
https://www.youtube.com/live/nS0Z1OilOas?feature=share&t=22050
> No API changes, and no internal ABI changes either
This is a bit disingenuous. The so called "Stable KABI" almost *always* breaks at a RHEL point release. And since this is Stream and the kernel will be continually updated with new changes during the lifetime of one RHEL point release so I would expect multiple KABI changes to happen during Stream's lifetime between one RHEL point release and the next. If you run RHEL then you just get used to the "stable" KABI not being stable over a point release. If you run Stream then it could break at any time.
"The goals of any kernel update are simple: [..] no changes in behaviour"
Somebody needs to brief Borkzilla on that.
Good article, but the headline made me think it was an "On Call" story about a Linux Sysadmin who'd been forced to build their own Linux Kernel, using bits from other kernels because their employer has a strange set of requirements that couldn't be completely met with one distro..
Missing fixes are hard to detect, one must not use these dangerous kernels
I'm not deying Jiri's massive amount of testing, but testing for absence of regression doesn't mean testing for lack of known bugs. The SOLE purpose of LTS kernels is to provide fixes for all known bugs. It should be seen as a collection of fixes. Whenever you skip a fix from a stable branch to maintain your own, you're in fact keeping a bug that was already fixed in -stable. This is almost undetectable, unless, of course, you know how to test all bugs. But given that reporters themselves don't always know how to test them and only rely on long observation, you cannot verify that you're having all needed fixes.
So please, distro vendors need to really stop this madness of reinventing a parallel maintenance effort that does not involve users. This huge amount of work would be so much better spent testing LTS kernels! And this particular vendor got caught in the past with severe local vulnerability (local privilege escalation IIRC) that had been fixed in mainline and stable something like two years ago but still present in their kernel as not identified as needed.
Sure, stable and LTS kernels occasionally regress. When they do so, they're immediately reverted. Nobody's asking vendors to ship the very latest patch, it's perfectly fine if they emit one release out of 5 after much longer testing. But it's really important that they closely follow the stream of fixes that go into mainline and -stable. And if they're doing the backport themselves, considering how many times a subsystem maintainer disagrees with a backport and proposes another one, that's a luxury they don't have here and they probably keep a list of incorrect or incomplete backports without knowing. I'm still really really irritated whenever I see non-LTS kernels on LTS distros. This definitely does not contribute to the perception of Linux' quality nor security in field.
Just because it's hard doesn't mean you're in the right
I do sympathise to some extent, but you can't have your cake and eat it. Want to do as you wish? Rewrite/with permission relicense all necessary code.
This is also still a considerable level below what Microsoft does with Windows. If you read Raymond Chen's Old New Thing blog you'll know Windows has custom installer code built in to shim 32 bit programs that still shipped with 16 bit installers, custom memory allocators to work around broken memory allocation, and many other shims to allow for incorrectly coded but popular applications.
Also, Windows has a driver model that doesn't involve sticking everything into the kernel, and moving functionality where possible up to userland/ring 3. Although I'll admit I'm not up to date on how much progress Linux has made moving driver functionality outside kernel space.
Alternatively, the OpenBSD route could be taken : security first, (almost) no compromises. If the change breaks applications, the applications have to fix it. In the end it works but can involve a lot of short to medium term pain.