Long-term supported distros' kernel policies are all wrong
- Reference: 1716286570
- News link: https://www.theregister.co.uk/2024/05/21/longterm_supported_distros_kernel_policies/
- Source link:
A recent blog post from Rocky Linux developer CIQ, subtitled [1]Cracks in the Ice , examines "Why a 'frozen' distribution Linux kernel isn't the safest choice for security." The post itself is an executive summary of a study the company conducted comparing the numbers of bugs and bug fixes in RHEL 8's kernel, which it has published in a white paper titled [2]Vendor Kernels, Bugs and Stability .
FOSS software, which provides the building blocks of projects like Linux distributions, is built around community development. The problem is that this is hard to monetize. Some companies have found ways to do this, but they aren't sharing enough of their work, leaving vital parts of the ecosystem starved. There are clearly visible and perfectly feasible ways around this. The snag is that implementing them would mean persuading billion-dollar companies to play nicely together.
[3]
The troublemaker behind the blog post is SAMBA co-founder (and regular Register commenter) [4]Jeremy Allison , who recently landed a [5]new job at Rocky Linux developer CIQ, after being [6]laid off from Google last year, in a decision [7]The Reg questioned at the time . (It's not Allison's first such rodeo, as we reported [8]way back in 2001 – when times are hard, even FOSS rock-star developers aren't safe.)
[9]
[10]
This is not a shocking revelation in and of itself. There is a core problem here, and as is often the case, it has at least two sides to it. The Reg FOSS desk has previously reported on both of them.
One side is that Red Hat has made FOSS software development pay, and pay very well indeed, by producing extremely slowly-changing distros, and then supporting them for a decade or more, which enables large, slow-moving organizations to run the same versions of a distro for years on end. Red Hat is the most visible player, but it isn't the only company doing this: SUSE does it too, and the Debian project does something similar.
[11]
The specific aspect of this that CIQ analyzes is that these very long-term enterprise distros pick one specific kernel version, and then keep it on life support for a decade by backporting bug-fixes – and entire new subsystems – from later kernels into it. We reported from last year's DevConf event on a Red Hat kernel developer's account of [12]What it takes to keep an enterprise 'Frankenkernel' alive .
That alone isn't the problem. The problem is that each vendor picks its own kernel version to do this with, and they don't coordinate with the team which develops the kernel. The kernel team has its own set of long-term support releases, which the enterprise vendors essentially ignore. As a result, last year the kernel team cut back its LTS kernel versions harshly, as [13]we reported from the Open Source Summit in September. The overworked and understaffed kernel team is cutting the support lifetime for its LTS kernels from six years to just two. At present, there are six [14]long term releases , one of which goes end-of-life at the end of this year, and another next year.
We feel that we must point out that CIQ does have a reason for highlighting this issue. Its Rocky Linux distro is a third-party rebuild of RHEL, and since [15]Red Hat stopped sharing all its source code with the world in June last year, that means that CIQ can't readily obtain RHEL's kernel source code any more. Although later in June CIQ [16]announced it had a workaround for this , which in early July [17]it described in more detail , questions remain over how long this can work.
[18]
So it is perhaps not a big shock that CIQ is now endorsing LTS kernel supremo Greg Kroah-Hartman's advice, which we quoted in our kernel-team story:
You have to take all of the stable/LTS releases in order to have a secure and stable system.
In other words, the kernel team feels that the only really secure option is its LTS options – whereas, as we quoted in the Frankenkernel story, Red Hat's kernel maintainers feel that its kernels are better-tested and safer. When we read this from CIQ, our first thought was [19]Mandy Rice-Davies applies .
If the enterprise Linux industry can somehow be forced to grasp the nettle, there is an obvious answer to this. We pointed it out when we looked at the [20]OpenELA extension of support for 4.14 announced in March.
All that is needed is for the enterprise distro vendors to choose the upstream long term support kernel releases for their long term support distro releases. Then, this problem would effectively go away, replaced by the smaller issue of sharing their bug-fixes upstream.
If Red Hat, SUSE, Debian and the other big distros confined themselves to LTS kernels and shared the burden of maintaining them, it would lower their costs. It wouldn't hurt the bottom line. The customers paying for double-digit years of updates don't care what specific version is supported, just so long as there is a version that keeps getting updates.
[21]Red Hat middleware takes a back seat in strategic shuffle
[22]Linux kernel 4.14 gets a life extension, thanks to OpenELA
[23]Rocky Linux details the loopholes that will help its RHEL rebuild live on
[24]Red Hat strikes a crushing blow against RHEL downstreams
There's even an obvious exemption here to aid monetization: exclude subsystem and driver backports from the patches that are shared upstream. If you want to run an old distro but keep getting new functionality and new device support, then you pay for an enterprise distro. If you don't want to pay, then run a newer kernel. It's as easy as that.
We're sure that some commentators will be able to come up with elaborate justifications why this is completely impossible, to which we pre-emptively respond: [25]cui bono? Keeping their kernels in-house is part of how these companies make big money from a free OS. If you want to know why things can't change even if it would help everyone, the answer is, as William Goldman put it, "Follow the money." ®
Get our [26]Tech Resources
[1] https://ciq.com/blog/why-a-frozen-linux-kernel-isnt-the-safest-choice-for-security/
[2] https://ciq.com/whitepaper/vendor-kernels-bugs-stability/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZkzFKRH2SfrEkBf-Ssjy1QAAAE4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.samba.org/~jra/
[5] https://x.com/jra_samba/status/16423191488009707520
[6] https://x.com/jra_samba/status/1616585564043767808
[7] https://www.theregister.com/2023/01/27/google_open_source/
[8] https://www.theregister.com/2001/09/10/jeremy_allison_sambas_out/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZkzFKRH2SfrEkBf-Ssjy1QAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZkzFKRH2SfrEkBf-Ssjy1QAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZkzFKRH2SfrEkBf-Ssjy1QAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2023/06/30/enterprise_distro_feature_devconf/
[13] https://www.theregister.com/2023/09/26/linux_kernel_report_2023/
[14] https://www.kernel.org/category/releases.html
[15] https://www.theregister.com/2023/06/23/red_hat_centos_move/
[16] https://www.theregister.com/2023/06/28/rocky_linux_rhel_ripples/
[17] https://www.theregister.com/2023/07/04/rocky_linux_rhel_loopholes/
[18] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZkzFKRH2SfrEkBf-Ssjy1QAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[19] https://en.wikipedia.org/wiki/Well_he_would,_wouldn%27t_he%3F
[20] https://www.theregister.com/2024/03/19/kernel_414_life_extension/
[21] https://www.theregister.com/2024/05/20/red_hat_prunes_middleware/
[22] https://www.theregister.com/2024/03/19/kernel_414_life_extension/
[23] https://www.theregister.com/2023/07/04/rocky_linux_rhel_loopholes/
[24] https://www.theregister.com/2023/06/23/red_hat_centos_move/
[25] https://www.merriam-webster.com/dictionary/cui%20bono
[26] https://whitepapers.theregister.com/
Re: I'm missing something
They are cutting LTS to 2 years because they don't have the resources to maintain them.
If enterprise Linux companies chip in with upstream patches, they don't need as many resources to maintain those LTS'.
IBM long-term vs. CIQ/SUSE/Oracle long-term
The industry has grown familiar with RHEL-compatible LTS kernels, the source of which is now controlled by IBM, who would very much like to monetize it more strongly.
The question is if CIQ/SUSE/Oracle (as members of OpenELA) are willing to provide an alternative.
Oracle already distributes their own custom kernel (the UEK), but it does not occupy the same niche and it has a much smaller development team.
"Follow the money."
Who would have thought that the parasite corporations greed is a source of problems. Must have been a new ground breaking AI used that lead this study to come up with the blinding obvious for literally well over a century of experience with it happening.
RE: "Who would have thought"
No, no, never.
Don't you know that everything that happens today is original ?? Shirley we could never learn from those *old* things back then!
I've noticed odd happenings where Redhat Fedora and FOSS databases collide
Specifically, about a month ago I was running Fedora 38 and wanted to update PostgreSQL to its latest version (14 or 15), but couldn't get that to work. Further research showed that PostgreSQL recommended not installing it on any release of Redhat Linux later than 38 but didn't give any reasons for this advice.
Consequently, I decided to switch to MariaDB. Annoyingly, this hasn't helped because the current Fedora 39's MariaDB packages don't include a package which the current version of MariaDB requires for the process of creating a new database copy to complete. Despite this package being mentioned as a requirement in the database creation recipe, its not included in the Fedora 39 download library. This the situation is not at all helped by the mess otherwise known as the MariaDB manual lacking an index.
I'm about to upgrade to Fedora 40 and then try again, but am uncertain which relational database to try installing after the upgrade: PostgtreSQL, MariaDB, or Firebird H2 or Derby.
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
If you're debating postgres vs H2 then you need to take another look at your requirements.
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
Prior to the move to Fedora 39 and a simultaneous move new hardware (previously on an elderly Dual Athlon box and now on a Ryzen-based desktop) I was using PostgreSQL to implement an email archive, so a fairly simple DB schema: basically a single entity type (each instance an email, indexed by sender, date and title), updated once a day with new email after rejecting spam and mail ffrom unwanted senders. This worked fine until the the old box crashed and it turned out that there was an incompatibilitr brtween Fedora 38 and Postgres.
I also wrote search tool and a data management toolfor the archive, both in Java and using JDBC because my DB loaders etc are all Java..
I initially used PostgreSQL to find out about it (my DBA experience before this includes Access, IDMSX, Sybase, and RedBrick (one if the first data wharehouses).
After an upgrade to Fedora 40 I'll see if the PostgreSQL incompatibility still exists. If it does, I'd appreciate experience with FOSS databases such as Firebird, H2, HSQL, or SQLite, though I'd prefer to stick with PostgreSQL since I have known schemas, JDBC modules etc have been well tested with it.
Any suggestions and/or comments appreciated.
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
A simpler solution might be to run it in a Docker container.
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
simpler is to stop using rhat crap
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
Which crap? Fedora? Postgres? Or Docker? I'm not a huge fan of Docker but it does make it easy to have unified developer environments, especially in cases like this.
Re: I've noticed odd happenings where Redhat Fedora and FOSS databases collide
or try not using Rhat crap
The snag is that implementing them would mean persuading billion-dollar companies to play nicely together.
Be careful what you wish for. The snag with businesses playing nicely together is that, as Adam Smith pointed, it always ends in "a conspiracy against the public".
[Author here]
Yes, in the long run. But:
"In the long run, we're all dead." (J M Keynes.)
FOSS _can_ be an exception to this.
FreeBSD got it right.
Clear and consistenty strategy and systems that measure uptimes in decades. AFAICS Linux just has more drivers.
Re: FreeBSD got it right.
That really depends what you're trying to get FreeBSD to do. As a desktop Linux is at least one order of magnitude of an improvement, and possibly more.
As a server whilst FreeBSD has some decent technologies, it is not uncommon to find a similar functionality gulf between BSD and Linux.
I'm trying to move off Windows 10, and use FreeBSD as my Unix of choice. Even in areas you'd expect it to be strong I'm running into situations where it's almost immediately necessary to write your own code or scripts, and any pre-existing work is applied to the very specific use case of whoever wrote it, with limited flexability or error checking.
Thinking logically I would not recommend FreeBSD as first choice unless the BSD license offers substantial benefits. However, I'm a fan of BSD in general, so I continue to learn/waste my time getting it to achieve what I need. I *would* recommend OpenBSD, provided you're only using it for firewalling or its base networking, and you're not using wireless.
For quality of experience I would very definitely rate it below what mid nineties OS/2 offered at the time.
I'm missing something
If the FOSS community have switched to LTS versions where L = 2 years and RHEL (et al) want L to be 10 years then how does persuading RHEL to choose a FOSS designated LTS version change anything? After 2 years their back-port efforts won't be applicable just as at present?