Long-term support for Linux kernels is about to get a lot shorter
(2023/09/26)
- Reference: 1695733211
- News link: https://www.theregister.co.uk/2023/09/26/linux_kernel_report_2023/
- Source link:
Open Source Summit This year's Kernel Report at the Open Source Summit in Bilbao revealed the long-term support releases of the Linux kernel will soon not be that long at all.
Referring back to the list of stable kernels, which goes back about six years, [1]Linux Weekly News editor Jonathan Corbet [2]said :
They have come to the conclusion that there's really no point in maintaining them that long, because people are not using them. So the six-year update policy is going away. When 4.14 goes out of support… early next year… there will not be another six-year kernel to replace it. We're likely to go back to a long-term stable kernel maintained for about two years. After that, people will simply be expected to update to a newer kernel.
The new regime
When 4.19 reaches its end of life, so will 5.4, leaving the [3]end-of-2020 kernel 5.10 the oldest longterm release. This change [4]was discussed at the time , but now it looks like what is essentially the worst-case outcome has been chosen.
Since the big enterprise players don't use the existing long-term releases, maintaining their own instead, and as is now clear in the case of the 800-pound gorilla of the Linux world, keeping that kernel and its source code [5]reserved for customers only , the kernel developers are dropping the very considerable maintenance burden of keeping those old releases alive.
Bring out the popcorn, it's the Kernel Report
For techie types tired of PowerPoint presentations and words like "synergy" and "leverage," Corbet's [6]Kernel Report is one of the highlights of what used to be called LinuxCon.
This year's [7]presentation [PDF] led in gently with some statistics, and held the big reveal about shortening support cycles back until later in the talk. There have been five releases in version 6 so far, and they are all of comparable size, with some 15,000 commits in each. Each point release has seen some 2,000 individual developers contributing, of whom around 250 to 300 are newcomers. As Corbet put it:
It's really the same old chart that I put up every time I do this talk. It doesn't change a whole lot… but it is worth pointing out that 6.2, released back in February, set a new record for the number of developers participating.
He noted that those are "mainline" releases, the ones Linus Torvalds himself puts out before moving on to the next new version. But, he said, "these are not the kernels that most of us are actually running. Most people are smarter than that and try to run something else; usually, something based off the stable kernel updates."
At present, there are six releases under long-term maintenance: versions 6.1, 5.15, 5.10, 5.4, 4.19, and 4.14 from [8]back in November 2017 . The number of commits into each of these long-term kernels increases for each older version, with the oldest having over 300 updates and by now about 28,000 commits.
So which should you choose? Corbet – himself a kernel maintainer – says that one answer to this is simple:
Run the latest stable update. That is the most stable, the most secure, the best kernel we know how to create at this time. That's the very best we can do. You should run that.
Manager of the stable kernels Greg Kroah-Hartman's explanation is:
You have to take all of the stable/LTS releases in order to have a secure and stable system. If you attempt to cherry-pick random patches you will not fix all of the known, and unknown, problems, but rather you will end up with a potentially more insecure system, and one that contains known bugs.
Corbet went on to point out that the oldest long-term kernel is now approaching two full development cycles' worth of commits: "An awful lot of development is going on after our stable kernel is released." Many of the fixes are for bugs that appeared in earlier kernel releases – his analysis goes back to the beginning of the history in Git, back with kernel 2.6.12 in 2005.
There's an awful lot of bugs that lurk in the kernel for a long time before turning up somewhere and needing to be fixed. All this stuff gets scooped up and put into the stable kernel releases.
And this goes both ways: bugs discovered and fixed in later kernel versions must be back-ported and fixed in older, stable releases.
This, he pointed out, is the other way to choose which kernel to run: pick an old, stable kernel, and backport all the fixes that you consider important to it. It's the "enterprise" kernel model, and leads to old kernels with thousands and thousands of fixes applied to them. "We've seen an awful lot of fuss recently about what certain enterprise vendors are doing with regard to access to their code and their distributions, and the fuss around Red Hat Enterprise Linux in general. This here is what that fuss is about," he said.
[9]
It's a lot of work, and although it applies across the whole distribution, a huge amount is at the kernel level. What you're left with are old kernels, but with tens of thousands of fixes, leaving them very unlike what was originally released. The result, Corbet said, is that it cuts their users off from community support.
[10]
[11]
In a way, this is the flip side of Red Hat's view, which [12]we described back in June . Red Hat considers that the levels of testing and quality control in its enterprise kernels exceed that of the community ones. As a result, it maintains its own stable versions, and it doesn't use the upstream long-term releases.
That's when Corbet dropped his LTS bombshell.
[13]
There was a lot more in the talk, and we recommend watching it in full if you want to know where things are going. He discussed the new extended Berkeley Packet Filters (eBPF) subsystem, where it was going, and importantly, where it isn't – such as vendor-specific kernel schedulers. There was also discussion of security, confidential computing, and more.
[14]GNU turns 40: Stallman's baby still not ready for prime time, but hey, there's cake
[15]If GNU please: Rust support merged for the forthcoming GCC 13
[16]Linux 6.1: Rust to hit mainline kernel
[17]How Citrix dropped the ball on Xen ... according to Citrix
The much-vaunted new Rust support, [18]discussed in last year's Kernel Report , was intended to be experimental. Experiments can fail, and failure for the Rust module support would mean that it could be removed from the kernel again. It's already imposing a significant extra load, inasmuch as subsystem maintainers must be able to read code submissions in order to approve them or not, and that means reading Rust code, which Corbet described as sometimes resembling [19]line noise to C programmers.
However, substantial work is already going into Rust kernel modules. Notably, as [20]we described at the end of 2022 , the drivers that the Asahi Linux team are working on for Apple's new on-die GPUs are being built in Rust. Corbet noted that the decision point where an experiment is considered a success is when the kernel developers merge the first feature that users depend on. As such, that point for Rust in the kernel is coming very soon. If the Apple Silicon GPU support does get merged, that's a quite significant user-facing function. It means that if Rust support were to be removed again, anyone running Linux on Apple Silicon Macs would lose their graphics drivers. As such, the Rust support would thereby cease to be experimental.
The closing words were on the subject of the ongoing maintainer crisis. The kernel team is understaffed, and there are no members of the team devoted to documentation, for one critical example. He quoted the former maintainer of the XFS filesystem, Darrick Wong, who [21]stepped down in August:
Being a maintainer feels like a punishment, and that cannot stand. We need help.
Corbet continued:
Scott McNealy, for all his faults, actually said something useful a while back: "Open source is free like a puppy is free."
From this vulture's perspective, it seems like despite the corporate feel of the Open Source Summit these days, with big companies from around the world proudly talking about their use of open source and their large-scale adoption of Linux, the core project behind it all, the kernel itself, is under-resourced and under-funded. If there are around a couple of thousand developers working on any given release of the kernel, and about 10 percent of those are newbies for each point release, that implies that as many again are leaving the project each release, burning out and quitting, or perhaps simply being [22]redeployed to other areas by their employers.
As in so many things, although there is lots of good news, these are worrying times. ®
[23]Youtube Video
Get our [24]Tech Resources
[1] https://lwn.net/
[2] https://osseu2023.sched.com/event/1OGde/the-kernel-report-jonathan-corbet-lwnnet
[3] https://www.theregister.com/2020/12/14/linux_5_10_released/
[4] https://www.theregister.com/2021/01/28/long_term_support_for_linux_510/
[5] https://www.theregister.com/2023/06/23/red_hat_centos_move/
[6] https://osseu2023.sched.com/event/1OGde/the-kernel-report-jonathan-corbet-lwnnet
[7] https://lwn.net/talks/2023/kr-osseu.pdf
[8] https://www.theregister.com/2017/11/12/linux_4_14_released/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2023/06/30/enterprise_distro_feature_devconf/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2023/09/20/gnu_turns_40/
[15] https://www.theregister.com/2022/12/15/gcc_13_rust_support_merged/
[16] https://www.theregister.com/2022/10/05/rust_kernel_pull_request_pulled/
[17] https://www.theregister.com/2022/09/30/citrix_xen/
[18] https://www.theregister.com/2022/09/16/rust_in_the_linux_kernel/
[19] http://catb.org/jargon/html/L/line-noise.html
[20] https://www.theregister.com/2022/12/06/arm64_laptops/
[21] https://www.phoronix.com/news/XFS-Maintainer-Steps-Down
[22] https://www.theregister.com/2023/08/16/red_hat_redeploys_desktop_dev/
[23] https://www.youtube.com/watch?v=ggx9ZssPHU8
[24] https://whitepapers.theregister.com/
Referring back to the list of stable kernels, which goes back about six years, [1]Linux Weekly News editor Jonathan Corbet [2]said :
They have come to the conclusion that there's really no point in maintaining them that long, because people are not using them. So the six-year update policy is going away. When 4.14 goes out of support… early next year… there will not be another six-year kernel to replace it. We're likely to go back to a long-term stable kernel maintained for about two years. After that, people will simply be expected to update to a newer kernel.
The new regime
When 4.19 reaches its end of life, so will 5.4, leaving the [3]end-of-2020 kernel 5.10 the oldest longterm release. This change [4]was discussed at the time , but now it looks like what is essentially the worst-case outcome has been chosen.
Since the big enterprise players don't use the existing long-term releases, maintaining their own instead, and as is now clear in the case of the 800-pound gorilla of the Linux world, keeping that kernel and its source code [5]reserved for customers only , the kernel developers are dropping the very considerable maintenance burden of keeping those old releases alive.
Bring out the popcorn, it's the Kernel Report
For techie types tired of PowerPoint presentations and words like "synergy" and "leverage," Corbet's [6]Kernel Report is one of the highlights of what used to be called LinuxCon.
This year's [7]presentation [PDF] led in gently with some statistics, and held the big reveal about shortening support cycles back until later in the talk. There have been five releases in version 6 so far, and they are all of comparable size, with some 15,000 commits in each. Each point release has seen some 2,000 individual developers contributing, of whom around 250 to 300 are newcomers. As Corbet put it:
It's really the same old chart that I put up every time I do this talk. It doesn't change a whole lot… but it is worth pointing out that 6.2, released back in February, set a new record for the number of developers participating.
He noted that those are "mainline" releases, the ones Linus Torvalds himself puts out before moving on to the next new version. But, he said, "these are not the kernels that most of us are actually running. Most people are smarter than that and try to run something else; usually, something based off the stable kernel updates."
At present, there are six releases under long-term maintenance: versions 6.1, 5.15, 5.10, 5.4, 4.19, and 4.14 from [8]back in November 2017 . The number of commits into each of these long-term kernels increases for each older version, with the oldest having over 300 updates and by now about 28,000 commits.
So which should you choose? Corbet – himself a kernel maintainer – says that one answer to this is simple:
Run the latest stable update. That is the most stable, the most secure, the best kernel we know how to create at this time. That's the very best we can do. You should run that.
Manager of the stable kernels Greg Kroah-Hartman's explanation is:
You have to take all of the stable/LTS releases in order to have a secure and stable system. If you attempt to cherry-pick random patches you will not fix all of the known, and unknown, problems, but rather you will end up with a potentially more insecure system, and one that contains known bugs.
Corbet went on to point out that the oldest long-term kernel is now approaching two full development cycles' worth of commits: "An awful lot of development is going on after our stable kernel is released." Many of the fixes are for bugs that appeared in earlier kernel releases – his analysis goes back to the beginning of the history in Git, back with kernel 2.6.12 in 2005.
There's an awful lot of bugs that lurk in the kernel for a long time before turning up somewhere and needing to be fixed. All this stuff gets scooped up and put into the stable kernel releases.
And this goes both ways: bugs discovered and fixed in later kernel versions must be back-ported and fixed in older, stable releases.
This, he pointed out, is the other way to choose which kernel to run: pick an old, stable kernel, and backport all the fixes that you consider important to it. It's the "enterprise" kernel model, and leads to old kernels with thousands and thousands of fixes applied to them. "We've seen an awful lot of fuss recently about what certain enterprise vendors are doing with regard to access to their code and their distributions, and the fuss around Red Hat Enterprise Linux in general. This here is what that fuss is about," he said.
[9]
It's a lot of work, and although it applies across the whole distribution, a huge amount is at the kernel level. What you're left with are old kernels, but with tens of thousands of fixes, leaving them very unlike what was originally released. The result, Corbet said, is that it cuts their users off from community support.
[10]
[11]
In a way, this is the flip side of Red Hat's view, which [12]we described back in June . Red Hat considers that the levels of testing and quality control in its enterprise kernels exceed that of the community ones. As a result, it maintains its own stable versions, and it doesn't use the upstream long-term releases.
That's when Corbet dropped his LTS bombshell.
[13]
There was a lot more in the talk, and we recommend watching it in full if you want to know where things are going. He discussed the new extended Berkeley Packet Filters (eBPF) subsystem, where it was going, and importantly, where it isn't – such as vendor-specific kernel schedulers. There was also discussion of security, confidential computing, and more.
[14]GNU turns 40: Stallman's baby still not ready for prime time, but hey, there's cake
[15]If GNU please: Rust support merged for the forthcoming GCC 13
[16]Linux 6.1: Rust to hit mainline kernel
[17]How Citrix dropped the ball on Xen ... according to Citrix
The much-vaunted new Rust support, [18]discussed in last year's Kernel Report , was intended to be experimental. Experiments can fail, and failure for the Rust module support would mean that it could be removed from the kernel again. It's already imposing a significant extra load, inasmuch as subsystem maintainers must be able to read code submissions in order to approve them or not, and that means reading Rust code, which Corbet described as sometimes resembling [19]line noise to C programmers.
However, substantial work is already going into Rust kernel modules. Notably, as [20]we described at the end of 2022 , the drivers that the Asahi Linux team are working on for Apple's new on-die GPUs are being built in Rust. Corbet noted that the decision point where an experiment is considered a success is when the kernel developers merge the first feature that users depend on. As such, that point for Rust in the kernel is coming very soon. If the Apple Silicon GPU support does get merged, that's a quite significant user-facing function. It means that if Rust support were to be removed again, anyone running Linux on Apple Silicon Macs would lose their graphics drivers. As such, the Rust support would thereby cease to be experimental.
The closing words were on the subject of the ongoing maintainer crisis. The kernel team is understaffed, and there are no members of the team devoted to documentation, for one critical example. He quoted the former maintainer of the XFS filesystem, Darrick Wong, who [21]stepped down in August:
Being a maintainer feels like a punishment, and that cannot stand. We need help.
Corbet continued:
Scott McNealy, for all his faults, actually said something useful a while back: "Open source is free like a puppy is free."
From this vulture's perspective, it seems like despite the corporate feel of the Open Source Summit these days, with big companies from around the world proudly talking about their use of open source and their large-scale adoption of Linux, the core project behind it all, the kernel itself, is under-resourced and under-funded. If there are around a couple of thousand developers working on any given release of the kernel, and about 10 percent of those are newbies for each point release, that implies that as many again are leaving the project each release, burning out and quitting, or perhaps simply being [22]redeployed to other areas by their employers.
As in so many things, although there is lots of good news, these are worrying times. ®
[23]Youtube Video
Get our [24]Tech Resources
[1] https://lwn.net/
[2] https://osseu2023.sched.com/event/1OGde/the-kernel-report-jonathan-corbet-lwnnet
[3] https://www.theregister.com/2020/12/14/linux_5_10_released/
[4] https://www.theregister.com/2021/01/28/long_term_support_for_linux_510/
[5] https://www.theregister.com/2023/06/23/red_hat_centos_move/
[6] https://osseu2023.sched.com/event/1OGde/the-kernel-report-jonathan-corbet-lwnnet
[7] https://lwn.net/talks/2023/kr-osseu.pdf
[8] https://www.theregister.com/2017/11/12/linux_4_14_released/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2023/06/30/enterprise_distro_feature_devconf/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZRMAJjkqHifiLjetq7HZAgAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2023/09/20/gnu_turns_40/
[15] https://www.theregister.com/2022/12/15/gcc_13_rust_support_merged/
[16] https://www.theregister.com/2022/10/05/rust_kernel_pull_request_pulled/
[17] https://www.theregister.com/2022/09/30/citrix_xen/
[18] https://www.theregister.com/2022/09/16/rust_in_the_linux_kernel/
[19] http://catb.org/jargon/html/L/line-noise.html
[20] https://www.theregister.com/2022/12/06/arm64_laptops/
[21] https://www.phoronix.com/news/XFS-Maintainer-Steps-Down
[22] https://www.theregister.com/2023/08/16/red_hat_redeploys_desktop_dev/
[23] https://www.youtube.com/watch?v=ggx9ZssPHU8
[24] https://whitepapers.theregister.com/
Doctor Syntax
It's about the age of a lot of Debian by the time it's released!
Currently the two previous versions are still maintained as Old Stable and the two previous to that have LTS which I believe is commercial rather than community. The oldest was released in 2015. Current Stable was released in June this year running kernel 6.1.
Like the meme
aerogems
You've probably seen it. It depicts modern software development as being sort of like a jenga tower, and there's one tiny little pillar at the bottom with the note "Project some random person from Nebraska has been thanklessly maintaining since 2003"
https://www.explainxkcd.com/wiki/images/d/d7/dependency.png
Stable not in the stable
adam 40
I'd like to see the old binaries hang around for longer, and not get deleted.
I'm very much an "if it ain't broke don't fix it type", so when I install a linux, I get a bunch of packages to go with it, and turn off updates, unless there is instability straight away.
However, sometime later I might want to install a program I forgot earlier. It would be better if the packages were still around so I could pick up these binaries. After all, they don't need "maintaining".
This also applied to embedded linux projects, where we fix on a certain version and don't update in the field.
If the kernel is dead in 2 years that's barely enough time for all the apps to have been built to go with it (if there are install dependencies, which there usually are.)
mark l 2
I still suspect that your Ubuntu, RHEL, SUSE etc will still carry on doing their own LTS kernel builds anyway so for your LTS servers and desktops distros nothing is going to change regarding their kernel support life cycles.
Six years is rather long but two years is rather short.