Kernel kerfuffle kiboshes Debian 12.3 release
(2023/12/12)
- Reference: 1702373412
- News link: https://www.theregister.co.uk/2023/12/12/kernel_6_1_ext4_corruption/
- Source link:
The Debian maintainers have identified a problem in kernel 6.1 that can cause corruption on ext4 volumes. As a result, the planned 12.3 release won't happen.
The issue is [1]Debian bug #1047843 , and it's been assigned a [2]severity level of "grave" – which is worse than "serious" and only one step down from "critical."
Unfortunately, the issue was identified while the rollout of Debian 12.3 to the various mirror servers was already under way. As a result, the rollout of 12.3 has been cancelled. It will be [3]replaced with version 12.4 instead.
[4]
What happened is a little complicated. At the time of writing, it seems that a [5]small, performance-enhancing change from November 1 was back-ported to the long-term support version of kernel 6.1. SUSE kernel developer Jan Kara isolated the problem, and [6]reported it to the kernel mailing list.
[7]
[8]
It turned out that the November 1 change required [9]another, different change from June 10, which was not back-ported with it. With the June 10 change, the November 1 one is fine, so this is not a bug as such – there's nothing wrong with it, unless the earlier change wasn't made.
The issue seems to affect only kernel versions 6.1.64 and 6.1.65 – it was fixed in kernel 6.1.66. However, 6.1.66 introduced a new, different issue, also caused by the same type of problem: a [10]minor fix was back-ported without the [11]earlier change it required. This change was [12]reverted on December 11, resulting in kernel 6.1.67, which at the time of writing is the current version of 6.1 and resolves both issues.
[13]
We hope that's clear.
These dependency problems remind us of the bad old days of Linux in the late 1990s. During that time, package-management tools such as Red Hat's RPM command were not smart enough to track when one package required another package and work out on their own what other packages were needed to make something work. Installing small programs – such as a text editor – might involve manually tracing all its dependencies, making notes on paper or in a text file, and installing all of them. But of course the dependencies might have dependencies of their own, making this a recursive process. You might have to download a dozen packages just to get one to install.
[14]Systemd 255 is here with improved UKI support
[15]Veteran editors Notepad++ and Geany hit milestone versions
[16]Two new versions of OpenZFS fix long-hidden corruption bug
[17]Steam client drops support on macOS, but adds it on Linux
The Reg FOSS desk still recalls the dread caused by upgrading KDE 1.x to KDE 2.x on an early version of Red Hat Linux, which required in excess of 200 packages to be manually identified, downloaded individually, and hand installed – in the right order , of course.
The first distro to fix this horrific process was Debian in 1999, when version 2.1 "Slink" came with the new Advanced Package Tool – or apt – which could do this for you automatically. Frankly, while Debian back then was still very intimidating, it gained a massive technical edge over Red Hat on the spot.
Red Hat eventually picked up the [18]yum package manager from the [19]PowerPC-specific Yellow Dog Linux , and regained some kind of parity. We think that it might have been included in Red Hat Linux 8 in 2002 – but if so, it wasn't mentioned in the [20]release announcement .
[21]
It took nearly a decade for automatic dependency resolution to come to Linux distributions, and until it did, this just seemed to be an insolubly complex problem. Perhaps it's time for someone to invent some sort of automatic dependency tracking and resolution mechanism for Linux kernel code changes. ®
Get our [22]Tech Resources
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843
[2] https://www.debian.org/Bugs/server-control
[3] https://www.debian.org/News/2023/2023121002
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://github.com/torvalds/linux/commit/91562895f8030cb9a0470b1db49de79346a69f91
[6] https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuhc3@quack3/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://github.com/torvalds/linux/commit/936e114a245b6e38e0dbf706a67e7611fc993da1
[10] https://github.com/torvalds/linux/commit/7e7efdda6adb385fbdfd6f819d76bc68c923c394
[11] https://github.com/torvalds/linux/commit/076fc8775dafe995e94c106bb732bf2d42dedcea
[12] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.67
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2023/12/08/systemd_255_is_here/
[15] https://www.theregister.com/2023/12/06/veteran_editors_notepad_and_geany/
[16] https://www.theregister.com/2023/12/04/two_new_versions_of_openzfs/
[17] https://www.theregister.com/2023/12/04/steam_mac_lin_changes/
[18] https://webhome.phy.duke.edu/~rgb/General/yum_HOWTO/yum_HOWTO/yum_HOWTO-1.html
[19] https://www.theregister.com/1999/04/01/yellow_dog_ships_eponymous_mac/
[20] https://www.redhat.com/en/about/press-releases/press-redhatlinux80
[21] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[22] https://whitepapers.theregister.com/
The issue is [1]Debian bug #1047843 , and it's been assigned a [2]severity level of "grave" – which is worse than "serious" and only one step down from "critical."
Unfortunately, the issue was identified while the rollout of Debian 12.3 to the various mirror servers was already under way. As a result, the rollout of 12.3 has been cancelled. It will be [3]replaced with version 12.4 instead.
[4]
What happened is a little complicated. At the time of writing, it seems that a [5]small, performance-enhancing change from November 1 was back-ported to the long-term support version of kernel 6.1. SUSE kernel developer Jan Kara isolated the problem, and [6]reported it to the kernel mailing list.
[7]
[8]
It turned out that the November 1 change required [9]another, different change from June 10, which was not back-ported with it. With the June 10 change, the November 1 one is fine, so this is not a bug as such – there's nothing wrong with it, unless the earlier change wasn't made.
The issue seems to affect only kernel versions 6.1.64 and 6.1.65 – it was fixed in kernel 6.1.66. However, 6.1.66 introduced a new, different issue, also caused by the same type of problem: a [10]minor fix was back-ported without the [11]earlier change it required. This change was [12]reverted on December 11, resulting in kernel 6.1.67, which at the time of writing is the current version of 6.1 and resolves both issues.
[13]
We hope that's clear.
These dependency problems remind us of the bad old days of Linux in the late 1990s. During that time, package-management tools such as Red Hat's RPM command were not smart enough to track when one package required another package and work out on their own what other packages were needed to make something work. Installing small programs – such as a text editor – might involve manually tracing all its dependencies, making notes on paper or in a text file, and installing all of them. But of course the dependencies might have dependencies of their own, making this a recursive process. You might have to download a dozen packages just to get one to install.
[14]Systemd 255 is here with improved UKI support
[15]Veteran editors Notepad++ and Geany hit milestone versions
[16]Two new versions of OpenZFS fix long-hidden corruption bug
[17]Steam client drops support on macOS, but adds it on Linux
The Reg FOSS desk still recalls the dread caused by upgrading KDE 1.x to KDE 2.x on an early version of Red Hat Linux, which required in excess of 200 packages to be manually identified, downloaded individually, and hand installed – in the right order , of course.
The first distro to fix this horrific process was Debian in 1999, when version 2.1 "Slink" came with the new Advanced Package Tool – or apt – which could do this for you automatically. Frankly, while Debian back then was still very intimidating, it gained a massive technical edge over Red Hat on the spot.
Red Hat eventually picked up the [18]yum package manager from the [19]PowerPC-specific Yellow Dog Linux , and regained some kind of parity. We think that it might have been included in Red Hat Linux 8 in 2002 – but if so, it wasn't mentioned in the [20]release announcement .
[21]
It took nearly a decade for automatic dependency resolution to come to Linux distributions, and until it did, this just seemed to be an insolubly complex problem. Perhaps it's time for someone to invent some sort of automatic dependency tracking and resolution mechanism for Linux kernel code changes. ®
Get our [22]Tech Resources
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843
[2] https://www.debian.org/Bugs/server-control
[3] https://www.debian.org/News/2023/2023121002
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://github.com/torvalds/linux/commit/91562895f8030cb9a0470b1db49de79346a69f91
[6] https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuhc3@quack3/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://github.com/torvalds/linux/commit/936e114a245b6e38e0dbf706a67e7611fc993da1
[10] https://github.com/torvalds/linux/commit/7e7efdda6adb385fbdfd6f819d76bc68c923c394
[11] https://github.com/torvalds/linux/commit/076fc8775dafe995e94c106bb732bf2d42dedcea
[12] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.67
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2023/12/08/systemd_255_is_here/
[15] https://www.theregister.com/2023/12/06/veteran_editors_notepad_and_geany/
[16] https://www.theregister.com/2023/12/04/two_new_versions_of_openzfs/
[17] https://www.theregister.com/2023/12/04/steam_mac_lin_changes/
[18] https://webhome.phy.duke.edu/~rgb/General/yum_HOWTO/yum_HOWTO/yum_HOWTO-1.html
[19] https://www.theregister.com/1999/04/01/yellow_dog_ships_eponymous_mac/
[20] https://www.redhat.com/en/about/press-releases/press-redhatlinux80
[21] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZXg9WNPFTgGmjQUGB07xNAAAAIk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[22] https://whitepapers.theregister.com/
Crypto Monad
Sigh. Another case of "Debian knows better than the upstreams". Like the "improvements" to ssh key generation which removed almost all entropy.
Please: just leave the software alone, and package it. In particular, give me a kernel that's as close to the one Linus released as possible.
Just escaped that one!
The problematic Debian kernel version was 6.1.0-14.landed here at the weekend. I installed that about 6 pm yesterday but didn't immediately reboot so continued running on 13. I'd seen a news item about an ext4 problem but it didn't register that this might contain it. Later in the evening I rebooted and maybe an hour afterwards an alert came up for a new update. I checked that & found it was for kernel 15. Realising that 2 kernel updates a day meant that there must have been a problem with the first so immediately installed and rebooted and then looked into the history finding the references in the article and this discussion https://lwn.net/Articles/954285/
This left me with the problem - did I do the big sorting out of email archives before or after the reboot? I think i was after but AFAICS, no damage done. Deep breath! This, BTW, was in Devuan but as far as non-systemd stuff is concerned, it's Debian
Debian 12.4 with the corrected kernel was out by the end of the day, BTW. See amacater's post towards the end of the LWN comments.
As far as I can make out from the discussion the later patch was - ironically - supposed to deal with the possibility of corruption in the event of a system crash under certain circumstances which is the sort of thing that's like;y to get a high priority for back-porting and the earlier patch that didn't get back-ported may have been the performance-related one.