Data-destroying defect found in OpenZFS 2.2.0
(2023/11/27)
- Reference: 1701091342
- News link: https://www.theregister.co.uk/2023/11/27/bug_openzfs_2_2_0/
- Source link:
A data-destroying bug has been discovered in a new feature of OpenZFS 2.2.0, as found in FreeBSD 14 among other OSes.
OpenZFS 2.2.0 was released just [1]last month with a new feature called [2]block cloning , as we reported when we [3]looked at release candidate 3 . Unfortunately, there's a bug in that code somewhere, as found by Gentoo-toting [4]Terin Stock , who [5]reported bug #15526 . As a result, [6]OpenZFS 2.2.1 is already out, which disables the new feature.
This is a bit of an embarrassment for OpenZFS, a project with an enviable reputation for data integrity. It's also less than ideal for fixed-release-cycle OSes that have the new version of OpenZFS, including the [7]newly released FreeBSD 14 . Fortunately for FreeBSD, though, version 14.0 ships with the feature [8]disabled by default .
[9]
We have [10]mentioned the [11]work of BSD boffin Colin Percival before, but anyone brave enough to have already installed this point-zero release should heed his [12]warning on
Twitter X: "FreeBSD 14's ZFS code supports 'block cloning'. This is turned off by default. DO NOT ENABLE THIS FEATURE UNLESS YOU WANT TO LOSE DATA."
[13]
[14]
The bug manifests as corruption of the contents of files when they're copied; instead of their expected contents, there are stretches of zeroes, mixed with blocks of what looks like Base64-encoded data. It showed up when using Gentoo's portage command, the distro's package-management tool – an operation that typically involves copying lots of data. Worse still is that the file system's own health checks don't detect any problem. For now, release 2.2.1 simply disables the feature.
At the time of writing, it's not certain exactly what causes it. It seems to be an extremely specific (and therefore unlikely) combination of circumstances, which means it almost never happens, as Bronek Kozicki [15]spells out on GitHub:
You need to understand the mechanism that causes corruption. It might have been there for decade and only caused issues in a very specific scenarios, which do not normally happen. Unless you can match your backup mechanism to the conditions described below, you are very unlikely to have been affected by it.
a file is being written to (typically it would be asynchronously – meaning the write is not completed at the time when writing process "thinks" it is)
at the same time when ZFS is still writing the data, the modified part of file is being read from. The same time means "hit a very specific time", measured in microseconds (that's millionth of a second), wide window. Admittedly, as a non-developer for ZFS project, I do not know if using HDD as opposed to SSD would extend that time frame.
if it is being read at this very specific moment, the reader will see zeros where the data being written is actually something else
if the reader then stores the incorrectly read zeroes somewhere else, that's where the data is being corrupted
One of the bug hunters has written a tiny script, [16]reproducer.sh , which hammers ZFS volumes and checks to see if files are getting corrupted. One of the problems around this issue is that there's no way to write a program that can report if a file has been corrupted or not by inspecting its contents: it's perfectly normal for some types of file to contain long stretches of zeroes. The only way to be sure is comparing checksums from before and after copy operations – so concerned users who lack backups held on other types of file system cannot readily tell. OpenZFS's built-in scrub tool for checking the validity of storage pools cannot detect the problem.
[17]FFmpeg 6.1 drops a Heaviside dose of codec magic
[18]Revival of Medley/Interlisp: Elegant weapon for a more civilized age sharpened up again
[19]Rocky Linux and Oracle Unbreakable Linux also hit 9.3
[20]Will anybody save Linux on Itanium? Absolutely not
A [21]possible fix is open, and the investigation looks like it may have uncovered an underlying, different, and pre-existing bug, which could have been present [22]as long ago as 2013 . The bug revolves around [23]ZFS dnodes , and the logic of how the code checks whether a dnode is "dirty" or not, which governs whether it must [24]flush it : sync any the changes to disk.
It's possible that this single cause was deeply hidden, and so very unlikely to be hit. Unfortunately, the new faster copy functionality meant that what used to be a bug that would only corrupt data once in tens of millions of file copies, suddenly became more likely, especially on machines with lots of processor cores all in simultaneous use.
[25]
For Linux users, an additional condition seems to be that the OS has a recent version of the [26]coreutils package – above version 9.x. This is the tool that provides the functionality of the cp command. So far, we have also not been able to verify if Ubuntu 23.10 has the block clone feature enabled by default in its [27]recently returned (but still experimental) support for being installed onto ZFS, but at least one comment to the original bug is by someone who has reproduced it on Ubuntu.
It seems very likely that OpenZFS 2.2.1, which simply turns off block-cloning, will quickly be followed by a 2.2.2 release to fix the underlying dnode handling. ®
Get our [28]Tech Resources
[1] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0
[2] https://github.com/openzfs/zfs/pull/13392
[3] https://www.theregister.com/2023/08/16/openzfs_zfsbootmenu_2_2/
[4] https://terinstock.com/
[5] https://github.com/openzfs/zfs/issues/15526
[6] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1
[7] https://www.theregister.com/2023/10/24/freebsd_14_rc2/
[8] https://vermaden.wordpress.com/2023/11/17/valuable-freebsd-14-0-release-updates/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[10] https://www.theregister.com/2022/10/19/freebsd_comes_to_amazons_lightweight/
[11] https://www.theregister.com/2023/08/29/freebsd_boots_in_25ms/
[12] https://twitter.com/cperciva/status/1727742802556293391
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[15] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826412289
[16] https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b22031a84#file-reproducer-sh
[17] https://www.theregister.com/2023/11/24/ffmpeg_6_1/
[18] https://www.theregister.com/2023/11/23/medley_interlisp_revival/
[19] https://www.theregister.com/2023/11/21/rocky_n_oracle_linux_9_3/
[20] https://www.theregister.com/2023/11/21/saving_linux_on_itanium/
[21] https://github.com/openzfs/zfs/pull/15571
[22] https://github.com/openzfs/zfs/issues/15526#issuecomment-1825181463
[23] https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSBroadDiskStructure
[24] https://openzfs.org/wiki/Documentation/DnodeSync
[25] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[26] https://www.gnu.org/software/coreutils/
[27] https://www.theregister.com/2023/10/14/ubuntu_2310_released/
[28] https://whitepapers.theregister.com/
OpenZFS 2.2.0 was released just [1]last month with a new feature called [2]block cloning , as we reported when we [3]looked at release candidate 3 . Unfortunately, there's a bug in that code somewhere, as found by Gentoo-toting [4]Terin Stock , who [5]reported bug #15526 . As a result, [6]OpenZFS 2.2.1 is already out, which disables the new feature.
This is a bit of an embarrassment for OpenZFS, a project with an enviable reputation for data integrity. It's also less than ideal for fixed-release-cycle OSes that have the new version of OpenZFS, including the [7]newly released FreeBSD 14 . Fortunately for FreeBSD, though, version 14.0 ships with the feature [8]disabled by default .
[9]
We have [10]mentioned the [11]work of BSD boffin Colin Percival before, but anyone brave enough to have already installed this point-zero release should heed his [12]warning on
Twitter X: "FreeBSD 14's ZFS code supports 'block cloning'. This is turned off by default. DO NOT ENABLE THIS FEATURE UNLESS YOU WANT TO LOSE DATA."
[13]
[14]
The bug manifests as corruption of the contents of files when they're copied; instead of their expected contents, there are stretches of zeroes, mixed with blocks of what looks like Base64-encoded data. It showed up when using Gentoo's portage command, the distro's package-management tool – an operation that typically involves copying lots of data. Worse still is that the file system's own health checks don't detect any problem. For now, release 2.2.1 simply disables the feature.
At the time of writing, it's not certain exactly what causes it. It seems to be an extremely specific (and therefore unlikely) combination of circumstances, which means it almost never happens, as Bronek Kozicki [15]spells out on GitHub:
You need to understand the mechanism that causes corruption. It might have been there for decade and only caused issues in a very specific scenarios, which do not normally happen. Unless you can match your backup mechanism to the conditions described below, you are very unlikely to have been affected by it.
a file is being written to (typically it would be asynchronously – meaning the write is not completed at the time when writing process "thinks" it is)
at the same time when ZFS is still writing the data, the modified part of file is being read from. The same time means "hit a very specific time", measured in microseconds (that's millionth of a second), wide window. Admittedly, as a non-developer for ZFS project, I do not know if using HDD as opposed to SSD would extend that time frame.
if it is being read at this very specific moment, the reader will see zeros where the data being written is actually something else
if the reader then stores the incorrectly read zeroes somewhere else, that's where the data is being corrupted
One of the bug hunters has written a tiny script, [16]reproducer.sh , which hammers ZFS volumes and checks to see if files are getting corrupted. One of the problems around this issue is that there's no way to write a program that can report if a file has been corrupted or not by inspecting its contents: it's perfectly normal for some types of file to contain long stretches of zeroes. The only way to be sure is comparing checksums from before and after copy operations – so concerned users who lack backups held on other types of file system cannot readily tell. OpenZFS's built-in scrub tool for checking the validity of storage pools cannot detect the problem.
[17]FFmpeg 6.1 drops a Heaviside dose of codec magic
[18]Revival of Medley/Interlisp: Elegant weapon for a more civilized age sharpened up again
[19]Rocky Linux and Oracle Unbreakable Linux also hit 9.3
[20]Will anybody save Linux on Itanium? Absolutely not
A [21]possible fix is open, and the investigation looks like it may have uncovered an underlying, different, and pre-existing bug, which could have been present [22]as long ago as 2013 . The bug revolves around [23]ZFS dnodes , and the logic of how the code checks whether a dnode is "dirty" or not, which governs whether it must [24]flush it : sync any the changes to disk.
It's possible that this single cause was deeply hidden, and so very unlikely to be hit. Unfortunately, the new faster copy functionality meant that what used to be a bug that would only corrupt data once in tens of millions of file copies, suddenly became more likely, especially on machines with lots of processor cores all in simultaneous use.
[25]
For Linux users, an additional condition seems to be that the OS has a recent version of the [26]coreutils package – above version 9.x. This is the tool that provides the functionality of the cp command. So far, we have also not been able to verify if Ubuntu 23.10 has the block clone feature enabled by default in its [27]recently returned (but still experimental) support for being installed onto ZFS, but at least one comment to the original bug is by someone who has reproduced it on Ubuntu.
It seems very likely that OpenZFS 2.2.1, which simply turns off block-cloning, will quickly be followed by a 2.2.2 release to fix the underlying dnode handling. ®
Get our [28]Tech Resources
[1] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0
[2] https://github.com/openzfs/zfs/pull/13392
[3] https://www.theregister.com/2023/08/16/openzfs_zfsbootmenu_2_2/
[4] https://terinstock.com/
[5] https://github.com/openzfs/zfs/issues/15526
[6] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1
[7] https://www.theregister.com/2023/10/24/freebsd_14_rc2/
[8] https://vermaden.wordpress.com/2023/11/17/valuable-freebsd-14-0-release-updates/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[10] https://www.theregister.com/2022/10/19/freebsd_comes_to_amazons_lightweight/
[11] https://www.theregister.com/2023/08/29/freebsd_boots_in_25ms/
[12] https://twitter.com/cperciva/status/1727742802556293391
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[15] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826412289
[16] https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b22031a84#file-reproducer-sh
[17] https://www.theregister.com/2023/11/24/ffmpeg_6_1/
[18] https://www.theregister.com/2023/11/23/medley_interlisp_revival/
[19] https://www.theregister.com/2023/11/21/rocky_n_oracle_linux_9_3/
[20] https://www.theregister.com/2023/11/21/saving_linux_on_itanium/
[21] https://github.com/openzfs/zfs/pull/15571
[22] https://github.com/openzfs/zfs/issues/15526#issuecomment-1825181463
[23] https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSBroadDiskStructure
[24] https://openzfs.org/wiki/Documentation/DnodeSync
[25] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWTLN1JR4tTuIkjx2iv15wAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[26] https://www.gnu.org/software/coreutils/
[27] https://www.theregister.com/2023/10/14/ubuntu_2310_released/
[28] https://whitepapers.theregister.com/
mattaw2001
I'm sticking to mint for that reason, and Debian on my servers, however had to go to the latest kernel and mesa for my laptop to work well.
zuckzuckgo
But now the definition of 'bleeding edge' is any change made since 2013.
ZFS here we go again
DaemonProcess
"ZFS is fast, more fully featured and totally safe"
" Oh, there's another teeny-weenie buggette that may corrupt some data but most likely not"
I'm still steering clear of this. From the description it sounds like multi-user / multi-process testing may be a bit lacking at the moment. Could all fanbois please volunteer.
The older I get, the less happy I feel about using a distro (or any software really) that is 'bleeding edge'. The last release is fine for me thanks, I'll let the rest of you check for bugs in the most recent version ;)