OpenZFS 2.3 is here, with RAID expansion and faster dedup
- Reference: 1737630328
- News link: https://www.theregister.co.uk/2025/01/23/openzfs_23_raid_expansion/
- Source link:
[1]OpenZFS release 2.3.0 is out, and will be in Linux distros that include ZFS, such as Ubuntu, Proxmox, NixOS and Void Linux – and eventually in FreeBSD, too. This release can be built for FreeBSD from 13.3 up to 14.2, and it is compatible with Linux kernel versions up to the [2]latest LTS version 6.12 .
The 2.3.0 version lets users expand an existing array by adding additional drives, and ZFS's built-in deduplication feature is now much faster. Advanced applications which do their own caching, such as some databases, can now bypass the ZFS Advanced Read Cache (ARC) with the new [3]Direct IO support .
[4]
The RAID-expansion feature has taken a few years to arrive: we [5]talked about it way back in February 2022. However, the OpenZFS team does have a good reason for moving slowly and cautiously: the last new-feature release of OpenZFS was [6]version 2.2.0 in October 2023, which we [7]looked at shortly beforehand . Unfortunately, that release was closely followed by the [8]discovery of a data-corrupting bug . This had been lurking unseen for years, and was followed by [9]fixed versions, both 2.2.2 and 2.1.14 .
[10]
[11]
Since the core selling points of ZFS are its sophistication and its reputation for keeping your data safe, this bug was a serious blow to the project's good name – and confidence. It's therefore understandable that it has moved forwards very slowly and carefully since. We feel it is worth pointing out that the developers haven't been idle: there have been several more releases in both the 2.1.x and 2.2.x series since then, and as of December 2024, they were up to versions [12]2.1.16 and [13]2.2.7 respectively.
So, you shouldn't be surprised by the dates on the project pages that describe the headline new features: [14]RAIDZ expansion and [15]Fast Dedup . RAIDZ is the ZFS version of RAID: it lets you amalgamate multiple disks into one larger volume, with redundancy.
[16]
The expansion feature lets you add another drive to an existing array, and then enlarge the array to add the extra space. The data is read from the existing drives and then rewritten across all of them including the new ones, so the expansion is not instant by any means – it could take days, and the new space only becomes available when it's complete. However, the array can be used while this is in progress, and it remains fault-tolerant throughout. If it's interrupted, for instance by a reboot, it resumes where it left off, and it can handle if one of the drives fails during the expansion. Expansion can be repeated multiple times.
The main thing that it doesn't support is changing [17]RAIDZ levels . RAIDZ1 is the ZFS flavor of RAID5, meaning it can tolerate the failure of a single drive; RAIDZ2 has dual parity, like [18]RAID 6 , with two redundant drives; RAIDZ3 has triple parity. ZFS has let you provision [19]hot spares for many years, but even with RAID expansion, while you can make a RAIDZ1 bigger, you can't turn a RAIDZ1 into a RAIDZ2: you can't add additional parity drives.
Another desirable fresh feature is Fast Dedup. ZFS deduplication isn't new: ZFS has had that for many years – Oracle's [20]documentation page is dated 2010. However, it was relatively slow and made the server work hard. The Reg FOSS desk has two elderly HP Microservers running TrueNAS, each with four-drive RAIDZ1 arrays, and as they're low-end boxes with a mere 8 GB of RAM in each, wiser heads counselled us not to enable dedup; we were told it would make them bog down very badly, possibly for an indefinite period. The new Fast Dedup feature shouldn't impose such a load.
[21]
The company behind TrueNAS, iXsystems, developed Fast Dedup and [22]last February donated it to the OpenZFS project. It's been available since March 2024 in the nightly testing releases of TrueNAS SCALE, which is iXsystems's next-generation NAS OS, based on Debian Linux.
[23]FreeBSD 14.2 wants to woo Docker fans, but still struggles with Wi-Fi
[24]Linux updates with an undo function? Some distros have that
[25]Nix forked, but over politics instead of progress
[26]Ubuntu 24.04, Fedora 40, EndeavourOS, and TrueNAS 24.04 all arrive at once
For those who prefer to stick to stable versions of their storage software, OpenZFS 2.3 will be [27]included in TrueNAS Core 25.04 , which is due in April. Don't expect these features to come to the older FreeBSD-based TrueNAS Core product, though. As we [28]wrote in March 2024 , TrueNAS Core 13 is the final FreeBSD version of TrueNAS. The following month, iXsystems told Register sister site Blocks and Files that [29]no users were being "marooned" by the move. This month, iXsystems announced that TrueNAS SCALE 25.04, codenamed "Fangtooth," is designed to [30]unify the CORE and SCALE lines under a common TrueNAS Community Edition (CE).
The main [31]TrueNAS Scale product page extols its ability to run "Apps, Linux Containers, & VMs." This may not commend it to those who just want FreeBSD reliability for their storage boxes and don't care about running Linux Docker containers. (For that matter, as of version 14.2, FreeBSD includes support for [32]running OCI-compatible container images .) There's now a preliminary website for the community-led fork of the FreeBSD version of TrueNAS, although there isn't really anything to see there yet – but it's called [33]zVault . ®
Get our [34]Tech Resources
[1] https://github.com/openzfs/zfs/releases/tag/zfs-2.3.0
[2] https://www.theregister.com/2024/12/11/linux_612_lts/
[3] https://github.com/openzfs/zfs/pull/10018
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z5J2AlPLBgOPLAjC-o4wCAAAAE0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://www.theregister.com/2022/02/11/raid_expansion_openzfs/
[6] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0
[7] https://www.theregister.com/2023/08/16/openzfs_zfsbootmenu_2_2/
[8] https://www.theregister.com/2023/11/27/openzfs_2_2_0_data_corruption/
[9] https://www.theregister.com/2023/12/04/two_new_versions_of_openzfs/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z5J2AlPLBgOPLAjC-o4wCAAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z5J2AlPLBgOPLAjC-o4wCAAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://github.com/openzfs/zfs/releases/tag/zfs-2.1.16
[13] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.7
[14] https://github.com/openzfs/zfs/pull/15022
[15] https://github.com/openzfs/zfs/discussions/15896
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z5J2AlPLBgOPLAjC-o4wCAAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[17] https://www.raidz-calculator.com/raidz-types-reference.aspx
[18] https://blocksandfiles.com/2022/05/05/raid/
[19] https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gcvcw/index.html
[20] https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gjhav/index.html
[21] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z5J2AlPLBgOPLAjC-o4wCAAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[22] https://www.ixsystems.com/blog/fast-dedup-is-a-valentines-gift-to-the-openzfs-and-truenas-communities/
[23] https://www.theregister.com/2024/12/05/freebsd_142/
[24] https://www.theregister.com/2024/08/01/linux_rollback_options/
[25] https://www.theregister.com/2024/05/14/nix_forked_but_over_politics/
[26] https://www.theregister.com/2024/04/29/ubuntu_2404_fed_40_et_al/
[27] https://www.truenas.com/blog/fangtooth-openzfs-23/
[28] https://www.theregister.com/2024/03/18/truenas_abandons_freebsd/
[29] https://blocksandfiles.com/2024/04/08/ixsystems-no-one-is-getting-marooned/
[30] https://www.truenas.com/blog/truenas-fangtooth-25-04/
[31] https://www.truenas.com/truenas-scale/
[32] https://www.theregister.com/2024/12/05/freebsd_142/
[33] https://www.zvault.io/
[34] https://whitepapers.theregister.com/
Re: Give ZFS another try
> move ZFS datasets between pools of physical disks while keeping the mountpoint available and online
Wow. TBH I have never even contemplated this might be possible, let alone tried to do it.
Re: Give ZFS another try
> and a pool destroy/re-create, which necessitates a short downtime to flip the replacement dataset to the desired mountpoint.
why do you need to do a destroy/re-create?
By default the mountpoint for a filesystem is inherited from the parent pool/filesystem (e.g. poolname tank, filesystems will by default inherit that and be mounted at /tank/
So if you send from tank/mydata mounted at default /tank/mydata to new_tank (so be default mounted at /new_tank/mydata), you can change the mountpoinf of /new_tank/mydata to /tank/mydata:
zfs set mountpoint=/tank/mydata new_tank/mydata
The mountpoint doesn't need to be the same as the pool or filesystem name:
zfs set mountpoint=/data/fred/bob new_tank/mydata
Of course, you'll need to either take the old filesystem ofline first,
zfs unmount tank/data
Or unmount the new filesystem, specify the new mountpoint, then unmount the old and mount the new:
zfs unmount new_tank/mydata
zfs set mountpoint=/tank/mydata new_tank/mydata
zfs unmount tank/mydata
zfs mount new_tank/mydata #which will now be mounted at /tank/mydata
Note however, if you already have manually set the mountpoint on a filesystem, doing a zfs send/receive of all properties will also apply that mountpoint name to the new copy (the target of the receive), but in that case if it's on the same host you probably weant to use a 'zfs receive -u' to prevent the destination filesystem from being mounted pre-maturely if the source filesystem is still mounted.
"HP Microservers running TrueNAS"
The old Gen 7? There was a point where they were going for £190 new (with a 250Gb drive and 2GB RAM) with a £100 cashback offer. Lost track of how many of those became very well prices NAS boxes for customers. You can still pretty much trade them on eBay for as much as they cost new.
I'm running a pair with an add in LSI SAS card and a 4x 2.5" HDD adapter in the 5.25" optical bay. Bumping into 8GB of RAM being a little limiting now and if I use ZSTD compression rather than LZ4 it can bottleneck on the CPU, but they still perform 'well enough'.
What I can tell you is that the PSUs really like to go pop and make burning smells! I'm down to my last couple of spares now and I think that'll be what finally kicks me off the platform. I'll take that as a sign I have to suck it up and try the Linux based Scale over the FreeBSD Core TrueNAS offerings. Then grumble about how they used to be fine with 2GB of RAM doing exactly the same job before shouting 'get off my lawn!' at some kids.
Re: "HP Microservers running TrueNAS"
> The old Gen 7?
In daily use with TrueNAS Core, one N54L and one Gen 8.
There's an older N40L with a trifling 0.9TB array made of 4 old 300GB drives that only has 6GB RAM, and its fan controller is shot so it's noisy. It currently runs OMV but I'm thinking of trying Proxmox on it as an experiment. Not much room for VMs in so little but it wouldn't be for production use.
Still no
The lack of raid expansion is the one thing that keeps me away from ZFS. However, what they've added here is half arsed and inflexible, and doesn't meet my needs. I'm still waiting for BcacheFS with erasure coding, which would be perfect for my needs, but who knows when that will be ready * . BTRFS promised to do what I want, but has been a big let-down thanks to being badly designed (raid56 breaks copy-on-write, looks unfixable without adding journaling to btrfs). It seems the only way to get flexible erasure coding with self healing bitrot detection is mdadm on top of dm-integrity, which is dog-shit slow, and comes with a big write-amplification penalty on ssd's. May just as well have several independant mdadm arrays with btrfs single disk fs on top, and manually fix any read errors from another good copy.
* My understanding is that BcacheFS existed and worked before the current effort to integrate it into the Linux kernel, so I don't understand why it seems to be being re-written from scratch
Re: Still no
> I don't understand why it seems to be being re-written from scratch
I don't think it is at all.
I think it being in the kernel means it's now exposed to millions more people than are prepared to build custom kernels, and it is also subjecting it to far more bug scrutiny.
TBH I think there is some truth in the argument that it _wasn't_ truthfully ready for it, but OTOH, without the pressure of being in the main tree, maybe it never would have been -- and now it's having to mature fast and suffering growing pains.
There _is_ a long-term plan to rewrite it in Rust, and that will be interesting if it ever happens.
Re: Still no
I switched to ZFS years ago, so it's been a while since I used MD - but I recall you could add disks, and also upsize an array if you replaced (eg) all your 1TB with 2TB. I don't recall being able to change things like RAID parity, but it's a fairly niche requirement.
ZFS can certainly upsize arrays, and now that it can add a new disk to a volume I'm curious what you need to do that MD can manage and that ZFS can't? I've been pretty satisfied with it overall, the main issues I've had are poor support from GRUB, and the fact it preallocates memory in an odd way which means it plays badly with other processes that preallocate (ilike Java). Also a niche requirement, to be fair.
Kitchen-Sink and Reliability
The new storage-related features seem great.
Running apps, containers, and VMs on your storage boxes? Why?
That-radical of an expansion of the codebase screams, "bug-breeding grounds!" Isn't reliability the #1 priority of a storage subsystem?
Re: Kitchen-Sink and Reliability
> Running apps, containers, and VMs on your storage boxes? Why?
Because the big k8s users will pay $LOTS for it, and that is driving sales.
iXsystems said FreeBSD wasn't developing fast enough. I do not know about the big-systems side of things, but it certainly is true that FreeBSD lags behing in areas like Wifi chipsets and graphics card support.
Perhaps this is also true in other cutting-edge hardware area, like very high-speed networking and storage drivers. That's more in its core territory but it is not beyond the bounds of the believable. Linux certainly is stronger there.
Interesting.. I've always avoided dedup as the standard advice is don't bother as it's a slow memory hog, but might be worth a look.
raidz expansion isn't a thing I've ever needed in practice (generally I just swap in larger drives over time instead.. esp. with nvme storage being limited by lanes).but should boost takeup.
Give ZFS another try
I have used ZFS for home storage briefly over the years and always found it slightly restrictive and ended up going back to mdraid+LVM. For basic filesystem I was very pleased but the physical volume management felt clunky. This zRAID expansion sounds like it is worth me taking ZFS for a spin again though.
My main confusion point with ZFS was that I never figured out a sequence (perhaps I just didn't RTFM enough) to move ZFS datasets between pools of physical disks while keeping the mountpoint available and online. Every process of migration seems to need a manual copy of data (or snapshot) and a pool destroy/re-create, which necessitates a short downtime to flip the replacement dataset to the desired mountpoint. It's even more complicated if you are limited on chassis bays/ports/drives which might require an intermediate copy to a temporary larger-but-lower-redundancy disk pool. With LVM the killer feature is pvmove (I have done SAN migrations using pvmove between LUNs or arrays and keeping everything mounted and running is superb). Is there an equivalent method for online backend disk/volume migration in ZFS that I have perhaps missed?