How Windows NTFS finally made it into Linux
(2021/10/13)
- Reference: 1634116514
- News link: https://www.theregister.co.uk/2021/10/13/how_ntfs_finally_made_it/
- Source link:
Opinion Love it or hate it, Linux users in a Windows world must deal with Microsoft's New Technology File System (NTFS). This has always been a pain in the rump. Even after Microsoft finally gave up on its anti-Linux rhetoric and released its patents to the open-source community and expressively opened up its exFAT patents, we still couldn't get into NTFS.
Things have changed. Starting with the [1]Linux 5.15 kernel, NTFS is finally being fully supported in Linux . This journey hasn't been easy.
Microsoft introduced NTFS, a proprietary – naturally – journaling file system in Windows NT 3.1 in 1993. From there, it replaced 1977's [2]File Allocation Table (FAT) file system across Windows.
[3]
Now just because something is proprietary doesn't mean that Linux and open-source developers can't reverse-engineer it or finagle some way of using it. Just see what the [4]Samba crew has done over the years with Server Message Block (SMB) and the [5]Wine folks, who have got numerous Windows applications to run on the *nix operating system family. NTFS, however, proved to be much more difficult to port over.
[6]
[7]
Oh, there were ways to work with NTFS drives from Linux, but it wasn't pretty. There were two main earlier open-source Linux NTFS drivers: [8]NTFS-3G and the [9]Captive NTFS driver .
NTFS-3G, which works with the [10]Filesystem in Userspace (FUSE), is slow, really, really slow. On Linux, it also can only read from NTFS systems. On other *nix systems, like macOS, can [11]write as well, but it's, in a word, ugly . You can try to do things like resizing an NTFS partition, but as the [12]Arch Linux program notes spell out, " [13]Please ensure you have a backup before attempting this if your data is important." Hmmm… let's not and say we did.
[14]
The Captive NTFS driver could read and write to NTFS. To pull this trick off, however, it used the original Windows ntfs.sys driver. Using a proprietary driver in open-source software is always troublesome, especially back in those bad old days. In the event, the project didn't last for long. The code itself is still around, but the project itself is long dead.
While the open-source community was working on these projects, the company Paragon was taking a different approach. Its [15]Microsoft NTFS for Linux by Paragon Software used the proprietary Paragon File System Link, a cross-platform file system drivers to read and write from NTFS drives. The program name says it all. Paragon produced its software in partnership with Microsoft.
The result was a fast, efficient, and, yes, proprietary NTFS Linux system. But, while Paragon did well with this as a business for years, Microsoft was no longer the proprietary powerhouse it had been under Bill Gates and Steve Ballmer. Today's Microsoft has [16]realized it was on the wrong side of history with open source .
[17]
What was Paragon to do with a pro-open-source Microsoft? Well, at first, the company rapidly went into the first three stages of grief: Denial and isolation, anger, and bargaining. In May 2020, it published a piece entitled The [18]hidden cost of "free" exFAT , where you can see all three stages on display.
By September 2020, Paragon had moved on to the final stage: Acceptance. The company " [19]contributed the read-write NTFS kernel driver under the GPL to the Linux community, for hopeful inclusion in due time to the mainline kernel.
That was wonderful news… except for this one little thing. The [20]NTFS3 code was in no way, shape, or form ready to be added to the kernel. That's the thing with proprietary code – and why I'm cynical about the idea that Microsoft could simply open-source, say, all of Windows 7 – it's often badly written. Unlike open-source, where everyone can see your work, proprietary software can hide its sins from watchers.
First, the code, with 27,000 lines, was much too big. Linux kernel developer Nikolay Borisov wrote: "So, how exactly do you [21]expect someone to review this monstrosity ?"
Good question. The answer: You can't.
Besides, as Linus Torvalds said, it would be nice if [22]Paragon would, you know, "actually submit it."
Would it be too much to ask for "Paragon [to] just make a git pull request for it"?
[23]2001: Linux is cancer, says Microsoft. 2019: Hey friends, ah, can we join the official linux-distros mailing list, plz?
[24]Microsoft loves Linux – as in, it loves Linux users running Linux desktop apps on Windows PCs
[25]GitHub merges 'useless garbage' says Linus Torvalds as new NTFS support added to Linux kernel 5.15
[26]Open-source software starts with developers, but there are other important contributors, too. Who exactly? Good question
Paragon, to no surprise, didn't know how to deal with the Linux kernel e-mail and Git-driven workflow. Mea culpa, [27]Paragon agreed that it "will be maintaining this implementation ," but that "we'll need several days to prepare a proper pull request before sending it to you."
Torvalds was glad to hear this, but he admonished the Paragon developers: "The one other thing I do want when there are big new pieces like this being added is to ask you to make sure that [28]everything is signed-off properly , and that there is no internal confusion about the GPLv2 inside Paragon, and that any legal people, etc. are all aware of this all and are on board. The last thing we want to see is some 'Oops, we didn't mean to do this' brouhaha six months later."
Over a year later, they didn't have a GPL fight. Thank goodness for small miracles. But, after a long, messy process, NTFS3, and thus NTFS, is finally in the Linux 5.15 kernel release candidate. If it all goes well, you can expect to see this kernel arrive in time for Halloween. Feel free to make a [29]Microsoft Halloween documents joke if you like, but with Microsoft's tacit blessing, NTFS will finally be fully and efficiently within Linux.
Who would have dreamed that – even five years ago after [30]Microsoft claimed it loved Linux ? We live in interesting times. ®
Get our [31]Tech Resources
[1] https://www.theregister.com/2021/09/06/github_merges_useless_garbage_says/
[2] https://www.theregister.com/2006/01/11/microsoft_wins_patent_case/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2019/07/09/samba_sans_one_smb1/
[5] https://www.theregister.com/2021/06/10/wine601/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.tuxera.com/company/open-source/
[9] http://www.jankratochvil.net/project/captive/
[10] https://github.com/libfuse/libfuse
[11] https://github.com/osxfuse/osxfuse/wiki/NTFS-3G
[12] https://archlinux.org/
[13] https://wiki.archlinux.org/title/NTFS-3G
[14] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[15] https://www.paragon-software.com/us/home/ntfs-linux-professional/#
[16] https://www.theregister.com/2020/05/15/microsoft_brad_smith_open_source/
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://www.paragon-software.com/exfat-license/
[19] https://lore.kernel.org/lkml/20200918162204.3706029-1-almaz.alexandrovich@paragon-software.com/
[20] https://www.paragon-software.com/us/home/ntfs3-driver-faq/
[21] https://www.theregister.com/2020/08/18/paragon_tries_to_contribute_ntfs
[22] https://lore.kernel.org/lkml/CAHk-=whfeq9gyPWK3yao6cCj7LKeU3vQEDGJ3rKDdcaPNVMQzQ@mail.gmail.com/
[23] https://www.theregister.com/2019/06/27/microsoft_linux_distro_list/
[24] https://www.theregister.com/2021/04/22/windows_linux_gui/
[25] https://www.theregister.com/2021/09/06/github_merges_useless_garbage_says/
[26] https://www.theregister.com/2021/09/08/open_source_beyond_the_programmers/
[27] https://lore.kernel.org/lkml/CAHk-=whfeq9gyPWK3yao6cCj7LKeU3vQEDGJ3rKDdcaPNVMQzQ@mail.gmail.com/
[28] https://lore.kernel.org/lkml/CAHk-=wjn4W-7ZbHrw08cWy=12DgheFUKLO5YLgG6in5TA5HxqQ@mail.gmail.com/
[29] https://www.theregister.com/2001/11/12/ms_promotes_linux_from_threat/
[30] https://www.theregister.com/2016/02/17/we_love_linux_claims_microsoft/
[31] https://whitepapers.theregister.com/
Things have changed. Starting with the [1]Linux 5.15 kernel, NTFS is finally being fully supported in Linux . This journey hasn't been easy.
Microsoft introduced NTFS, a proprietary – naturally – journaling file system in Windows NT 3.1 in 1993. From there, it replaced 1977's [2]File Allocation Table (FAT) file system across Windows.
[3]
Now just because something is proprietary doesn't mean that Linux and open-source developers can't reverse-engineer it or finagle some way of using it. Just see what the [4]Samba crew has done over the years with Server Message Block (SMB) and the [5]Wine folks, who have got numerous Windows applications to run on the *nix operating system family. NTFS, however, proved to be much more difficult to port over.
[6]
[7]
Oh, there were ways to work with NTFS drives from Linux, but it wasn't pretty. There were two main earlier open-source Linux NTFS drivers: [8]NTFS-3G and the [9]Captive NTFS driver .
NTFS-3G, which works with the [10]Filesystem in Userspace (FUSE), is slow, really, really slow. On Linux, it also can only read from NTFS systems. On other *nix systems, like macOS, can [11]write as well, but it's, in a word, ugly . You can try to do things like resizing an NTFS partition, but as the [12]Arch Linux program notes spell out, " [13]Please ensure you have a backup before attempting this if your data is important." Hmmm… let's not and say we did.
[14]
The Captive NTFS driver could read and write to NTFS. To pull this trick off, however, it used the original Windows ntfs.sys driver. Using a proprietary driver in open-source software is always troublesome, especially back in those bad old days. In the event, the project didn't last for long. The code itself is still around, but the project itself is long dead.
While the open-source community was working on these projects, the company Paragon was taking a different approach. Its [15]Microsoft NTFS for Linux by Paragon Software used the proprietary Paragon File System Link, a cross-platform file system drivers to read and write from NTFS drives. The program name says it all. Paragon produced its software in partnership with Microsoft.
The result was a fast, efficient, and, yes, proprietary NTFS Linux system. But, while Paragon did well with this as a business for years, Microsoft was no longer the proprietary powerhouse it had been under Bill Gates and Steve Ballmer. Today's Microsoft has [16]realized it was on the wrong side of history with open source .
[17]
What was Paragon to do with a pro-open-source Microsoft? Well, at first, the company rapidly went into the first three stages of grief: Denial and isolation, anger, and bargaining. In May 2020, it published a piece entitled The [18]hidden cost of "free" exFAT , where you can see all three stages on display.
By September 2020, Paragon had moved on to the final stage: Acceptance. The company " [19]contributed the read-write NTFS kernel driver under the GPL to the Linux community, for hopeful inclusion in due time to the mainline kernel.
That was wonderful news… except for this one little thing. The [20]NTFS3 code was in no way, shape, or form ready to be added to the kernel. That's the thing with proprietary code – and why I'm cynical about the idea that Microsoft could simply open-source, say, all of Windows 7 – it's often badly written. Unlike open-source, where everyone can see your work, proprietary software can hide its sins from watchers.
First, the code, with 27,000 lines, was much too big. Linux kernel developer Nikolay Borisov wrote: "So, how exactly do you [21]expect someone to review this monstrosity ?"
Good question. The answer: You can't.
Besides, as Linus Torvalds said, it would be nice if [22]Paragon would, you know, "actually submit it."
Would it be too much to ask for "Paragon [to] just make a git pull request for it"?
[23]2001: Linux is cancer, says Microsoft. 2019: Hey friends, ah, can we join the official linux-distros mailing list, plz?
[24]Microsoft loves Linux – as in, it loves Linux users running Linux desktop apps on Windows PCs
[25]GitHub merges 'useless garbage' says Linus Torvalds as new NTFS support added to Linux kernel 5.15
[26]Open-source software starts with developers, but there are other important contributors, too. Who exactly? Good question
Paragon, to no surprise, didn't know how to deal with the Linux kernel e-mail and Git-driven workflow. Mea culpa, [27]Paragon agreed that it "will be maintaining this implementation ," but that "we'll need several days to prepare a proper pull request before sending it to you."
Torvalds was glad to hear this, but he admonished the Paragon developers: "The one other thing I do want when there are big new pieces like this being added is to ask you to make sure that [28]everything is signed-off properly , and that there is no internal confusion about the GPLv2 inside Paragon, and that any legal people, etc. are all aware of this all and are on board. The last thing we want to see is some 'Oops, we didn't mean to do this' brouhaha six months later."
Over a year later, they didn't have a GPL fight. Thank goodness for small miracles. But, after a long, messy process, NTFS3, and thus NTFS, is finally in the Linux 5.15 kernel release candidate. If it all goes well, you can expect to see this kernel arrive in time for Halloween. Feel free to make a [29]Microsoft Halloween documents joke if you like, but with Microsoft's tacit blessing, NTFS will finally be fully and efficiently within Linux.
Who would have dreamed that – even five years ago after [30]Microsoft claimed it loved Linux ? We live in interesting times. ®
Get our [31]Tech Resources
[1] https://www.theregister.com/2021/09/06/github_merges_useless_garbage_says/
[2] https://www.theregister.com/2006/01/11/microsoft_wins_patent_case/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2019/07/09/samba_sans_one_smb1/
[5] https://www.theregister.com/2021/06/10/wine601/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.tuxera.com/company/open-source/
[9] http://www.jankratochvil.net/project/captive/
[10] https://github.com/libfuse/libfuse
[11] https://github.com/osxfuse/osxfuse/wiki/NTFS-3G
[12] https://archlinux.org/
[13] https://wiki.archlinux.org/title/NTFS-3G
[14] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[15] https://www.paragon-software.com/us/home/ntfs-linux-professional/#
[16] https://www.theregister.com/2020/05/15/microsoft_brad_smith_open_source/
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWauWTzdiVHh4vbMGGgS@wAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://www.paragon-software.com/exfat-license/
[19] https://lore.kernel.org/lkml/20200918162204.3706029-1-almaz.alexandrovich@paragon-software.com/
[20] https://www.paragon-software.com/us/home/ntfs3-driver-faq/
[21] https://www.theregister.com/2020/08/18/paragon_tries_to_contribute_ntfs
[22] https://lore.kernel.org/lkml/CAHk-=whfeq9gyPWK3yao6cCj7LKeU3vQEDGJ3rKDdcaPNVMQzQ@mail.gmail.com/
[23] https://www.theregister.com/2019/06/27/microsoft_linux_distro_list/
[24] https://www.theregister.com/2021/04/22/windows_linux_gui/
[25] https://www.theregister.com/2021/09/06/github_merges_useless_garbage_says/
[26] https://www.theregister.com/2021/09/08/open_source_beyond_the_programmers/
[27] https://lore.kernel.org/lkml/CAHk-=whfeq9gyPWK3yao6cCj7LKeU3vQEDGJ3rKDdcaPNVMQzQ@mail.gmail.com/
[28] https://lore.kernel.org/lkml/CAHk-=wjn4W-7ZbHrw08cWy=12DgheFUKLO5YLgG6in5TA5HxqQ@mail.gmail.com/
[29] https://www.theregister.com/2001/11/12/ms_promotes_linux_from_threat/
[30] https://www.theregister.com/2016/02/17/we_love_linux_claims_microsoft/
[31] https://whitepapers.theregister.com/
Peter Christy
"NTFS-3G, which works with the Filesystem in Userspace (FUSE), is slow, really, really slow."
Seems fast enough here!
"On Linux, it also can only read from NTFS systems."
Strange, because my Linux system (Slackware64) seems unaware of this limitation, and writes quite happily to NTFS drives.
I think you are confusing the FUSE implementation with the old, native kernel NTFS driver, which was extremely limited as well as being very slow.
NTFS-3G might not be fast by industrial standards, but its fine for domestic and small office use.
Allonymous Coward
Agree with both the comments above. Article could do with a fact check.
Am I missing something?
My box at home is Fedora 29 and Suse Leap15 and both Linuxes can write to NTFS disks just fine.