Explained: The thinking behind the 32GB Windows Format limit on FAT32
- Reference: 1609759930
- News link: https://www.theregister.co.uk/2021/01/04/windows_format_fat32/
- Source link:
While many welcomed 2021 within the walls of their own home, retired Microsoft engineer Dave Plummer marked the end of 2020 with the confession in the latest of a series of anecdotes hosted on his YouTube channel [1]Dave's Garage .
The limit has always seemed somewhat arbitrary, particularly when one considers the theoretical 16TB maximum volume size of the file system. Using a different formatting tool or dropping into the command line can handily override the presets, but sticking with the stock UI meant sticking with Plummer's Format dialog. And that meant 32GB for FAT32.
In the closing years of the last century, Plummer was involved in porting the Windows 95 shell to Windows NT. Part of that was a redo of Windows Format ("it had to be a replacement and complete rewrite since the Win95 system was so markedly different") and, as well as the grungy lower-level bits going down to the API, he also knocked together the classic, stacked Format dialog over the course of an hour of UI creativity.
As he admired his design genius, he pondered what cluster sizes to offer the potential army of future Windows NT 4.0 users. The options would define the size of the volume; FAT32 has a set maximum number of clusters in a volume. Making those clusters huge would make for an equally huge volume, but at a horrifying cost in terms of wasted space: select a 32-kilobyte cluster size and even the few bytes needed by a "Hello World" file would snaffle the full 32k.
"We call it 'Cluster Slack'," explained Plummer, "and it is the unavoidable waste of using FAT32 on large volumes."
"How large is too large? At what point do you say, 'No, it's too inefficient, it would be folly to let you do that'? That is the decision I was faced with."
At the time, the largest memory card Plummer could lay his hands on for testing had an impossibly large 16-megabyte capacity.
"Perhaps I multiplied its size by a thousand," he said, "and then doubled it again for good measure, and figured that would more than suffice for the lifetime of NT 4.0. I picked the number 32G as the limit and went on with my day."
While Microsoft's former leader may have struggled to put clear water between himself and the infamous "640K" [2]quote of decades past, Plummer was clear that his decision process was aimed at NT 4.0 and would just be a temporary thing until the UI was revised.
"That, however, is a fatal mistake on my part that no one should be excused for making. With the perfect being the enemy of the good, 'good enough' has persisted for 25 years and no one seems to have made any substantial changes to Format since then..."
NTFS and exFAT (now widely used in removable storage devices, and the specification of which [3]was recently published by Microsoft ) will cheerfully dispense with the limits imposed by Plummer's decades-old design choices (and there is always the command line option "so you can make a disk as big and inefficient as you'd like, subject to the FAT32 limits").
However, as Plummer put it: "At the end of the day, it was a simple lack of foresight combined with the age-old problem of the temporary solution becoming de-facto permanent."
Been there, done that? Just perhaps not in an operating system running on billions of PCs around the world? Perhaps an email to [4]Who, Me? might be in order. ®
Get our [5]Tech Resources
[1] https://www.youtube.com/channel/UCNzszbnvQeFzObW0ghk0Ckw
[2] https://quoteinvestigator.com/2011/09/08/640k-enough/
[3] https://www.theregister.com/2019/08/28/microsoft_exfat_spec_linux_kernel/
[4] mailto:whome@theregister.com
[5] https://whitepapers.theregister.com/
Re: Future proofing size constraints
Adding to your Gordon Moore and Bill Gates a certain Thomas Watson, president of IBM who declared in 1943: "I think there is a world market for maybe five computers."
Then again, hindsight is a wonderful thing.
Thankfully there are alternatives to FAT32 which overcome the limitation.
Re: Future proofing size constraints
Don't forget Stephen Morse with '1MB is a lot for 1976'; indeed it was, but not having 32-bit segment registers (even with lower 4 bits forced zero to begin with) as a priority? Maybe making SI and DI 32 bits, like having a couple of 16-bit index registers in 8 bit CPUs? That decision alone has cost us a hell of a lot of progress :(
Re: Future proofing size constraints
From memory this was also due to die size, possibly complexity too, and therefore also a significant cost saving.
Re: Future proofing size constraints
Leave us not forget Ken Olsen (DEC) who is said to have asked "Why would anyone want a computer in their house?"
Also, I don't think it was Bill Gates who said the 640K quote, it was an IBM engineer working on the first IBM PCs.
Also, I had assumed that the Format GUI was a simple front end for the DOS Format command.
Re: Future proofing size constraints
And it's always going to be tricky. Use bigger block sizes and you end up with wasted space for small files as described. Use small block sizes and increasing levels of indirect Unix-style and you end up with the performance hit of multiple indirection. It's never going to be easy and the next use case can shatter your ingenious solution.
Re: Future proofing size constraints
"And it's always going to be tricky. Use bigger block sizes and you end up with wasted space for small files as described."
And especially not forgetting that back then, a "program" often came with many, many tiny "support" files not wrapped up in DLLs. eg multiple .ICO files. There was often an enormous amount of wasted space in the cluster "slack space". Windows itself had 1000's of small support files, wasting significant space on large drives. I remember spending many hours optimising my own system at home by trying to get rid of as much of the smaller cruft as possible and trying to balance cluster size against speed, usability and waste.
Re: Future proofing size constraints
I remember making similar calculations as a callow youth in 1982 that we'd have systems with 1-4GB ram in 2000 and thinking it was impossibly huge
"the age-old problem of the temporary solution becoming de-facto permanent"
Hardly surprising. Everything in computing has been a continuous discovery, and why change what works ?
That is why file systems have evolved under different names, and will continue to do so. NTFS is much better than FAT32, but FAT32 has its uses.
Not a mistake, a demonstration that computing has evolved ans will continue to do so.
You can't be perfect the first time around.
Re: "the age-old problem of the temporary solution becoming de-facto permanent"
>>> You can't be perfect the first time around.
Speak for yourself!!
Re: "the age-old problem of the temporary solution becoming de-facto permanent"
"Not a mistake, a demonstration that computing has evolved ans will continue to do so.
Until the de facto changes, enter Electron.
While all my opinion, there's a lot of backwards going on right now in computing. Cloud, flat wasteful interfaces, AIO runtimes, pop-ups Captcha, rental software, OS spying data gathering... if you stop to take a look, you might see the opposite of progression. One thing is for sure, faster computers are almost solely developed to build even faster computers to run all of the above, as if the human element is entirely ignored.
What about FAT file transfer?
3.96gb? Where did that limitation come from? Is this part of the bigger "Because I said so!" movement?
Re: What about FAT file transfer?
File size is stored in 32 bits, giving you a 4 GiB limit.
Re: What about FAT file transfer?
And hit most people when DVDs arrived. Building a 4.7GB ISO image wasn't possible. The only option was to "build and burn" and hope your computer didn't decided to "pause" while burning the disc. (Yes, there were "burn-proof" options and other methods, but most people probably had a few failures before "discovering" how do it successfully)
MS, how about recognising EXT,HFS+ formats so it doesn't result in the format dialog box. FFS 2021.
I have to use 4 different OS's as part of my job, yes, you can use exFAT, FAT32 across these, but it's about time Windows 10 20H2 could recognise these 'unrecognised file systems' and at least exit showing the format type instead, instead of offering a very dated modal format dialog box, which can accidentally result in formatting the bloody thing (usb drive) if you're not careful, as the focus is left dangerously on the 'Format disk" as the default choice.
Come on Microsoft, sort it, you have an underlying Linux shell in Windows 10 20H2 FFS, this has gone beyond a joke. I get it, its aimed to keep people within the Windows ecosystem, but it's a bloody annoying modal dialog box, there has to be a better way of dealing with unrecognised formats.
It's such a poor interface design in 2020, we all have to use different OS's in 2021, as part of our day jobs.
Re: MS, how about recognising EXT,HFS+ formats so it doesn't result in the format dialog box....
That's standard Microsoft Behaviour. As far as possible to only support their own technologies, other systems can make themselves compatible with Microsoft's proprietary standards (often nothing standard about them either).
Re: MS, how about recognising EXT,HFS+ formats
I have to use 4 different OS's as part of my job, yes, you can use exFAT, FAT32 across these, but it's about time Windows 10 20H2 could recognise these 'unrecognised file systems'
W7 throws up a BSOD if you offer it an ODS-2 formatted USB drive; haven't tried with W10 yet and at the moment I don't have that stick at hand (it's at the office, haven't been there since March and I doubt I'll be going there any time soon)
"temporary solution becoming de-facto permanent"
Like the increasingly common but incorrect hyphen in de facto, ab initio and similar?
Re: "temporary solution becoming de-facto permanent"
sim-ilar?
Command line?
My probably dumb assumption is that the dialogue box is merely a front end to save the dumb user from having to use the command line and remember the options.
If so it shouldn't (hah!) be a major development to update the dialogue box to allow a larger cluster size and larger volume.
Unless I am missing something obvious.
Re: Command line?
I would imagine that it has to go through many layers of planning because it will have to have both the accessibility and internationalisation updated, and any documentation for it would also need changing. Not impossible by any stretch, but I doubt it could be done with less than a person-week of resources, and they're too busy removing useful functionality out of Control Panel to update legacy UI
Weekend viewing
Happened across this and the reasoning behind task mangler this weekend.
I'd recommend his "tweaking" the source code for Tempest as well.
"Def-Pro"
"Definitively Provisional"
Reminds me of "provisional buildings" we were in at school, which were there for at least 20 years...
Re: "Def-Pro"
It's even better when the "provisional building" is deemed too small, and an extension is added on!
Re: "Def-Pro"
" ... provisional buildings ... "
When Manchester Central railway station opened in 1880 it had a temporary wooden building housing the ticket offices etc.
That temporary building was still in use when the station closed in 1969.
Re: "Def-Pro"
the primary school I was at had a number of such buildings added in the 1970s, intended to be removed by 1980. Several were dropped on playing areas (tennis courts, etc)
come 2021, They're all still there - and extensions added.
I just hope the coal-burning pot belly stoves have been removed. Then again they were bloody cold in winter even with those
Re: "Def-Pro"
I see your 50-(or so)-year-old buildings and raise you the old gym. at my secondary school, built as a temporary expedient in the 1940s and - apparently - still there today if Google Maps is anything to go by.
Re: "Def-Pro"
Was it just at my school that these temporary buildings were called 'Terrapins' or was that terminology more widespread? I guess that was the name of a brand or model.
UID
I work for yet another big startup having a 32 bit user ID throughout the system. Yes, tell me that "2 billion users would be a nice problem to have" while I browse through this arcane code that grants permanent user IDs to non-paying customers with burner e-mail accounts.
FAT fail
File Allocation Table is terrible design for SSD devices because it requires that the first storage blocks (where the table is stored) are re-written over and over again reducing the life of the device, and is a fixed size irrespective of the size or number of the files .. but has the single advantage of being simple (and standard - copied from CP/M). Things would be different had MS licenced the NTFS format or Unix i-node file-system been open source.. but we are where we are
Re: FAT fail
"File Allocation Table is terrible design for SSD devices because ...."
It's a terrible design for discs too, causing unnecessary head movements, undue wear of the first few data tracks and so on and so forth.
It's good for indexed tapes as you read the file-information first so you know where everything is.
DECTape anyone? (I still have one or two in a box somewhere.)
Future proofing size constraints
I remember a discussion with a developer in the late 90's who was designing gaming software for casinos, and I remember incrediously listening to him telling me that within 20 years we will have 2 tb storage on something the size of a finger nail.
Whelp, last year ( dammit, year before last, we are 2021 now ) Sandisk came out with a 1 terabyte microSD card. I suppose two stacked on top of each other would meet that 2tb definition given to me in 1997, yet in the days of when high tech was a 2 gb hard drive and 16 meg of RAM, 2 tb on a postage stamp was star trek science fiction.
So, I guess we should channel both Moore and his law, and Bill Gates and his 640k is enough for anyone, and actually not try to limit ourselves - Storage is always going to be bigger, and it's going to arrive faster than we think. Flexibility is key, and if you have absolutes, make sure they are actual technical ones - like 32 bits is always going to be 2 or 4 billion depending on signing, and not arbitrary "lets set this number limit at 10 million and call it a day". It will trip you up in database ID sequences just as bad as file system design one day in the - possibly not so distant - future!