New year, new rant: Linus Torvalds rails at Intel for 'killing' the ECC industry
- Reference: 1609768813
- News link: https://www.theregister.co.uk/2021/01/04/linus_torvalds_intel_killed_ecc/
- Source link:
ECC stands for error-correcting code. ECC memory uses additional parity bits to verify that the data read from memory is the same as the data that was written. Without this check, memory is vulnerable to occasional corruption where a bit is flipped spontaneously, for example, by background radiation. Memory can also be attacked using a technique called [1]Rowhammer , where rapid repeated reads of the same memory locations can cause adjacent locations to change their state.
ECC memory solves these problems and has been available for over 50 years yet most personal computers do not use it. Cost is a factor but what riles Torvalds is that Intel has made ECC support a feature of its Xeon range, aimed at servers and high-end workstations, and does not support it in other ranges such as the Core series.
A [2]promotional video on Intel's website explains that "with today's fast CPU and memory speeds, soft memory errors occur more and more frequently... corruption of critical data can be disastrous to business performance and company reputation. Intel's Xeon processors and error-correcting code, or ECC memory, work together to automatically find and fix soft memory errors."
An [3]Intel ad proclaims that "1 in 3 systems experience one or more correctable memory errors a year... ECC memory available on servers, not available on desktop PCs."
The topic came up in a discussion about AMD's new Zen 3 Ryzen 9 5000 series processors on the Real World Tech forum site. AMD has semi-official ECC support in most of its processors. "I don't really see AMD's unofficial ECC support being a big deal," said an unwary contributor.
"ECC absolutely matters," [4]retorted Torvalds. "Intel has been detrimental to the whole industry and to users because of their bad and misguided policies wrt ECC. Seriously. And if you don't believe me, then just look at multiple generations of rowhammer, where each time Intel and memory manufacturers bleated about how it's going to be fixed next time... And yes, that was – again – entirely about the misguided and arse-backwards policy of 'consumers don't need ECC', which made the market for ECC memory go away."
The accusation is significant particularly at a time when security issues are high on the agenda. The suggestion is that Intel's marketing decisions have held back adoption of a technology that makes users more secure – though rowhammer is only one of many potential attack mechanisms – as well as making PCs more stable.
"The arguments against ECC were always complete and utter garbage. Now even the memory manufacturers are starting to do ECC internally because they finally owned up to the fact that they absolutely have to," said Torvalds.
Torvalds [5]said that Xeon prices deterred usage. "I used to look at the Xeon CPU's, and I could never really make the math work. The Intel math was basically that you get twice the CPU for five times the price. So for my personal workstations, I ended up using Intel consumer CPU's." Prices, he said, dropped last year "because of Ryzen and Threadripper... but it was a 'too little, much too late' situation."
By way of mitigation, he added that "apart from their ECC stance I was perfectly happy with [Intel's] consumer offerings."
Intel's marketing strategy is perhaps now working against it as ECC could become a selling point for AMD, though ECC memory requires support from both motherboard and CPU. "Intel's marketing antics meant that there was a pent up demand for alternatives. Obviously the ARM people would have loved to fill that demand, but right now it's AMD that stepped up instead," Torvalds said.
Currently AMD's ECC support in its Ryzen range is unofficial, which is a problem: many motherboards do not support it, or do not advertise support, and business users will be correspondingly wary of relying on it.
We have asked Intel and AMD for comment. ®
Get our [6]Tech Resources
[1] https://www.theregister.com/2020/01/06/roundup_january_2/
[2] https://www.intel.com/content/www/us/en/processors/xeon/the-value-of-ecc-memory-for-servers.html
[3] https://www.intel.com/content/www/us/en/processors/xeon/xeon-ecc-memory-infographic.html
[4] https://www.realworldtech.com/forum/?threadid=198497&curpostid=198647
[5] https://www.realworldtech.com/forum/?threadid=198497&curpostid=198534
[6] https://whitepapers.theregister.com/
Re: Maximise Profit Margins.
Maximise short term profit margins
FTFY
Alternatives are available
For example, see https://www.intelligentmemory.com/dram-components/ecc/
This allows automatic single-bit correction for CPUs that don't support ECC natively.
Re: Alternatives are available
That's very nifty. I wonder how they do it without adding read latency. Are they underspeccing the array performance?
Re: Alternatives are available
I wouldn't dream of hosting a ZFS file server on hardware without ECC. What's the point of being paranoid about your disks and not your RAM?
Re: Alternatives are available
ZFS has a lot of advantages, even without ECC (and the "scrub of death" is a myth).
I would like to have ECC on my home-brew FreeNAS, but I can't justify the additional motherboard and RAM costs or the extra power draw (server grade boards tend to be much more power hungry).
However, I insist on it for business grade NAS solutions.
I don't see it that way
The Linuxheads will flame me for this, but I'm sorry Linus, I don't believe you are correct.
My Lenovo is equipped with a Xeon, and supports ECC. Yet it didn't come equipped with it. And the huge majority of the models in the same line didn't, from the factory.
Intel built the Xeon with ECC support. Yet here's an entire model range that needed to be ordered as a custom config in order to have it from the factory.
Why? It wasn't Intel's fault: ECC memory is just too expensive for most people's cost/benefit equations.
Nothing to do with Intel. You want more ECC market penetration? Tell the OEM manufacturers, from memory to computer, to stop price gouging on that single extra bit. Some computer manufacturers charge a real premium just to upgrade to ECC.
Re: I don't see it that way
I don’t believe the cost argument. The cost difference should be minimal (and of course would evaporate almost entirely if ECC were used everywhere). If ECC memory is significantly more expensive than non-ECC then that’s an artificial sales/marketing thing - it’s not because of any intrinsic significant additional expense
Re: I don't see it that way
Isn't that the point? The costs should be minimal, but aren't.
Re: I don't see it that way
You need 9 RAM chips instead of 8, right? That is in line with last time (years ago, I admit) I compared prices for ECC vs. non-ECC RAM (all from the same place). ECC was about 15 per cent more expensive, as to be expected. And the prices of that dealer were quite OK. Has that changed so much?
Even longer ago ECC was just overpriced (similar to Xeons), costing twice as much or more than non-ECC RAM.
Re: 15% surcharge
But it's not. As one example, a quick search through Newegg showed only one 2666mHz ECC SODIMM from a major manufacturer, a 16gb part from Hynix
https://www.newegg.com/p/pl?storeName=Laptop-Memory&pageTitle=Laptop+Memory&
N=100007609+601204087+600006161&Submit=ENE
at $91.86. A 32gb non-ECC dual SODIMM kit from a major manufacturer only averages around $122 or so
https://www.newegg.com/p/pl?storeName=Laptop-Memory&pageTitle=Laptop+Memory&N=100007609+500002048+601204087&Submit=ENE
Yes, that's laptop. But downgrade a Dell Precision 5820 desktop workstation to an Intel i7, from a Xeon, and the 16gb RAM kit is $124.98 less expensive. A shown, you can double the RAM for that amount of money.
So there is DEFINITELY a price difference.
Re: 15% surcharge
Oh dear, we seem to be back at "double price". Thanks for the information.
Re: I don't see it that way
ECC simply adds an extra 8 bits of the same memory device to a 64 bit interface for DDRx SDRAM systems (which is what just about everything in this arena uses).
So the cost of parts is 12.5% more for the memory device (and probably far less than 1% at the system level).
The ECC is done in the memory controller (part of the microprocessor) and as it is a standard thing, the marginal cost of adding it is tiny - a little bit more silicon real estate.
There are some extra PCB tracks to be added but given how many are already present they add little cost (might make PCB routing more interesting but it really is not that much more complexity). Initialisation of memory takes a bit longer (every memory location has to be written a valid value for the ECC to be valid) although it is possible to simply initialise a process space when it is first used.
The only reason I can see for Intel to take this stance is to make sure that those with a need for ECC have to pay a hefty premium.
I have done designs with ECC for decades and the marginal cost doesn't really even show up in the grands scheme of things.
Re: I don't see it that way
I have to agree with you. ECC requires an ecosystem, not just a chip, and Intel doesn't control that. As the article points out, AMD's chip doesn't work without supporting motherboards and memory chips either. It's not just the Xeon chip prices, the Xeon ecosystem prices are higher across the board with MB and RAM vendors charging more as well to fully support ECC.
Re: I don't see it that way
If you are using a Ryzen-based AMD system (maybe earlier models - I just haven't checked), all support at least unbuffered ECC out-of-the-box as long as you can stand the potential performance hit of running at 2666MHz/CL19 vs faster non-ECC memory and potentially being told that "the RAM isn't on the approved memory list for this motherboard" by the motherboard vendor although most vendors will support it on a selection of their offerings.
I have a 64GB Ryzen platform that hasn't had any unexplained crashes in a year versus it's Intel equivalent without ECC that has hung twice. It's a home lab so its just annoying rather than critical but it's nice not having to fix VM's following a crash. If only time was worth something... Kingston/Crucial do "affordable" unbuffered ECC so I only paid around 20% more for the priviledge.
Re: I don't see it that way
"Why? It wasn't Intel's fault: ECC memory is just too expensive for most people's cost/benefit equations."
The CPU supports ECC but does the chipset? And who makes the chipset? That is the artificial segmentation Linus is referring to and yes, it is created by Intel to differentiate their products and then OEM's are left to choose between the different pricing tiers.
In terms of the cost difference between ECC and non-ECC it SHOULD be around 15% (i.e. the cost of adding a 9th RAM chip for every 8 existing RAM chip plus a little more on the chipset/) but is instead often 100%+ more because of the relatively low volumes.
And as for error rates? A computer that is on 24x7 will see ~3 correctable errors a year (http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf) although that maybe a little on the low side with more than 4GB now being common.
Re: I don't see it that way
>It wasn't Intel's fault: ECC memory is just too expensive for most people's cost/benefit equations."
Intel kept ECC for the server market to charge a premium for Xeons
If you had ECC on all CPUs then cheap RAM would become available with ECC and everyone would just buy desktop machines instead
Remember you have to buy Xeon (or preferably Itanium) for 'real' work
The Party Line
The official talking point was that the added circuitry for ECC memory (including the extra bits of storage) would actually reduce the reliability of most systems because there would be more parts to fail. This while simultaneously claiming ECC was needed for servers with their massive memory capacities of up to 4 GB! (Windows NT for servers) Considering a typical consumer build of the last decade had as much memory as a server of the Y2K era, that argument sounds a little weak, doesn't it?
Re: The Party Line
> that argument sounds a little weak, doesn't it?
You have about the same chance of a single-bit error than you have with non-ECC. But that error is corrected! If you have two errors you at least find out. Note that the chance of a double error should be monumentally small, so a double error is an indication some module may be dying.
So, no, ECC really does not have any disadvantage to speak of.
Absolutely
Although the article overstates with "ECC memory solves these problems" when actually "ECC memory mitigates these problems" by correcting single bit errors reliably.
Re: The Party Line
Depends if you count slow performance (as opposed to repeated kernel panics) to be an issue. In my experience, when ECC RAM goes bad, it makes machines monumentally slow with no logical reason as to why, until you use the paid MemTest86 and discover thousands of "corrected errors" over a few days worth of rounds.
Re: The Party Line
Maybe you should post the results of Memtest86 on a machine with faulty ECC RAM and faulty non-ECC RAM to allow us to compare the difference?
Next you will be saying that a degraded RAID volume runs slower as well...
Rant?
From here it looks like the anger management classes are doing the trick. And breath
I've recently bought a Ryzen system, and just finding memory sticks that would work was a pain. The mainboard is clearly advertised as able to use DDR4 3800, but the first 2 sets I tried at merely 3600 did not work, even though the 2nd was on the manufacturer compatible list. And their support told me that AMD only supports 3200, anything else not working is too bad, but not their problem.
And that's not getting into the GPU issues I got afterwards.
So I'm not likely to bother hunting for ECC on AMD until it's officially supported. I'll continue stuffing important data on my Xeon which has it.
Recent AMD mobo
My b550 mobo claims to support ECC, though I haven't tried it. I just built a pretty nice system on one. Since I only use it for video editing, it doesn't
need huge long uptimes at one go, I use a less power hungry system for a daily driver.
Doesn't the ECC check slow things down a little bit? Or is it always pipelined so the CPU finds out a couple cycles later (if it hasn't already crashed)?
IIRC that's how it used to be done.
Re: Recent AMD mobo
ECC is built for long uptimes - speeds will be slower than the fastest RAM but likely within 1%-3% of most value RAM worst case. Caching will typically hide the difference on systems with large L2/L3 caches.
Who is this "Intel" ?
Why is Linus complaining about the predecessor to AMD? Those people are old news.
It could also be argued that complaining about x86 is a waste of time because the world is poised for a move to ARM anyway.
RAMBUS and ECC systems
Years ago - Intel was pushing ECC, but only on RAMBUS as they owned patents on it. Once their wanabe monopoly failed (licensing would have been so money generating) they dropped the ball and walked off the field like a cry baby. Don't cry when the world doesn't want to pay fees to you for ever for what should be opensource. Keep the ball rolling, and let the game get better.
I guess the lack of ECC support on desktop processors must be new?
My FreeNAS server uses an ASRock E3C224D2I server motherboard and a Core i3-4150 CPU. Although the motherboard is a server version, complete with IPMI and a "just trust us" BMC to make a headless life easier, the CPU is most definitely described by Intel as a desktop part.
The combination supports ECC just fine.
Re: I guess the lack of ECC support on desktop processors must be new?
"The combination supports ECC just fine."
It's the memory controller that limits the use of ECC on Intel motherboards and anything from the last 12 years should support ECC if the chipset supports it, although I'm sure Intel will have a handful of exceptions...
You have a C224 chipset and yes, it supports ECC for all supported CPU's.
Maximise Profit Margins.
There, explained in three words.