Can gamers teach us anything about datacenter cooling? Lenovo seems to think so
- Reference: 1667221242
- News link: https://www.theregister.co.uk/2022/10/31/gamers_datacenter_cooling/
- Source link:
Liquid cooling technologies developed decades ago for use in mainframe computers and now commonplace in supercomputers would seem to be the obvious answer. For some of these components, it may even be a [1]requirement . But for those who aren't ready to commit to rearchitecting their datacenters to support the tech, could cooling kit developed for gaming systems offer a stopgap?
Scott Tease, Lenovo's VP of HPC and AI, seems to think so. In addition to Lenovo's [2]Neptune direct liquid-cooled systems, his team is also working on adapting technologies you might find in gaming PCs and notebooks for use in the datacenter.
[3]
"We used to complain about 145W CPUs. This generation we're seeing 300W CPUs and by 2024, there's no doubt in my mind we will have 500W CPUs in market," he says. "We're kind of running out of room for air to do what we need it to do at the densities we're used to seeing."
Inspired by gamers
One of the technologies Lenovo is exploring is the use of closed-loop liquid-cooling systems, not unlike the all-in-one coolers used in high-end gaming systems.
For those who aren't familiar, these coolers are made up of three main components: a cold plate, which attaches to the CPU or GPU; a pump that moves liquid through the system; and a radiator that dissipates heat, usually with the assistance of fans.
[4]
[5]
It's not unlike how an internal combustion vehicle is cooled, Tease explains. Liquid is pumped through the engine and then cooled by a radiator at the front of the vehicle.
The tech differs from conventional datacenter liquid cooling setups in that the fluid doesn't actually evacuate the heat from the system, but rather relocates it within the chassis where it is easier to deal with. This, he says, means hotter components like 400W+ SXM GPUs can be packed into smaller chassis sizes. And because they're self-contained, these systems don't require any changes to existing rack or datacenter infrastructure.
[6]
Lenovo is also exploring the use of dense heat pipe arrays where liquid cooling might not be practical or desired. Heat pipes are self-contained vapor chambers designed to passively pump heat around. While the tech is by no means new, heat pipes are particularly popular in gaming systems and notebooks to move heat away from components in space-constrained environments.
Lenovo is applying this same concept in multi-socket blade servers to keep the rear CPU cool. Lenovo calls these "thermal transfer modules," and they are essentially a stretched-out heat sink.
"What it allows us to do is one of two things, either put in a higher-end processor – some HPC clients like that – or if you go with a more standardized processor, we're able to lower our fan speeds and then in turn lower acoustics and power consumption," Tease says.
Fanning the flames
And it's not just the CPUs and GPUs that are getting more power hungry, according to Tease. The percentage of power consumed just by a system's fans has skyrocketed in recent years.
"In the old days – by old days I mean 2014, 2015 – it was a small budget, maybe 5 percent. In failure mode, it was maybe 7 percent," he tells The Reg . "Because parts are so high-power and T-cases are so low, we're pushing 13 percent just in normal operation."
[7]
That means for a 1KW, 1U dual-socket system, 130W is taken up by the fans. And in environments where customers are operating at extended temperature ranges, Tease says as much as 20 percent of a system's power consumption can be attributed to the fans.
"We think that's going to get worse in the 2024 time frame," he says, adding that anything which reduces the number or the speed of the fans can have a major impact on power consumption.
[8]Oil company Castrol slips and slides into immersion cooling
[9]Intel's 13th-gen CPUs are hot, hungry, loaded with cores
[10]US Department of Energy has $42m to make datacenter cooling more efficient
[11]Lessons to be learned from Google and Oracle's datacenter heatstroke
Fans are also one of the reasons that liquid-cooled systems are so comparable in cost to air-cooled systems. According to Tease, Lenovo's liquid and air-cooled systems are usually within 2 percent of each other in terms of upfront cost.
Transitioning to direct liquid cooling for instance means you can ditch the fans entirely, and because you no longer have to power 130W of fans, you can opt for a smaller power supply or achieve greater performance per watt by using higher-power CPUs.
"It's far more cost effective, total picture, to go with liquid cooling," Tease says, adding that there are also opportunities to reuse heat captured from liquid-cooled datacenters for things like district heating.
Microsoft, for example, [12]plans to do just that at its new Helsinki datacenter announced in March.
The rack power bottleneck
While liquid cooling may have benefits in terms of power draw compared to fan-heavy, air-cooled systems, customers probably won't need to make the switch just yet.
"I would say we're probably five years away from really being extremely limited in what air quality is going to be able to do for us," Tease says.
Part of the problem relates to rack power. Even if liquid cooling allows users to achieve greater compute density in a smaller form factor, they'll still be limited by how much power can be supplied to a single rack.
With a very healthy 42KW budget, even if you could cram a bunch of 2KW 1U chassis into it, you'd run out of power before your rack is half full. And that's for 42KW, which many colocation providers don't even come close to.
"You could make the case that a 96-core or 128-core, one-socket system is equivalent to a two-socket system from last generation," Tease adds. "Or maybe you go from 1U to 2U with these giant heat sinks and lose density that way, but maybe one two-socket 2U is better than two one-socket 1Us."
In other words, for customers who aren't running HPC or AI/ML workloads, it may be better to opt for a bigger chassis with larger, slower fans and heat sinks, at least until TDPs reach the point at which datacenter operators are forced to increase their rack power budgets. ®
Get our [13]Tech Resources
[1] https://www.theregister.com/2022/08/23/intel_ponte_vecchio/
[2] https://www.nextplatform.com/2022/09/07/how-multi-scaled-hpc-enabled-genomics-will-help-save-your-life/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y1--MKOJWVTZyhQKItWH6wAAAFU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y1--MKOJWVTZyhQKItWH6wAAAFU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y1--MKOJWVTZyhQKItWH6wAAAFU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y1--MKOJWVTZyhQKItWH6wAAAFU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y1--MKOJWVTZyhQKItWH6wAAAFU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/10/17/castrol_to_open_immersion_cooling/
[9] https://www.theregister.com/2022/09/27/intel_13thgen_cpu_innovation/
[10] https://www.theregister.com/2022/09/26/coolerchips_program_doe/
[11] https://www.theregister.com/2022/08/22/google_oracle_heatstroke/
[12] https://www.theregister.com/2022/03/17/microsoft_datacenter_finland/
[13] https://whitepapers.theregister.com/
Re: Isn't it statingthe obvious that liquid cooling is more effective than air cooling?
In case anyone is wondering I've found the best anti-mould/corrosion agent is Sentinal X100 which is what we put in our central heating systems especially as the average central heating system contains the same mix of metals, seals, plastics as computer water cooling
Re: Isn't it stating the obvious that liquid cooling is more effective than air cooling?
It is stating the obvious that liquid cooling is more effective than air. The more interesting statement though, is that it is becoming _cheaper_. That is a (big) change - liquid cooling for mainframes went out of fashion because it was more expensive than air cooling. (Switching from ECL to CMOS helped, a lot, by reducing the amount of heat you needed to get rid of.)
Re: Isn't it stating the obvious that liquid cooling is more effective than air cooling?
Exactly, which is cheaper, easier to install, easier to maintain, causes less problems if something comes lose, Air cooling or water?
Water is only used if you need it, because air cooling is good enough. Air cooling doesn't destroy a rack if it leaks.
Doesn't require each need rack to be plumbed in to the cooling system.
If you don't need water cooling, you don't go with it, its more complex and prone to failure.
If you don't need it, but it will save you money, then you evaluate if the cost benefits outweigh the risks.
Re: Isn't it statingthe obvious that liquid cooling is more effective than air cooling?
Integrated liquid cooling absolutely is a thing, but it isn't particularly common for a number of reasons:
- Most data centres have a mix of equipment: specialised systems from vendors, a few generations of server hardware, networking gear, power handling equipment etc. Unless all of it supports liquid cooling, you need to implement an effective air cooling system across most of the datafloor anyway. At that point, adding liquid cooling is just extra expense.
- Risky failure modes - liquid cooling everything means an awful lot of plumbing and manifolds, and a lot of connections to all the systems. Leaks are bad news. Keeping all the lines air free is also demanding.
- Flow management - its hard for an individual system to manage its cooling demand, so you're likely to be over-cooling those systems not running at full capacity, which will reduce efficiency.
- Air handling efficiency can be improved significantly with simple implementations like enclosing hot/cold aisles.
- Systems often contain a bunch of components which passively use the airflow, but don't demand enough power to justify a liquid cooled variant - like RAID cards, or NICs, or SAS expanders, or... - so its difficult or expensive to make those general purpose systems work. These aren't considerations on desktop-type systems because there's still substantial passive cooling capacity to tolerate it.
- Datacentre operators often are in a position of supporting what their users want to run, rather than dictating (even for in-house DCs, but doubly so for colocation providers).
Think they're tackling the wrong problem
Designers really need to be looking to lower power consumption in new generations of chips, not increase it.
Re: Think they're tackling the wrong problem
Even if the overall power doesn't increase, having a smaller die so your watts per unit area go up causes issues.
Cost is key
I used to work in the rugged board industry.
Think, conduction (board edge) cooled, -45 to +85C ambient temperature, with many of the current generation high end processor and graphics devices.
Your water/refrigerant/etc. is contained in the solid rack holding the boards and so is kept well away from the electronics.
However, this all costs the sort of money which restricts the markets to those that can bear the expense.
My high-end home PC is all conduction cooled (with off-the-shelf parts) and solid state discs - so silent. :-) (about 20% more cost than a noisy behemoth!)
I have seen a total immersion cabinet at a trade show. That was pretty impressive! Needed to keep the coolant exceptionally high resistance though - exceptionally pure. Had a massive specific heat capacity, though.
"you'd run out of power before your rack is half full"
Okay, for me that begs the question : how many 1U units can a rack hold ?
Because if 2KW units are put in a 42KW rack, it should hold at least 21 of them.
But if 21 units is less than half the rack's capacity, that would mean that a rack can hold in excess of 42 units.
Did I get that right ? How tall are those racks anyway ?
Isn't it statingthe obvious that liquid cooling is more effective than air cooling?
Simply from the observation that the heat capacity of water (the liquid usually used for this, plus detergents, anti-mould agents and such) is much, much higher than that of air, and it's a lot easier to get a laminar flow through a pipe than it is blowing air through a case.
How is it only now that data centres are considering moving heat about effectively, rather than blowing air about, and then using secondary heat-pumps, in the form of air-conditioners, to cool the room down?
Rather than being behind the curve in terms of what gaming rigs manage, why aren't they looking at hooking those refrigerant circuits directly into the racks, and onto the components that need direct cooling? Because one thing that is more effective than pumping water about is pumping a refrigerant that changes phase when it is heated, and releases that heat when it is re-condensed, usually outside the building.