Italy's climate super computer, Cassandra, to combine HPC with AI
(2024/05/10)
- Reference: 1715322675
- News link: https://www.theregister.co.uk/2024/05/10/italian_climate_supercomputer/
- Source link:
Boffins in Italy are about to get their hands on a supercomputer that will more than double the resources available to study the effects of climate change.
Dubbed Cassandra, the big iron is based on Lenovo's liquid cooled Neptune system architecture and is slated for [1]deployment at the Euro-Mediterranean Center on Climate Change (CMCC) in Lecce, Italy.
Founded in 2005 as a joint effort by Italy's ministries of environment, finance, and agriculture and forestry, the CMCC is tasked with developing forecast models of the Earth and its oceans, predicting the course of climate change, and establishing policies to mitigate and adapt to the effects of a warming planet.
[2]
This work is currently being conducted on CMCC's Juno system, which was deployed in 2022 and is based on Intel's third-gen Xeon Scalable Platform backed by a complement of 20 Nvidia A100 GPUs. Despite spanning 170 nodes, the existing system isn't all that powerful – managing just 1.13 petaFLOPS of peak double precision performance.
[3]
[4]
With 180 nodes, CMCC's Cassandra is by no means a small system – at least not physically. However, the bulk of those nodes are CPU powered, so it's nowhere near as compute dense as you might expect from a GPU-accelerated cluster. The system's 20,160 Xeon Max cores and 26TB of HBM2e memory are expected to produce 1.2 petaFLOPS of FP64 performance when it comes online later this year.
By way of comparison, the Nvidia DGX SuperPOD that MITRE [5]announced it was deploying this week contains just 32 servers but has an estimated peak performance of 17 petaFLOPS – showing just how much of the heavy lifting GPUs are responsible for in modern systems.
It's not always about FLOPS
However, as researchers at Los Alamos National Lab have previously pointed out, different workloads have different bottlenecks – more peak FLOPS doesn't necessarily translate into higher performance.
"Given the hoopla in the press around the 'fastest supercomputer in the world,' one might think we should buy computers with the most FLOPS," Gary Grider, who heads up LANL's HPC division, said of the Crossroads system, which was installed late last summer.
[6]
While Crossroads is saddled with a far [7]darker purpose , it is a relevant comparison in this case. As far as we can tell, it is based on the same Xeon Max foundation as Cassandra.
Launched in early 2023, the Xeon Max is packed with up to 64GB of HBM, capable of feeding the chip's 56 cores with 1TB/sec of memory bandwidth – that's a lot for a CPU. Other chips, like Fujitsu's Arm-based A64FX, have previously [8]featured on-package HBM, but Xeon Max marked the first time we'd seen it on an x86 part.
As is the case in many HPC and AI workloads, memory bandwidth remains a major performance bottleneck and HBM offers a substantial advantage over traditional DRAM modules – certainly for nuclear weapons sims. As our sibling site The Next Platform has [9]discussed in the past, it also works for weather forecasting.
[10]
While the bulk of the Cassandra system will be CPU-based, and will likely target well established HPC workloads, that's not to say GPUs won't play a role.
CMCC plans to add a pair of GPU nodes totaling 16 H100 accelerators to Cassandra to accelerate AI workloads. While two nodes might not sound like much, it will actually add an additional petaFLOP of FP64 performance to the system. And for AI workloads, it'll add as much as 64 petaFLOPS at 8-bit precision.
The climate lab aims to run a variety of AI-based climate change simulations on this GPU partition.
[11]SpiNNcloud Systems unveils Arm-based 'neuromorphic supercomputer'
[12]MITRE promises a cute little 17-PFLOPS AI super for the rest of Uncle Sam's agencies
[13]Cheyenne supercomputer sells at auction for just $480K
[14]IBM quantum system elbows into Arm-powered Fugaku supercomputer
Maybe we don't need all that precision after all
Meteorological and climate modeling have traditionally been considered HPC workloads, where double precision FP64 calculations are the gold standard.
However, over the past few years work at various research institutions and companies has shown that many of the calculations can actually be done at single (32-bit) or even half (16-bit) precision at far higher performance without compromising on accuracy – particularly when it comes to weather and climate.
For instance, the European Center for Medium Range Weather Forecasts has already [15]demonstrated the benefits of 32-bit calculations in weather and climate modeling, while researchers at the University of Bristol managed to [16]eke out a 3.6x speed boost by dropping down to lower precisions.
At the more extreme end of the spectrum, we have Nvidia's Earth-2 climate model. Announced at GTC this spring, the massive digital twin uses AI to accelerate high-resolution simulations of the climate down to two kilometers of resolution.
Earth-2 [17]employs a diffusion model called CorrDiff, which Nvidia claims is capable of generating images at 12.5x higher resolution and 1,000x faster than current numerical models. Because of this, Taiwan is already eyeing the platform to improve its tsunami forecasting.
It's not hard to imagine CMCC using its GPU partition to augment or accelerate traditional HPC simulations using AI in a similar fashion. ®
Get our [18]Tech Resources
[1] https://news.lenovo.com/pressroom/press-releases/cmcc-lenovo-supercomputer-cassandra/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/05/08/mitre_ai_computer/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/09/02/los_alamos_supercomputer/
[8] https://www.theregister.com/2023/03/06/fujitsus_a64fx_successor_will_be/
[9] https://www.nextplatform.com/2019/01/08/gpus-gather-force-in-large-scale-weather-forecasting/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/05/08/spinncloud_systems_unveils_armbased_neuromorphic/
[12] https://www.theregister.com/2024/05/08/mitre_ai_computer/
[13] https://www.theregister.com/2024/05/06/cheyenne_supercomputer_sold/
[14] https://www.theregister.com/2024/04/30/fugaku_supercomputer_ibm_quantum_module/
[15] https://www.ecmwf.int/en/about/media-centre/news/2021/forecast-upgrade-innovates-single-precision-and-ensemble-resolution
[16] https://www.nextplatform.com/2021/08/23/more-proof-points-for-low-precision-hpc/
[17] https://nvidianews.nvidia.com/news/nvidia-announces-earth-climate-digital-twin
[18] https://whitepapers.theregister.com/
Dubbed Cassandra, the big iron is based on Lenovo's liquid cooled Neptune system architecture and is slated for [1]deployment at the Euro-Mediterranean Center on Climate Change (CMCC) in Lecce, Italy.
Founded in 2005 as a joint effort by Italy's ministries of environment, finance, and agriculture and forestry, the CMCC is tasked with developing forecast models of the Earth and its oceans, predicting the course of climate change, and establishing policies to mitigate and adapt to the effects of a warming planet.
[2]
This work is currently being conducted on CMCC's Juno system, which was deployed in 2022 and is based on Intel's third-gen Xeon Scalable Platform backed by a complement of 20 Nvidia A100 GPUs. Despite spanning 170 nodes, the existing system isn't all that powerful – managing just 1.13 petaFLOPS of peak double precision performance.
[3]
[4]
With 180 nodes, CMCC's Cassandra is by no means a small system – at least not physically. However, the bulk of those nodes are CPU powered, so it's nowhere near as compute dense as you might expect from a GPU-accelerated cluster. The system's 20,160 Xeon Max cores and 26TB of HBM2e memory are expected to produce 1.2 petaFLOPS of FP64 performance when it comes online later this year.
By way of comparison, the Nvidia DGX SuperPOD that MITRE [5]announced it was deploying this week contains just 32 servers but has an estimated peak performance of 17 petaFLOPS – showing just how much of the heavy lifting GPUs are responsible for in modern systems.
It's not always about FLOPS
However, as researchers at Los Alamos National Lab have previously pointed out, different workloads have different bottlenecks – more peak FLOPS doesn't necessarily translate into higher performance.
"Given the hoopla in the press around the 'fastest supercomputer in the world,' one might think we should buy computers with the most FLOPS," Gary Grider, who heads up LANL's HPC division, said of the Crossroads system, which was installed late last summer.
[6]
While Crossroads is saddled with a far [7]darker purpose , it is a relevant comparison in this case. As far as we can tell, it is based on the same Xeon Max foundation as Cassandra.
Launched in early 2023, the Xeon Max is packed with up to 64GB of HBM, capable of feeding the chip's 56 cores with 1TB/sec of memory bandwidth – that's a lot for a CPU. Other chips, like Fujitsu's Arm-based A64FX, have previously [8]featured on-package HBM, but Xeon Max marked the first time we'd seen it on an x86 part.
As is the case in many HPC and AI workloads, memory bandwidth remains a major performance bottleneck and HBM offers a substantial advantage over traditional DRAM modules – certainly for nuclear weapons sims. As our sibling site The Next Platform has [9]discussed in the past, it also works for weather forecasting.
[10]
While the bulk of the Cassandra system will be CPU-based, and will likely target well established HPC workloads, that's not to say GPUs won't play a role.
CMCC plans to add a pair of GPU nodes totaling 16 H100 accelerators to Cassandra to accelerate AI workloads. While two nodes might not sound like much, it will actually add an additional petaFLOP of FP64 performance to the system. And for AI workloads, it'll add as much as 64 petaFLOPS at 8-bit precision.
The climate lab aims to run a variety of AI-based climate change simulations on this GPU partition.
[11]SpiNNcloud Systems unveils Arm-based 'neuromorphic supercomputer'
[12]MITRE promises a cute little 17-PFLOPS AI super for the rest of Uncle Sam's agencies
[13]Cheyenne supercomputer sells at auction for just $480K
[14]IBM quantum system elbows into Arm-powered Fugaku supercomputer
Maybe we don't need all that precision after all
Meteorological and climate modeling have traditionally been considered HPC workloads, where double precision FP64 calculations are the gold standard.
However, over the past few years work at various research institutions and companies has shown that many of the calculations can actually be done at single (32-bit) or even half (16-bit) precision at far higher performance without compromising on accuracy – particularly when it comes to weather and climate.
For instance, the European Center for Medium Range Weather Forecasts has already [15]demonstrated the benefits of 32-bit calculations in weather and climate modeling, while researchers at the University of Bristol managed to [16]eke out a 3.6x speed boost by dropping down to lower precisions.
At the more extreme end of the spectrum, we have Nvidia's Earth-2 climate model. Announced at GTC this spring, the massive digital twin uses AI to accelerate high-resolution simulations of the climate down to two kilometers of resolution.
Earth-2 [17]employs a diffusion model called CorrDiff, which Nvidia claims is capable of generating images at 12.5x higher resolution and 1,000x faster than current numerical models. Because of this, Taiwan is already eyeing the platform to improve its tsunami forecasting.
It's not hard to imagine CMCC using its GPU partition to augment or accelerate traditional HPC simulations using AI in a similar fashion. ®
Get our [18]Tech Resources
[1] https://news.lenovo.com/pressroom/press-releases/cmcc-lenovo-supercomputer-cassandra/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/05/08/mitre_ai_computer/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/09/02/los_alamos_supercomputer/
[8] https://www.theregister.com/2023/03/06/fujitsus_a64fx_successor_will_be/
[9] https://www.nextplatform.com/2019/01/08/gpus-gather-force-in-large-scale-weather-forecasting/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/hpc&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zj3wQqCnUe@-XY@VV8h6cgAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/2024/05/08/spinncloud_systems_unveils_armbased_neuromorphic/
[12] https://www.theregister.com/2024/05/08/mitre_ai_computer/
[13] https://www.theregister.com/2024/05/06/cheyenne_supercomputer_sold/
[14] https://www.theregister.com/2024/04/30/fugaku_supercomputer_ibm_quantum_module/
[15] https://www.ecmwf.int/en/about/media-centre/news/2021/forecast-upgrade-innovates-single-precision-and-ensemble-resolution
[16] https://www.nextplatform.com/2021/08/23/more-proof-points-for-low-precision-hpc/
[17] https://nvidianews.nvidia.com/news/nvidia-announces-earth-climate-digital-twin
[18] https://whitepapers.theregister.com/
Re: No mention of power usage?
Filippo
What are you getting at? That we should... not study climate change, in order to save power?
That does not sound wise.
Re: No mention of power usage?
Zibob
Just that its conspicuous by its omission. Seems to be a metric in many other supercomputer posts. Why not this one?
And its a legitimate concern, we know having them in local built up areas is not a great idea. So why Italy, while they do have the alps, most of it is know for being sunny and warm. Cooling is a large part of the power requirements.
There are location in Europe far better equied to do what this needs but in a more environmentally freidnly way. Natural water cooling? Cold air intake from naturally cold areas. Areas with surplus renewable energy frequently.
It would seem odd to me that with its intended purposes every effort was not made to make it as efficient, cool running and supplied by as much renewable energy as possible.
And at the end of it, we will just confirm want we already know, then offer ideas on how to make a future possible, and have them ignored by the governments and corporations.
Unbelievable!
Doctor Syntax
Cassandra? Really?
No mention of power usage?
I would have thought that a climate modeling supercomputer would want to be extremely efficient energy usage wise, though I'm not surprised to see it not mentioned.
I'm sure there's no good way to spin your local area losing power due to this thing and its ilk using so much, only to have to explain that the coal/oil power plants could not keep up with demand and admit the renewable energy is no where near close for this application.
All while spinning up gas fired back up generators to run a computer trying to find a way to do less damage to the climate...