Brain-inspired neuromorphic computer SpiNNaker overheated when coolers lost their chill
- Reference: 1746530951
- News link: https://www.theregister.co.uk/2025/05/06/spinnaker_overheat/
- Source link:
Brain-inspired chips promise ultra-efficient AI, so why aren't they everywhere? [1]READ MORE
According to Professor Steve Furber, now retired (although he told El Reg "SpiNNaker is still seen as my baby!"), a failure with the cooling on April 20 led to a rise in temperatures until the servers were manually shut down the following day.
The [2]SpiNNaker (Spiking Neural Network Architecture) project is all about simulating a brain by connecting hundreds of thousands of Arm cores. While a human brain presents a huge challenge, Furber, one of the designers of the original Arm processor, reckoned a mouse brain was possible.
During a talk earlier this month to celebrate the [3]40th anniversary of the switch-on of the first Arm processor , Furber told the audience that the hope was to model "one whole mouse" at the required level of detail.
Assuming the hardware survived its baking.
[4]
"SpiNNaker," he told The Register , "is hosted in the Kilburn Building, which was completed in 1972 as a purpose-built computer building and, as such, has a plant room that supplies chilled water as a utility to all the central machine rooms.
[5]
[6]
"The SpiNNaker room was built to house the machine in 2016 in what used to be the mechanical workshop, and is cooled by circulating hot air from the back of the cabinets through a plenum chamber into chillers at either end that blow the air through a cooling system using the building's chilled water."
The problem was with the chilled water supply. Furber said, "If the chilled water isn't actually chilled, the chiller fans are adding to the problem rather than helping solve it."
[7]
And so the temperatures began to rise inexorably. Without an automatic shutdown, the servers struggled on. Furber told us that he believed there was an automatic over-temperature shutdown on the individual SpiNNaker boards, and said, "This may have protected the SpiNNaker hardware from damage," but even with the hard-to-replace boards off, the network switches and power supplies remained powered up.
[8]808 lines of BBC BASIC and a dream: Arm architecture turns 40
[9]Intel's neuromorphic 'owl brain' swoops into Sandia labs
[10]SpiNNcloud Systems unveils Arm-based 'neuromorphic supercomputer'
[11]Tencent explores a future where HPC, quantum, cloud and edge have converged
The latter two component types suffered some damage, and without them, the SpiNNaker boards cannot all be tested, "so there may be more issues hidden behind the ones we know about."
Furber added, "We have had a few issues with the cooling system in the nine years that the machine has been operational, but these have not previously led to any damage." He reckoned that the long Easter weekend (in the UK, where Easter Friday and Easter Monday are both public holidays) might have contributed to the length of time it took to contain the temperature rise.
"We are looking into ways to fully automate the shutdown process in the future!"
As for the system's current state, Furber told us, "The machine is back up for internal users at around 80 percent of full capacity but still undergoing tests."
[12]
The good news is that the software is designed to work around partial hardware failures. The bad news is that replacing the failed parts will likely require further shutdowns. ®
Get our [13]Tech Resources
[1] https://www.theregister.com/2022/09/12/brain_chips_interview/
[2] https://www.theregister.com/2011/07/07/arm_project_spinnaker_super/
[3] https://www.theregister.com/2025/04/29/arm_40/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/aiinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aBoyHU8kYPLX6harNY1iTwAAABU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/aiinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aBoyHU8kYPLX6harNY1iTwAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/aiinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aBoyHU8kYPLX6harNY1iTwAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/aiinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aBoyHU8kYPLX6harNY1iTwAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2025/04/29/arm_40/
[9] https://www.theregister.com/2024/04/17/intel_hala_point_neuromorphic_owl/
[10] https://www.theregister.com/2024/05/08/spinncloud_systems_unveils_armbased_neuromorphic/
[11] https://www.theregister.com/2024/01/29/tencent_huawei_tech_predictions/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/aiinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aBoyHU8kYPLX6harNY1iTwAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Had the same idea ... but I hear it's fully booked serving as [1]hairpiece to the Orange Incubus ... lotsa wasted heat to dissipate from that overagitated empty skullcap in the oblong orifice.
[1] https://www.biolib.cz/IMG/GAL/BIG/390595.jpg
Or model a Portia genus spider? They are remarkable for their intelligence, and do that wih 100000 neurons. For comparsion mouse brain has about 70 million neurons.
https://en.wikipedia.org/wiki/Portia_(spider)#Intelligence
I had to refresh my memory of these little beasties. Remarkable things aren't they?!
And very modern.. according to Wikipedia they implement a "cryptic REST posture", which of course is de facto now when it comes to API security.
Auto-Slowdown/Shutdown Systems
... were not fully implemented here because HVAC failures, brownouts, and UPS/line-conditioning failures never happen in the real world.
Re: Auto-Slowdown/Shutdown Systems
Oh, be fair: it was two British Bank Holidays in a row, perfectly reasonable to expect cold weather, high winds and rain. Snow at Easter isn't unknown. They probably had emergency temperature control plans: an undergraduate poised to run in with a space-heater
Re: Auto-Slowdown/Shutdown Systems
Yeah, and lucky this didn't happen a week later during the (could've been) end of times UK [1]heatwave apocalypse and resultant [2]Iberian peninsula synch fail blackout armageddon ! Especially if this mouse-brain neuromorph had been a GPU-driven computational fusion energy megaspace heater of doom, rather than power-sipping spike machinery ...
Safe to say we've averted a major [3]" New-Zealand China Syndrome " on this one, with Manchester core meltdown shooting cataclysmic jets of devastation, all over Wellington NZ. Missed it by that much!
I hope they get the remaining 20% of the Muridae -brain beastmachine back up soon, but things could have been just so much worse imho ...
[1] https://www.lbc.co.uk/weather/spring-heatwave-break-uk-april-may-records/
[2] https://www.theregister.com/2025/04/28/nationwide_power_outages_knock_spain/
[3] https://en.wikipedia.org/wiki/Nuclear_meltdown#China_syndrome
Re: Auto-Slowdown/Shutdown Systems
Snow at Easter isn't unknown.
In the UK it is statistically more likely that at Christmas.
and nothing of value lost
seems to be the usual worthless Ai garbage
so even if it had burn't building to the ground nothing of value would have been lost.
Re: and nothing of value lost
Says more about your knowledge than the Spinnaker project.
FWIW - spinnaker is a simulation of biological neurons.
Perhaps they should have modelled Acomys russatus ? Able to operate in desert temperatures exceeding 42°C