NASA's Solar Dynamics Observatory datacenter flooded, offline until 2025
- Reference: 1733201953
- News link: https://www.theregister.co.uk/2024/12/03/nasa_sdo_outage/
- Source link:
The pipe popped last Tuesday at Stanford's Joint Science Operations Center, which houses infrastructure used to handle data from the Solar Dynamics Observatory (SDO) and the Region Imaging Spectrograph (IRIS). The Center (JSOC) handles data from two of SDO's three scientific instruments: the Helioseismic and Magnetic Imager (HMI) instrument and Atmospheric Imaging Array (AIA).
Several inches of water pooled in the JSOC server room. By the next day it had been drained, but equipment needed to be dried. Ironically, hanging it in the sunshine was not an option.
[1]
"An initial assessment is that the water affected many systems but did not provide a time for return to service," [2]explained a mission blog post.
[3]
[4]
JSOC [5]referred to the damage as "severe" and a complete recovery is not expected until next year. No estimate was provided on how much rice might be required.
Japan switches on new astronomical supercomputer While the SDO struggles, Japan's Center for Computational Astrophysics on Monday inaugurated a supercomputer named ATERUI III that packs 1.99 petaflops of power.
The machine comprises two linked systems. System "M" runs 23,296 Intel Xeon Max 948 cores spread across 208 nodes and packs 26.6TB of memory. System "P" uses the Xeon Platinum 8480+ and 40.96TB of memory.
Both of the processors used are Sapphire Rapids parts that Intel launched in 2023.The supercomputer will be used for simulations that assist theoretical astronomers to do things like modeling how stars are born. – Simon Sharwood
NASA and the University haven't detailed what got wet, but the SDO in its entirety [6]delivers around 42TB of data each month. That number that may seem trivial now, but it was more significant when the observatory launched in 2010 and could require plenty of rack space if kit hasn't been upgraded.
[7]China launches first next-gen Long March 12 rocket, christens private spaceport
[8]Alibaba Cloud struggles after blaze at Digital Realty Singapore datacenter
[9]China starts building world's largest fully steerable radio telescope
[10]ServiceNow root certificate blunder leaves users high and dry
"Science data processing for HMI, AIA, and IRIS will be down for an extended length of time, as will access to the archived data at JSOC," it added.
Although data is unavailable, it is not lost. It is still being collected – just not processed. HMI data, for example is currently being stored in New Mexico. But in the meantime, realtime data needed for space weather, space environment data products, and mission planning is inaccessible.
As IRIS makes its near-realtime data available [11]online , data from before November 22 – four days before the pipe burst – is available. Incomplete AIA and HMI data can also be accessed from various sources.
[12]
The SDO is a space-based observatory that studies the Sun, focusing on its magnetic field, activity cycles, and impacts on the solar system. It is particularly focused on understanding the connection between the Sun and Earth.
IRIS launched in 2013, and studies the flow of plasma into the corona and heliospere of our local star. It is also intended to advance the understanding of the Sun-Earth relationship. ®
Get our [13]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z07k2DK4FuHbq-6fef67kQAAAMA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://sdoisgo.blogspot.com/2024/11/outage-at-stanford-jsoc-is-offline.html
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z07k2DK4FuHbq-6fef67kQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z07k2DK4FuHbq-6fef67kQAAAMA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://solarweb1.stanford.edu/JSOC_Emergency_Resources.html
[6] https://repository.gatech.edu/server/api/core/bitstreams/85068be9-5374-4b5b-a705-07550ca0d0f5/content
[7] https://www.theregister.com/2024/12/02/china_long_march_12_launch/
[8] https://www.theregister.com/2024/09/10/digtal_realty_singapore_datacenter_fire/
[9] https://www.theregister.com/2024/11/28/jilin_120_meter_radio_telescope/
[10] https://www.theregister.com/2024/09/23/servicenow_root_certificate_outage/
[11] https://iris.lmsal.com/search/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z07k2DK4FuHbq-6fef67kQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Re: How does a flood happen in a datacenter?
4" is a big pipe.
The only pipes that size in an average home are for sewage and wastewater - or if you live in a highrise, it's roughly the size of the risers used for firefighting.
In other words, places where you need very high flow rates.
A pressurised cold water line that size is going to spray a lot of water very quickly.
Did we just go around with this recently?
There should be no liquid lines other than for cooling computers routed through a data center. There should be space between the lowest electrical gear in the rack and the floor with a raised floor as being the most optimal. The room should have drainage so any water that gets in can go right out again. Water used on a fire, a damaged roof in a rain storm and other calamities can't always be identified so you build in a catch-all since liquids and computer systems don't mix well.
and they want a minimum of a Master's degree for their staff. Pfftttt.
It does say that it's a chilled line so maybe it was for cooling the computers. Still not an excuse to not have something in place in case of a break. If it had nothing to do with the data center... poor design rides again with the other horsemen.
Re: Did we just go around with this recently?
Every server room I have been designed had:
* water sensors at the 6" drainage outlet
* solid concrete roof overhead, with the server room solid walls up to the concrete support beams for that roof and no suspended ceiling (minimizing chance of water ingress from above and to the side)
* suspended floor
* chilled water feed under the suspended floor with only the smaller uprisers to the individual racks above that - yes a split in a 4" might blow the floor tiles up, but it is another 'early' warning sign
* other than doors, no holes in the walls above suspended floor level
* all necessary holes (electrical supply and comms) in the wall near the drainage outlet, and just above its top - will provide extra drainage in extreme circumstances, but likely to cause secondary damage.
I've never had a major leak to one of my server rooms to test all this, but I think I've got most opportunities for failure covered.
Of course, if you are not able to start from scratch then most of this is impossible and I admire your efforts.
Re: Did we just go around with this recently?
If NASA needs another billion for SLS congress will have a cheque ready and waiting. NASA has needed funds for basic building maintenance / replacement for decades.
Re: Did we just go around with this recently?
Knowing NASA, it's highly likely that this Data centre was shoe-horned into an existing space (the basement, because, let's face it, it's always in the basement, isnt it?), and done on an absolute shoestring budget because Congress only provides sufficient funding for cool sounding things and pork barrels. Funding for a state of the art data centre for processing the data provided by two already launched satellites, not a chance they got even 50% of what they actually asked for...
We supply servers and overhead water cooling equipment.
Personally, if I had to run a water pipe over a bunch of server racks I would put a second, larger backup pipe over the smaller pipe. But that's just me. I don't design data centers.
Just hope we don't get Carrington 2 before it's up and running again.
NASA has just one datacenter?
Why can't the data be processed in other data centers? For example where it's being stored currently?
How does a flood happen in a datacenter?
Surely there are some sort of water sensors installed that will cut power and shut off the water well before "several inches of water" can pool up? How could they miss that in a place that has a 4" water line in it?
Since the trend is away from running power & network under the floor and running them overhead instead, maybe they can continue building raised floors only using that space for flood management, until they can solve the problem of shutting off water before the countless number of gallons that would have to be spilled to cover an entire datacenter several inches deep.