NASA solar mission data recovering after server room flood fiasco
- Reference: 1738955709
- News link: https://www.theregister.co.uk/2025/02/07/nasa_solar_mission_data_recovering/
- Source link:
Data from two NASA solar missions is becoming [1]available again following an [2]outage that began in November 2024.
The affected missions are the [3]Solar Dynamics Observatory (SDO) , which was launched in 2010, and the [4]Interface Region Imaging Spectrograph (IRIS) , launched in 2013.
[5]
Both spacecraft have collected vast amounts of solar data over the years. However, new and historical data from the missions became unavailable in November after a broken pipe caused significant flooding of the building that houses the Joint Science Operations Center (JSOC). Data from two of SDO's instruments is processed at JSOC, as is data from IRIS.
[6]
[7]
The vehicles continued to collect and downlink data during the outage, but researchers were unable to access it.
Repairs at Stanford University's JSOC are ongoing. In a [8]January update , the SDO team noted, "Several pieces of electronics are delayed by the dreaded supply-chain issues we all thought were finally behind us."
[9]
An update this week from NASA confirmed that near real-time data from the two affected SDO imaging instruments has been resuming every 15 minutes since early January. Most other SDO data processing has also restarted, with more data becoming available as system repairs progress. Meanwhile, all IRIS data during the outage period is now available.
There is still work to do; two data partitions must be restored from tape, a process the JSOC team warned would take several months. As of the end of January, JSOC estimated that five percent of the data before December 23, 2023, wouldn't be available until the restoration was complete.
[10]NASA has just two Mars Sample Return mission lander options left
[11]NASA's Solar Dynamics Observatory datacenter flooded, offline until 2025
[12]A decade on from maiden flight, NASA's Orion is still waiting for its Moon moment
[13]NASA's solar sailing spacecraft is tumbling – but that's part of the plan
Data is coming back online, and alternative archive locations are available, but the prolonged outage was far from ideal. For an agency that prides itself on spacecraft redundancy, having several ground systems taken out by a break in a four-inch chilled water pipe in a server room is not a good look, even if no incoming data was lost during the outage.
The incident does, however, serve to highlight the importance of disaster recovery planning. According to JSOC, about 20 percent of the building's database and data servers were damaged in the incident, along with disk drives and UPS units. The floor was submerged in some places by "a few inches" of water, although most equipment was raised above floor level.
Something for the next budget meeting when some bean counter gripes about costs. ®
Get our [14]Tech Resources
[1] https://blogs.nasa.gov/sunspot/2025/02/05/data-being-restored-from-two-nasa-solar-missions/
[2] https://blogs.nasa.gov/sunspot/2024/12/03/data-from-2-nasa-solar-missions-temporarily-unavailable/
[3] https://sdo.gsfc.nasa.gov/
[4] https://iris.gsfc.nasa.gov/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z6aQlFs9Y8CBTdjUR5gf9gAAAUw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z6aQlFs9Y8CBTdjUR5gf9gAAAUw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z6aQlFs9Y8CBTdjUR5gf9gAAAUw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://sdoisgo.blogspot.com/2025/01/jsoc-update-and-schedule-of-events.html
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z6aQlFs9Y8CBTdjUR5gf9gAAAUw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2025/01/08/nasa_whittles_down_mars_sample/
[11] https://www.theregister.com/2024/12/03/nasa_sdo_outage/
[12] https://www.theregister.com/2024/12/11/nasa_orion_10/
[13] https://www.theregister.com/2024/09/06/nasa_solar_sail_acs3/
[14] https://whitepapers.theregister.com/
Beware Murphy.
Pipes are everywhere, hidden in walls and ceilings. They lurk, unobserved. And sometimes they strike. I was a curator at the time, in a museum devoted to electricity. One morning I came to work and found the ground-floor exhibit halls several inches deep in water. Museums are often put in historic buildings, and they are at times repaired, refurbished, or extended -- without full attention to the consequences. Some time before the museum moved in, the plumbing had been modernized -- well, some of the plumbing. Which meant that iron pipes were joined with copper pipes invisibly in the ceiling. The joint finally gave away to electrolytic corrosion and let loose the flood.
The scrambling to save artifacts was intense, even before the pipes had a chance for repair. But we saved the pipe junction and got an excellent artifact demonstrating electrolytic corrosion. Well-documented, too.
You are never completely safe. Murphy is too ingenious.
I do hope NASA survives the current fiasco but I fear it may not.
Given NASA is one of SpaceX's biggest clients, I'd be quite surprised if Elon closed it down.
"If you think business continuity is expensive, try having an outage."
Months to restore from tape?
Are they having to source some no longer produced hardware or something?
Sounds like they weren't quite expecting
that type of torrent from a fat pipe.