Hubble, Hubble, toil and trouble: NASA pores over moth-eaten manuals ahead of switch to backup hardware
- Reference: 1626266711
- News link: https://www.theregister.co.uk/2021/07/14/nasa_hubble_formal_review_backup_hardware/
- Source link:
The formal review comes more than a month after a [2]problem with a payload computer aboard the veteran observatory sent the telescope into [3]safe mode and engineers scurrying for contingency plans.
Attempts to restart the computer failed, and a [4]switch to a backup of the 1980s-era hardware resulted in the [5]same memory error . Engineers cast their net wider as time wore on, considering potential issues with a [6]power regulator as a possible culprit as well as the Command Unit/Science Data Formatter (CU/SDF), which sends and formats commands and data.
[7]
The hardware is highly redundant, but switching to backups at this level carries its own risks: "To switch to the backup CU/SDF or power regulator, several other hardware boxes on the spacecraft must also be switched due to the way they are connected to the SI C&DH [Science Instrument Command and Data Handling] unit," NASA posted at the end of June.
[8]
[9]
It's not entirely new ground for the team. A similar switch was done in 2008 after a CU/SDF module failed. However, back then a Space Shuttle servicing mission was on the cards for 2009. Today's engineers have no such luxury.
[10]Wanna feel old? It is 10 years since the Space Shuttle left the launchpad for the last time
[11]The James Webb Space Telescope, a project dating back to the late 1900s, may launch this very century
[12]It's about time! NASA's orbital atomic clock a boon for deep space navigation – if they can get it working for long enough
[13]Lego's Space Shuttle Discovery: No trouble with Hubble, but the stickers will drive a grown man to insanity
[14]‘Radiation upset’ confused computers, caused false alarm on International Space Station
NASA has therefore spent much of July testing and reviewing procedures on a high-fidelity simulator of the hardware. Tests were completed last week, and a review aimed at minimising the risks associated with the procedures went ahead earlier this week.
A formal review into all the operations related to the switch to backup hardware has now been completed and there is every chance that engineers will have a go at the procedure later this week – perhaps a bit risky since the investigation into what actually befell the payload computer is not complete.
The Register asked NASA for a look at the reviewed procedures and will update should the agency respond. ®
Get our [15]Tech Resources
[1] https://www.nasa.gov/feature/goddard/2021/operations-underway-to-restore-payload-computer-on-nasas-hubble-space-telescope
[2] https://www.theregister.com/2021/06/16/nasa_hubble_outage/
[3] https://www.theregister.com/2021/06/21/hubble_memory_fail/
[4] https://www.theregister.com/2021/06/24/hubble_space_computer/
[5] https://www.theregister.com/2021/06/28/hubble_backup_borked/
[6] https://www.theregister.com/2021/06/30/hubble_repair/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YO8KHOoyawq3W9bL4EiFaAAAAAU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YO8KHOoyawq3W9bL4EiFaAAAAAU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YO8KHOoyawq3W9bL4EiFaAAAAAU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2021/07/09/sts-135/
[11] https://www.theregister.com/2021/07/08/james_webb_launch/
[12] https://www.theregister.com/2021/07/01/nasa_atomic_space_clock/
[13] https://www.theregister.com/2021/04/21/lego_space_shuttle_discovery/
[14] https://www.theregister.com/2021/03/26/iss_radiation_false_alarm/
[15] https://whitepapers.theregister.com/
Re: Step 1
I would be extremely nervous about rebooting a computer remotely when there is no chance of a visit to the computer to turn it back on if the reboot command fails.
Re: Step 1
Only $134 for a single ILO licence. I bet those boffins are kicking themselves now...
Re: Step 1
Nervous ? I am not .. ill push the button myself if that's all that's needed . Hell , it's fried as it is .. What could possibly go wrong ? 8)
This has the potential to be all too familiar, only turned up to 11 – I'm not surprised they're carefully checking all possible scenarios. I've more than once found myself pressing Enter in an SSH session to a router only to realise my mistake a fraction of a second later as the connection is "unexpectedly closed" (yes, it is accepted that I'm an idiot). Difficult enough when it's three hundred miles away, even more so if the direction of those miles is up .
Most of us have done it at some point, perhaps a new firewall rule that blocked SSH, or a reboot command that failed to complete as a remote machine hung on failing hardware and crap firmware that never times-out, etc, etc.
That amp it up to 11 !
As I have said to people I work with, find an engineer who has never done any of the above and we have found either a liar or they have only worked in the field for a couple of weeks.
All of us have done this and the honest ones admit to doing it and will explain how they recovered from the problem.
I met an instructor who said they liked students who made lots of mistakes as the students learned a heck of a lot more from undoing the mistakes than those that got things right all the time.
It sounds corny, by mistakes are great learning experiences. Just don't make them a habit ;)
Experience and Luck
You start with an empty bag of experience, and a full bag of luck.
The trick is, to make sure you fill the former before you empty the latter.
Re: Experience and Luck
There's also a need for intuition - which is not guesswork, but in the words of the late Robt. Ornstein "arrival at a correct answer without recourse to inference". It's based on accumulated experience but experience alone is only a part of it.
For example, in the days of dumb terminals, I was thrown a dead one to fix. A huge board covered with TTL chips, no manual and no circuit diagram. It suddenly occurred to me that TTL is tough stuff, so I raided the digital chip drawers in the electronics shop and, one at a time, I sat a replacement chip on each of the onboard devices and turned on. Half a dozen chip tests down the line the terminal sprang to life. So I turned off, replaced the said chip and everything was fine. I haven't a clue why that approach occurred to me - it's distinctly oddball, but it succeeded. That's intuition, and without it the terminal would have been scrap.
Had this during training once. All the machines had been left from the previous class and only a few bits were "reset". During the training I had something fail and would consistently fail. Most of the class left and there was the trainer, myself and one other person left. It took us around 2 hours to find and fix the problem, but it did reoccur a few times over the next few years with that app.
Without that course I would never have known
The trainer said he preferred to leave the machines this way instead of reimaging as a freshly imaged machine would not have these problems.....
It sounds corny, by mistakes are great learning experiences.
A quote from James Davis Nicoll(*) that turned up on tor.com a few days ago:
Nothing teaches one not to try to stamp out burning thermite quite like real-life experience.
(*) If you've never heard of him, the man's a first class accident magnet with the resulting collection of tales to match.
Sounds Like...
Another repair mission is needed. I'd thinking a Dragon on Falcon Heavy could easily lift the necessary hardware and personnel to Hubble. The issue is reentry: Can Dragon take the higher reentry speeds?
If it can, then bring along new stabilization gyro's and a shortened Canada Arm to grab the satellite. Break out the shuttle space suites for on-orbit repairs while your at it.
Re: Sounds Like...
I would be thinking first of a small drone-like unit that can be sent up to visually inspect dear Hubble for F.O.D. (Foreign Object Damage)
Re: Sounds Like...
A repair mission would be far preferable to the "NASA has a plan to send up a de-orbiting module that attaches to Hubble and drives it into orbital decay mode to allow it to be brought down into the ocean or on unpopulated land" mission that El-Reg reported a few years ago...
Re: Sounds Like...
NASA fitted an attachment point at the last Shuttle servicing mission - but there is no plan to use it.
The current plan is to let it reenter and not worry because it can't hit the continental USA (except for bits of Florida that nobody cares about)
Re: Sounds Like...
There are definitely bits of Florida I would prefer to be impacted over others. Can no one think of the Manatees!?
Re: Sounds Like...
If only it were that simple
Re: Sounds Like...
Where's Elon's space Roadster these days?
Would......
closing and reopening all the windows work?
It's getting difficult to keep typing with my fingers crossed. let's hope it works OK. If it does then it's time to reflect on how much more life it might have and prioritise observations.
If it all fails to come back up then perhaps they could turn it into one of those IOT doorbell systems for alien visitors to use? Just a thought.
Step 1
“Turn it off and back on”
(Presumably they’ve already tried that…on to step 2 then)