Hibernating instrument on Hubble roused as engineers ponder message problem
- Reference: 1636114507
- News link: https://www.theregister.co.uk/2021/11/05/hubble_nicmos/
- Source link:
The good news is that the team might be close to isolating the problem. The bad news is that the finger of blame is hovering over a bit of hardware in the Science Instrument Command and Data Handling Unit. "Specifically," said NASA, "the team is analyzing the circuitry of the Control Unit, which generates synchronization messages and passes them onto the instruments."
NASA is working to resolve an issue that suspended science operations on Hubble.
The team is looking at taking steps to recover the Advanced Camera for Surveys instrument from safe mode to start collecting science with that instrument next week. [2]https://t.co/GGtwDYuIYg — Hubble (@NASAHubble) [3]November 4, 2021
Why bad news? Because if there is a hardware fault, there is no way to replace the failed component. Engineers will have to work around it using software, possibly with an update to check for the lost messages and compensate for them, thus avoiding other protection software throwing a hissy fit and sending the science instruments into safe mode.
We've all encountered that one weird component in our projects that occasionally does something strange and we end up having to code a bodge to deal with its funny little ways.
[4]Of course we've tried turning it off and on again: Yeah, Hubble telescope still not working
[5]31-year-old piece of hardware not working very well: Hubble telescope back in safe mode over 'synchronization issues'
[6]James Webb Space Telescope completes its voyage to French Guiana
[7]Hubble in another first: Water vapor spotted in atmosphere of Jupiter’s Ganymede
The problem is knowing just how often the problem occurs, and to that end the team has resurrected the Near Infrared Camera and Multi Object Spectrometer (NICMOS) instrument, which has been inactive for more than a decade. Some of the functions of NICMOS were replaced by the Wide Field Camera 3 (WFC), installed during the final Space Shuttle servicing mission, STS-125 in 2009.
Although every component on the veteran telescope is precious, bringing up NICMOS represented less of a risk than using an active instrument. It was recovered on 1 November, and no additional synchronisation messages have been lost.
The Advanced Camera for Surveys (ACS) is next in line, but the team is proceeding very cautiously for fear of "additional stresses" on the hardware. "ACS was selected as the first instrument to recover as it faces the fewest complications should a lost message occur," said NASA.
[8]
All eyes will be on those synchronisation messages and the frequency of the problem. Once the scale and source of the issue is understood, any required software changes can be sketched out and a plan made for resuming the collection of science data from Hubble's remaining instruments. ®
Get our [9]Tech Resources
[1] https://www.nasa.gov/feature/goddard/2021/hubble-instruments-remain-in-safe-mode-nasa-team-investigating
[2] https://t.co/GGtwDYuIYg
[3] https://twitter.com/NASAHubble/status/1456380781761384453?ref_src=twsrc%5Etfw
[4] https://www.theregister.com/2021/11/02/hubble_trouble/
[5] https://www.theregister.com/2021/10/26/hubble_safe_mode_again/
[6] https://www.theregister.com/2021/10/13/jwst_french_guiana/
[7] https://www.theregister.com/2021/07/27/ganymede_water_hubble/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YYlXyNzLv62WyhNRjUrD8QAAAIA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[9] https://whitepapers.theregister.com/
Re: Hardware Outsourcing
If it turns out to be a hardware fault, NASA could send up a small robotic maintenance craft equipped with a high precision hammer.
Re: Hardware Outsourcing
Perfect job for the X37B
Re: Hardware Outsourcing
I propose to call it "Maxwell"
Re: Hardware Outsourcing
I just happen to own a 7000 dollar hammer. I stole it from the Air Force in the 90's.
I see a service contract in my future.
Re: Hardware Outsourcing
HST Team raise ServiceNow problem ticket.
Outsourced Support Team immediately ask for more information, like "what is HST? what does it do?" thereby stopping the clock on the ticket SLA. That goes on for a bit, then OST passes problem onto some other team who have absolutely nothing to do with HST, still trying anything and everything possible to not do any investigation of the problem. Then eventually back to the HST team just before the system automatically closes the ticket as it has gone beyond a certain time frame.
Joke icon. But the reality in a company I have worked for
outdated hardware
it probably downloaded an auto software update.
Re: outdated hardware
Since it is in orbit, shouldn't that be upload rather, since it is pulling the data up to it?
And download is when it transmits the data to Earth.
I will leave for the pub now.
Re: outdated hardware
expired certificate??
Re: an auto software update.
I agree, my android is having problems too.
"The trouble with programmers is that you can never tell what a programmer is doing until its too late." - Seymour Cray
weird components doing funny things
" We've all encountered that one weird component in our projects that occasionally does something strange and we end up having to code a bodge to deal with its funny little ways. "
Too true. On an instrumentation project decades ago, we found that the hardware 'full' flag of a dual port memory didn't reliably signal memory full. We'd intended this to allow user substitution of devices with different capacities. To work round it we had to implement a user defined initialisation parameter in the software of the data sender so the system knew what size memory had been installed.
Re: weird components doing funny things
Mine in my first job as a schoolkid was a printer that would sometime ignore linefeed
So I set it to 1/64 of a line (the minimum) and always sent 64 line feeds - if it missed one you would need a micrometer to tell.
Still proud of that
Re: weird components doing funny things
Not one of those dodgy ICL printers by any chance ?
I vaguely remember some of them had some sort of paper tape loop - though plastic to stop it from being worn away - and they couldn’t do line feeds either.
Re: weird components doing funny things
ICL opd (the last vestige of the Sinclair QL) running nationwide Bingo games
Use the backup
When Nasa built the Hubble telescope they actually commissioned two identical ones, because when you're dealing with taxpayer money, why build one when you can build two at double the cost?
The second one has been in a museum since then. Just dust it off and strap it to a rocket.
There's an interesting story about that: the one in the museum apparently is better than the one in space because they were built by different contractors.
Re: Use the backup
I wonder if it has the same mirror error though?
Re: Use the backup
i SERIOUSLY doubt the museum "look a like" actually even HAS a mirror inside it. This is simply a "mock up" for people to VIEW in a museum, not like it actually has working instrumentation inside it.
Re: Use the backup
Building a single copy of all your unique instruments right from the get go is a seriously stupid idea though. The actual hardware isn't the big cost (apart from the main mirror, that's a very expensive bit of hardware). For all the other stuff you make 2 and start assembling at least the more basic stuff, pick the best and then integrate all the best choices into a single vehicle for launch to orbit. I highly doubt the museum piece would be fully functional and orbit capable hardware. At the very least it'll be missing a mirror (they only made one. Incorrectly as it turns out)
Re: Use the backup
There was a 'backup' mirror blank made but never polished.
It's in the Smithsonian (or it was last time I saw it)
This is normal practice for large or expensive mirrors - sometimes things happen in the grinding or polishing, a flaw is discovered or an accident takes a chunk out.
Zeiss made a spare 8m mirror blank for the VLT (made 5 used 4) that they were trying to sell off to anybody who needed an 8m mirror - don't know what happened to that, probably got melted down.
Re: Use the backup
Knowing Zeiss, they’ve probably forgotten where it is. A friend has some incredibly good, but definitely not cheap, Zeiss optical equipment.
As they couldn’t come and service it in the pandemic, they sent a 30k machine out to use until the original was serviced, around September last year. We’re still falling over the spare now…
Re: Use the backup
"There was a 'backup' mirror blank made but never polished."
IIRC Orrefors made a very large cut glass bowl for an international exhibition. It was finished near the deadline - and then someone dropped it. They had to make a new one from scratch.
Re: Use the backup
I knew there wee two mirrors made and a bit of Googling comes up with this:
https://airandspace.si.edu/collection-objects/hubble-space-telescope-backup-mirror/nasm_A20010288000
It was made but never silvered.
Re: Use the backup - THEY ARE
First, there is no need for a new mirror.
Second, all the computer kit, which is where the problem lies, has backups.
Third, they are already running on the backup kit since they brought the Hubble back to operational status in June when it encountered a similar issue.
Re: Use the backup
NSA built few for them self. That one in orbit is pointed at Earth and has a resolution about 30cm to 1 meter. How good it is remains a guess work because its classified on the technical specifications.
NSA donated the two remaining to NASA in 2012.
https://en.wikipedia.org/wiki/2012_National_Reconnaissance_Office_space_telescope_donation_to_NASA
Re: Use the backup
Every time we do this.
(1) The NRO carcasses are not Hubbles: their focal length is much shorter. (2) It is not clear if reusing one is even worth it for applications where wide field is suitable: I believe that NGRST is going to use one of the NRO carcasses, but there was (is?) much discussion over whether it would be cheaper to start from scratch. (3) Mirror is not this huge expense: consider VLT: this has 4 8.2m mirrors and 4 1.8m mirrors, and cost less in total than cost of a single shuttle flight to service HST.
And finally (4) It is not clear to me that a new HST is even interesting now: suspect that for wavelengths it covers ground-based telescopes are now better. Certainly think if asked to choose between 10 VLTs and one new vidible-light telescope in space most astronomers would pick 10 VLTs. Certainly also is the case that there are telescopes that need to be in space – JWST for instance will do things that could not be done under atmosphere, but not clear that visible-light super-HST is interesting. But I am theorist: will defer to practicing astronomers on this (just not to space romantics).
Re: Use the backup
Well I asked someone about the visible-light-telescope-in-space question. One advantage is that adaptive optics typically give good image over quite small part of FOV, while in space you get good image over whole FOV. So that is one reason for it.
Re: Use the backup
When HST was delayed by Challenger disaster they had to keep telescope in storage for about four years. Keeping it in flight condition cost about $6 million a month, in late 1980s dollars. If this second, incomplete telescope could have flown in 1990 (it could not), then it has been in a dirty, uncontrolled environment full of horrid oxygen and other corrosive gases for 30 yeas since then: it certainly can not now,
Re: Use the backup
The first rule of government spending, why build one when you can have two at twice the price!
- S.R. Hadden.
Re: Use the backup
Corollary rule of government spending: build two, pay thrice the price!
- Me, probably
We've all encountered that one weird component in our projects that occasionally does something strange and we end up having to code a bodge to deal with its funny little ways
It's called Visual Studio but I stopped finding it's time wasting quirks funny a long time ago.
We've all encountered that one weird component in our projects that occasionally does something strange and we end up having to code a bodge to deal with its funny little ways
It's called Visual Studio but I stopped finding it's time wasting quirks funny a long time ago.
They're not called "weird components" they're called "users".
Perspective
Geez...and we complain about trying to troubleshoot $RELATIVE's computer over the phone...
Re: Perspective
But in all fairness, Hubble probably said something along the lines of "Error: ADBEAx000"
Which is thousands of times more helpful than $RELATIVE:
There's an error in my pc
What error?
An error. I don't know
An error where?
In internet
What does the error say?
I closed it
Re: Perspective
I can't remember the machine, but I remember typing in a 2 or 3 page magazine listing. This was early '80s or so.
I remember typing "run" and all it did was say "syntax error"
No line number, nothing else. I think that machine went in the garbage soon after.
Re: Perspective
Yup we even get our own staff reporting errors to us by sending us a screenshot of the dialog box. Every single freakin' time our first reply is 'Can you send us the log files please?
Re: Perspective
Since they are all now using W10 or Apple - I can figuratively shrug my shoulders and honestly say "not in my competence". Ahh - bliss.
Malfunction
nEEd inPuT
Wonder if Hubble still uses 486's ?
"...A software workaround for an iffy component looms. Sound familiar?"
Of course it sounds familiar. It not only sounds familiar, this is the mind-set of anyone--- and their reptilian-IQ managers ---who is (laughingly) referred to as an "engineer", nowadays; to wit:
"Software can fix any hardware problem."
Just ask Boeing about its illustrious use of this "fix", to create---with software ---a completely brand-new aircraft, the 737-MAX, out of a completely old airframe ...which was---at the time it was designed---designed the old-fashioned way of designing aircraft: by real engineers, using real hardware, created for the design of real, safe aircraft .
Now you know:
Software can fix any hardware problem.
Re: "...A software workaround for an iffy component looms. Sound familiar?"
Completely agree, especially the reference to Boeing. Remember software operates on GIGO (garbage in / garbage out) principle.
Re: "...A software workaround for an iffy component looms. Sound familiar?"
It could have worked fine if they had used 3 sensors and implemented a "voting" system to reject spurious data. The fault wasn't necessarily "fixing it in software". The fault was in doing so badly, ignoring the added hardware requirements and not providing training. Oh, and lying to aviation authorities about the nature and effect of the new systems of course. That's kind of a big issue all by itself even if it hadn't gone wrong.
Airbus has very similar systems in all their aircraft, the difference being that they implemented them properly and pilots are trained in their existence, effects and failure modes.
Can SpaceX send up a couple of people to fix it? Sounds well within their technical capabilities and they'd get lots of nice publicity.
Lift capability, yes. Crew facilities for space walking, nope.
X37B with remote control - maybe. But if it’s electronics giving up ghost, that’s some awkward cct board replacements to do without a human in the loop.
Hubble is approaching the point of decommissioning while it can still be commanded to do so. No pressure on JWST or anything to take over.
As has been said elsewhere building a bunch of expendable orbital scopes without the cost of the shuttle programme would have been cheaper. But politics versus efficiency is a long standing known issue. And A scope is better than no scope at all. Nobody “Had” to build Hubble.
"We've all encountered that one weird component in our projects that occasionally does something strange and we end up having to code a bodge to deal with its funny little ways."
It has always surprised me how many hardware designers don't allow for metastability*** in logic components when interfacing with asynchronous inputs. It produces glitches that fulfil Murphy's Law with the Sod's Law rider "..at the worst possible moment".
**Metastability is what happens to you when the doorbell chimes - and then the phone rings at the other end of the hall. You pause for an indeterminate moment - not sure which one to service. In logic gates there are input set-up timing constraints for reliable propagation times.
metastable donkeys
Place one donkey exactly in the middle between two identical stacks of hay.
The donkey will starve to death trying to decide which stack to eat first.
Just Shoot It
And put it out of it's misery.
It's like giving a 97 year old a heart transplant. The new heart saves them, but the operation kills them in the long run.
Hardware Outsourcing
Just imagine if NASA had outsourced hardware support for the Hubble and some wee poor soul draws the short straw. "I've to go where ?"