Known software issue grounds Ingenuity Mars copter as it attempted fourth flight
- Reference: 1619770506
- News link: https://www.theregister.co.uk/2021/04/30/ingenuity_fourth_flight_flops/
- Source link:
The delay was not unexpected nor prevents future flights. In fact, NASA [1]re-assured a waiting world that the helicopter is "safe and in good health."
[2]
The reason for the failure is also known: a [3]software bug that results in a watchdog timer expiration prior to mode transition. And if Ingenuity doesn't make it into flight mode, it doesn't fly.
NASA identified the bug two weeks ago and found two ways to solve the problem. One was to change the timing of the transition by adjusting the command sequence from Earth. The other was to modify and reinstall Ingenuity's existing flight control software.
[4]
NASA chose the first option as it considers it the less disruptive option. The rover’s flight control software has been stable and healthy for almost two years, and why mess with a good thing there?
NASA’s tested its preferred strategy on both planets and knows it creates a 15 percent chance Ingenuity won't transition and that some flights may fail.
NASA’s getting really good at this flying a helicopter on Mars thing [5]DIDN'T AGE WELL
As a backup plan, NASA's boffins have uploaded a new version of Ingenuity’s flight software to the Perseverance rover. Should this option become necessary, the helicopter will be out of commission for a few days while engineers load and test the new software, then attempt flight again.
NASA remains cheery about the issue. In a blog post, the agency [6]said :
We’re doing everything we can to make it a success, but we also know that we may have to scrub and try again. In engineering, there is always uncertainty, but this is what makes working on advanced technology so exciting and rewarding.
[7]
The fourth flight is now scheduled for Friday 30 April at 10:46 a.m. EDT with first data expected to land at JPL at 1:39 pm EDT. On Friday at 11:30 am EDT, NASA will discuss [8]what’s in store for the helicopter. ®
Get our [9]Tech Resources
[1] https://mars.nasa.gov/technology/helicopter/status/296/mars-helicopters-flight-four-rescheduled/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YIvVQsAo7E0dhl2xv2hydQAAAJQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.theregister.com/2021/04/13/ingenuity_mars_software_upgrade_needed/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YIvVQsAo7E0dhl2xv2hydQAAAJQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/04/26/nasas_ingenuity_mars_helicopter_third_flight/
[6] https://mars.nasa.gov/technology/helicopter/status/293/why-we-choose-to-try-our-first-helicopter-flight-on-monday/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/science&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YIvVQsAo7E0dhl2xv2hydQAAAJQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.nasa.gov/press-release/nasa-sets-briefing-to-discuss-ingenuity-mars-helicopter-next-steps
[9] https://whitepapers.theregister.com/
Re: Ace PFY skills
Works every time...
[1]IT Crowd - Have you tried turning it off and back on again?
[1] https://youtu.be/nn2FB1P_Mn8
Did....
....it get a new, forced Windows update*
I'll get my coat.
*I know its not running Windows.
In my fanciful imagination, I picture these guys working on this and not feeling that they are working at all. Just playing with some of the best kit in the world. And when I say "playing" I don't mean like toying.
A quite common problem
To ensure a system stays alive, a watchdog timer periodically resets it unless interrupted by valid code. It should be timed to prevent it interfering with functionality, but as systems get more complicated this becomes quite a hard nut to crack. In such a case as this, the watchdog appears to time out (and reset the system) before a valid process has completed. Reducing the time the valid process takes is indeed the quick fix, but it appears in this case that they're having to wait a variable time for some real time event. A difficult enough problem on Terra Firma, but at that distance I take my hat off to NASA.
Re: A quite common problem
> Reducing the time the valid process takes is indeed the quick fix, but it appears in this case that they're having to wait a variable time for some real time event
Just speculation on my part but maybe they tried to put too much into one step - for good reasons - and are now splitting it into two steps so that the watchdog can be safely reset in between?
That split probably has knock-on effects so a bit of a pain when you want to do it in a hurry.
Have you tried switching it off and on again?
> The rover’s flight control software has been stable and healthy for almost two years, and why mess with a good thing there?
A good question.
If the only (small) issue is that 15% of the time the helicopter won't initialise into flight mode, it doesn't sound too risky to simply wait a while and try it again.
Isn't that the answer that software support gives, 90% of the time?
Re: Have you tried switching it off and on again?
I only hope that the transition to "landing" can't get similarly borked.
Re: Have you tried switching it off and on again?
Hopefully also they don't hit the end of the 30-day flight window before they fix/test it, otherwise Perseverance may be off on the rest of its mission.
I know it's not part of the plan, but it would be cool if they could put Ingenuity into a "follow me" mode where it keeps (say) 20m away in a series of hops from Perseverance until it finally breaks.
I suppose that the risk is that something breaks and it then hits the rover and jams/breaks something.
What a cool job to have!!
Re: Have you tried switching it off and on again?
Depends on how much power is used before initialisation fails - it may have to go into a recharge state for a while before retry.
I find it surprising that a 15% chance of invalid faults being flagged under no-fault conditions wasn't discovered and fixed during pre-flight testing. After all the basic requirement is that it flies and this fault report causes it not to fly. When something is sitting on Mars you don't really want to be sorting out whether a fault flag is real or not before you try again as the result could terminate the mission ...
Remote server, extreme edition
Managing remote computers is a pretty common thing in today's IT world, whether it's some industrial kit or a stack of servers running in a data center half-way across the country or globe.
Those happily have the option of a call-of-shame to a colleague or support person whenever the gear in question does not come back online in a healthy state after a reboot. Here it sounds like the nearby rover is the closest thing that JPL's engineers have to a 'call support to push the power button on the server' fallback :)
Re: Remote server, extreme edition
Yea, but I still have that moment of panic.
Even though I've done it hundreds of times, I know it will work and how long it takes to boot. But, I guess, I'm not the only one who suffers 'time dilation' during the wait so it never comes up quite when expected. The alternative of looking at a clock seems to make it even worse.
Boy, I wonder how many of Ingenuity's engineers have a pacemaker ...
Ace PFY skills
So, am I getting this right, turn it off and back on and it should be fine?