News: 1694023271

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Scared of flying? Good news! Software glitches keep aircraft on the ground

(2023/09/06)


It has been a bad few days for anyone with a fear of flying or, perhaps more accurately, a fear of getting to an airport only to find that flying is the last thing that will happen.

United Airlines was the latest to suffer problems after a software issue resulted in all its aircraft being briefly held at their origin airports yesterday before a fix allowed flights to resume approximately an hour later.

A United Airlines spokesperson told The Register that "a software update caused a widespread slowdown in United's technology systems."

[1]

Although operations are normal now, the cause is still being investigated, but was not a cybersecurity issue, according to the airline.

[2]

[3]

United's woes have come as a preliminary report into a "major incident" filed with the UK's Civil Aviation Authority (CAA) [4]was published . The incident was a failure at National Air Traffic Systems (NATS) that sparked chaos over UK airspace on [5]August 28 .

More than 1,500 flights were cancelled or delayed due to the failure, which NATS blamed on "an extremely rare set of circumstances" where a flight plan had two identically named but separate waypoint markers outside of UK airspace.

[6]

These waypoints are used to describe the route an aircraft will take. For a flight going through UK airspace, the route includes waypoints outside the UK required for its onward journey. This, and other information, is bundled into a file defined by the ADEXP specification and uploaded into the systems.

However, in this instance, a flight plan included two waypoints, which, although 4,000 nautical miles apart, had the same name. A duplicated identifier, if you will.

The system searched for the UK airspace entry point in the file and then went hunting for the exit point. Not finding one – which is perfectly acceptable – it eventually came across the duplicate and a critical exception occurred.

[7]

Realizing it was broken, the primary system wrote a log to that effect and handed over duties to the secondary system, which did the same thing.

[8]UK air traffic woes caused by 'invalid flight plan data'

[9]UK flights disrupted by 'technical issue' with air traffic computer system

[10]ESA sees satellite-based air traffic monitoring on near horizon

[11]American and Southwest Airlines pilot candidate data exposed

With both systems now down, flight plans could not be processed, and manual intervention was required. Chaos ensued.

A source opined: "It's an astonishing software design screw-up combined with cost-saving silliness. Root cause was the flight plan interpretation module thingy being written to shut itself down instead of throwing the flight plan out for human analysis and carrying on."

We're sure the software engineers among our readership will also be scratching their heads at how such a problem could happen and how it was acceptable that the result was to fall into maintenance mode and write to a log rather than simply record and flag the failure and move onto the following file.

Assertions that the system had processed 15 million flight plans without issue over the five years preceding the incident will cut little mustard with passengers left stranded and airlines forced to scramble to deal with chaos not of their making.

One pilot, who preferred to remain anonymous, told The Register that he had been forced to de-plane several checked-in passengers 30 minutes before departure due to a missing crew member.

"It almost got me lynched," he said, "...not much fun."

Martin Rolfe, CEO of NATS, said: "Keeping the sky safe is what guides every action we take, and that was our priority during last week's incident. I would like to reiterate my apology for the effects it had on so many people, including our airline and airport customers. Incidents like this are extremely rare and we have put measures in place to ensure it does not happen again."

The Register approached NATS for comment on how the software was purchased and validated. We were told that those areas would form part of the onward investigation, as noted at the end of the report. ®

Get our [12]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZPj2hLGGH111dap-7RDPOAAAA9U&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZPj2hLGGH111dap-7RDPOAAAA9U&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZPj2hLGGH111dap-7RDPOAAAA9U&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://www.nats.aero/news/nats-report-into-air-traffic-control-incident-details-root-cause-and-solution-implemented/

[5] https://www.theregister.com/2023/08/28/uk_flights_disrupted/

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZPj2hLGGH111dap-7RDPOAAAA9U&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZPj2hLGGH111dap-7RDPOAAAA9U&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[8] https://www.theregister.com/2023/08/30/uk_air_traffic_woes_invalid_data/

[9] https://www.theregister.com/2023/08/28/uk_flights_disrupted/

[10] https://www.theregister.com/2023/07/26/esa_awards_contract_for_demo/

[11] https://www.theregister.com/2023/06/26/american_southwest_airline_breach/

[12] https://whitepapers.theregister.com/



I wonder if United's software update

DS999

Was made during business hours, or it was made overnight but caused reduced performance which meant their daily load they would have otherwise been able to handle was too much? If the latter kudos to them for figuring that out so quickly.

Hmmm...

Anonymous Coward

Assuming the official version of events is genuine, are we seriously expected to believe that the previous 15 million flight plans were entirely error-free?

Something doesn't add up here.

At least three systems are required

Inventor of the Marmite Laser

If one has two systems offering two different results, the simple and embarrassing question is which one should you believe.

At least with three systems one could hope to vote 2-oo-3.

Dual systems? Forget it.

Re: At least three systems are required

S4qFBxkFFg

"If one has two systems offering two different results, the simple and embarrassing question is which one should you believe."

neither

No problem.

Tron

Just compensate everyone - passengers, airlines, freight shippers and airports - and things will be fine.

NATS crashed.

Richard 12

The description given in public very obviously means it simply crashed on unexpected and/or bad data.

And then the backup went ahead and crashed as well, as expected. Same code, same assumptions, same data, same crash.

Worse, it clearly didn't create a useful log (or even core dump?)

If it had done then the staff would have been able to figure out which flight plan crashed the system, remove it from the automated queue and try again before the four hour "major disruption" deadline.

Or at least which small block of 10-100 flight plans contain the problem. Drop those out, continue.

Then they could manually process the funky flight plan(s), and finally set someone to work on figuring out why that flight plan crashed the software - without a nasty deadline hanging over them.

Asking someone to manually process ten flight plans in the knowledge that one of them made the automation fall over is also a very effective way of finding the flaw. Handing them 10,000 is a very effective way of making sure they ... don't.

Re: NATS crashed.

Boris the Cockroach

Missed something as simple as

Try

{

Process.nextFlightplan;

}

Catch

{

Deleteflightplan_from_list();

PrintF("Incorrect flightplan, please manually process");

}

Guess NATS will be adding a sanitation module to the code in the next week.....

Re: NATS crashed.

anothercynic

By the sounds of it in other media, the systems vendor was eventually called in because the engineers couldn't figure it out (but then again, if you're an engineer with no flight management experience where you might've spotted two identically-named waypoints with different coordinates, it's entirely possible to just go "WTF won't it work"). Apparently some changes have been applied since to stop this happening again (probably said sanitation/validation module).

I would not be surprised if this is something that was brought up during acceptance testing at some point, but given low priority because the chances of this happening were very low (1:15 million is low to those who don't realise how many flight plans are filed daily) or non-existent. Now that it's happened and it ground the whole thing to a halt, someone over at the vendor's probably dug this out of their "low priority, won't fix" queue and added a fix that was probably documented already.

This is yet another Swiss cheese "all holes align at the wrong time" moment. It happens. The system and the backup did what they were meant to - fail safe. That it then caused a massive backlog with manual re-entering/validation is more of a problem, especially when there are a lot of flights in a day that sort-of run by the skin of their teeth in terms of filing, getting clearance and flying out, and then repeating it at the other end.

There's no doubt that there'll be a historical RCA to see why this slipped through or wasn't approved/wasn't mitigated against at the time this was raised, and NATS will find itself sailing along with no major incident in the future when it (inevitably) happens again.

More

elsergiovolador

Expect more events like this due to Tory destruction of our IT services.

No expert in their right mind is going to work in these conditions we have now (poor rates, extremely high taxes, no employment rights).

Re: More

Anonymous Coward

And this is the mindless Tory bashing we get on this site. Even though it has nothing to do with Governments of any colour as it happens. People who are vehemently anti Tory are generally foaming at the mouth, this comment proves it. Best of luck with your Labour Government next year.

Re: More

Boris the Cockroach

Dont worry..... when labour get in all will be sweetness and light............ until they start making the stupid decisions led by a stupid idology

<<

Re: More

elsergiovolador

Can't be any worse than this lot of corrupt incompetents.

Re: More

anothercynic

As much as I detest the Tories, this has nothing to do with the Tories or "Tory destruction of our IT services".

This is more to do with something that happens less often than one thinks slipping through and gumming up the works. It happens. It's been fixed. NATS and its systems vendor have learned from it. It (most likely) won't happen again in the future.

Design for errors

Primus Secundus Tertius

Designing systems to cope with errors, i.e. raw user input, is difficult. To give constructive error messages you have to parse a range of inputs that include error cases, not just the perfect working case. So system design is a bigger and more expensive task, not always appreciated by techies let alone by beancounters.

Phil O'Sophical

Forget DDOS attacks, the next time some unfriendly state wants to cause chaos, they just need to file a corrupt flight plan. I can't help but feel there will be a lot of random junk being fired into NATS from Russia and China over the next months.

anothercynic

And chances are that none of that random junk will make any difference because that 'vector of attack' will have been closed/mitigated against.

"Does it worry you that you don't talk any kind of sense? "