UK air traffic woes caused by 'invalid flight plan data'
- Reference: 1693402329
- News link: https://www.theregister.co.uk/2023/08/30/uk_air_traffic_woes_invalid_data/
- Source link:
The problem that caused [1]thousands of flight cancellations and delays across the UK was unusual and "should not happen again," NATS CEO Martin Rolfe [2]told the BBC on Wednesday.
But the precise nature of that problem has yet to be fully explained, and may not come to light before investigators deliver a preliminary report to the Secretary of State for Transport next Monday.
[3]
In a [4]statement posted on Tuesday, NATS disclosed that early investigations pointed to flight plan data that was received from an airline and which could not be processed for some reason. There are no indications that this was caused by a cyber-attack, the organization said.
[5]
[6]
Some of the UK's more excitable tabloid media outlets have already reported that a French airline's flight plan submission may be to blame.
"Our systems, both primary and the backups, responded by suspending automatic processing to ensure that no incorrect safety-related information could be presented to an air traffic controller or impact the rest of the air traffic system," Rolfe said in the NATS statement.
[7]
Suspending automatic processing meant that flight plans had to be input manually by air traffic control staff, who were unable to handle the volume usually processed by the automatic systems, leading to traffic flow restrictions being imposed.
"Very occasionally technical issues occur that are complex and take longer to resolve. In the event of such an issue our systems are designed to isolate the problem and prioritise continued safe air traffic control," Rolfe added. "At no point was UK airspace closed but the number of flights was significantly reduced."
Tim Jeans, formerly managing director of Monarch Airlines, told the BBC's [8]Today program this morning: "There are four hours of flight data stored during the course of any working day, so if there is a fault, basically NATS has four hours to fix it. Now clearly they weren't able to fix it in four hours, the airlines will be rightly asking 'Why isn't it four hours or eight hours or 24 hours?' That's the service they're paying for."
[9]
About 1,100 flights were canceled with hundreds of thousands of travelers affected.
Rolfe responded that the four-hour time frame was down to the airlines, which only load their flight plans four hours in advance.
Travel chaos continues
NATS said that the glitch was dealt with and all of its systems have been running normally since Monday afternoon in order to support airline and airport operations as they recover from the incident.
There was still disruption on Wednesday, however, with many UK travelers abroad still reportedly not able to get alternative flights following Monday's cancellations.
Rolfe told the BBC that measures were now in place to protect against the "incredibly rare" system failure, and should it happen again, it can be resolved "very, very quickly."
The Register understands that air traffic control staff have responsibility for processing invalid flight plan messages submitted to the Flight Plan Processing System by manually correcting them as they collect in the invalid queue, suggesting that such issues are not uncommon.
It is conceivable that the problems seen on Monday may stem from human error when attempting to correct an invalid flight plan message. We asked NATS if this might be the case, but we did not receive an answer.
Resiliency – we've heard of it
A spokesperson for the UK Strategic Aviation Special Interest Group (SASIG) told us it was concerned that the supporting computer systems at NATS are so stretched that if they should fail, even for a relatively short period, the disruption is enormous.
[10]Lufthansa flights grounded by major IT snafu, 'construction work' blamed
[11]Airline 'in talks' with Kyndryl after failed network card grounds flights
[12]A few reasons why cops didn't immediately shoot down London Gatwick airport drone menace
[13]British Airways flights grounded due to glitch in flight planning app
"This suggests a lack of resiliency provision and raises the question whether NATS regard the costs of such as being too great and that because they do not have to bear the costs of disruption it's a risk they are willing to take."
Willie Walsh, Director General of the International Air Transport Association and former boss of British Airways, said in a statement that NATS had crucial questions to answer about their responsibility for what he called a "fiasco."
"The failure of this essential service is unacceptable and brings into question the oversight of the CAA (Civil Aviation Authority) who are required to review the NATS resilience plan under the terms of its licence," he stated.
Walsh told the Today program that the airline industry was facing substantial costs as a result of Monday's incident, possibly as high as a £100 million ($126 million) in total. He added: "I find it staggering, I really do – you know, the system should be designed to reject data that's incorrect, not to collapse the system.
"And if that is true, it demonstrates a considerable weakness that must have been there for some time.
"And I'd be amazed if that is the cause of this.
"Clearly we'll wait for the full evaluation of the problem but that explanation doesn't stand up from what I know of the system." ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2023/08/28/uk_flights_disrupted/
[2] https://www.bbc.co.uk/news/uk-66654338
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZO9noxJCarbLiPg-ukvbYgAAAg4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.nats.aero/statement/air-traffic-control-system-update/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZO9noxJCarbLiPg-ukvbYgAAAg4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZO9noxJCarbLiPg-ukvbYgAAAg4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZO9noxJCarbLiPg-ukvbYgAAAg4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.bbc.co.uk/sounds/play/m001q0rw
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZO9noxJCarbLiPg-ukvbYgAAAg4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2023/02/15/lufthansa/
[11] https://www.theregister.com/2022/10/10/kyndryl_aer_lingus/
[12] https://www.theregister.com/2018/12/20/gatwick_drone_non_shootdown_reasons/
[13] https://www.theregister.com/2022/12/20/ba_flights_delayed/
[14] https://whitepapers.theregister.com/
Re: Expertise
A passenger manifest including [1]Little Bobby Tables , perhaps?
[1] https://xkcd.com/327/
Re: Expertise
Ryanair fly to the new "London ^C" airport
Re: Expertise
If someone submitted wrong data and managed to take down the whole system?
From what has been said so far, I think it is more a case of the system not being able to parse input data submitted by one of airlines and activating what can only be described as an extreme fail-safe strategy (i.e. shut down all automated processing until the problem has been rectified). The question I would like to ask is: was that strategy actually appropriate, or could another strategy (for example, rejecting the dodgy data set and forcing the airline responsible to resubmit a corrected one) have been employed that is more appropriate given the impact that a complete system shutdown will have?
Re: Expertise
as an extreme fail-safe strategy (i.e. shut down all automated processing until the problem has been rectified)
In olden they you would call it brittle and something like this wouldn't get shipped to a customer.
But you can certainly reframe poor error handling as a "fail-safe strategy".
try {
data = extractFormData(message);
} catch () {
// TODO: implement error handling, but for now call it fail-safe strategy
volatile int32_t planeWankers = 1 / 0;
}
Re: Expertise
Its sounds like the poisoned message problem. You design a system so that no data is lost during the switchover.
A message with invalid data crashes the primary system, the backup system starts up from where the main system left of and starts by processing the poisoned message............
Re: Expertise
probably caused by this poor bloke who decided on a change of career after taking down fastly by submitting bad data
[1]https://www.theregister.com/2021/06/09/fastly_explains_web_blackout/
[1] https://www.theregister.com/2021/06/09/fastly_explains_web_blackout/
Re: Expertise
Always assume that the form may be filled by a dog or someone sitting on the keyboard .
In my first job, we had someone whom the dog would probably have outscored in an IQ test.
I don't think he ever actually sat on the keyboard, but he certainly did things to systems which you wouldn't have thought possible.
Invaluable, he was. If you want to make a system idiot-proof, first catch your idiot...
Re: Expertise
We had two testers, Cheryl and Caroline, who were incredibly devious in the scenarios they could think of. e.g. "I want to put -1 in this children field as he had a child, but his wife got custody in the divorce." (A deceased child only took it down to zero. The bug was that the field filled an integer rather than an unsigned integer, and so would let them do that.)
This was a trivial case that I am allowed to recount.
Once you prised them from the keyboard, they were wonderful people.
Re: Expertise
>I want to put -1 in this children field
Checking what happens if you can put -1 in an uint field is a good test.
Wasn't there a story here about a POS that let you enter a negative number as a tip and get a free meal ?
Re: Expertise
Invaluable, he was. If you want to make a system idiot-proof, first catch your idiot...
In my case it was Pete. Pete could be totally relied upon to do the most improbable and ridiculous thing when faced with any situation. Absolutely brilliant for ensuring my code could keep running no matter what got chucked at it.
Strangely, the one time all our kit got fried, Pete was in the room but not actually touching anything. He turned up in my office white as a sheet making "bu, bu, bu" noises. What nobody had known was that building management had for some reason run the lightning conductor down the wall outside the computer room. Pete was the only person in the room when a lightning strike hit the building and every piece of equipment turned into a plasma ball toy due to induced EMF while Pete cowered in the middle of the room thinking he was going to die.
Resiliency – we've heard of it
Input validation - what's that?
One wonders how bad the rest of the system must be if they couldn't even code simple checks to reject an incorrectly formatted flight plan.
Re: Resiliency – we've heard of it
I hadn't. I'd heard of resilience, though.
Re: Resiliency – we've heard of it
Not simple. Flight plans can be quite complex documents and doing a thorough input sanitation early on may not be feasible. I wonder if the whole truth of what happened will be released publicly. They may want to keep quiet about exactly what caused it to fail, if indeed it was a duff flight plan issue.
It's getting harder and harder for those excitable papers to blame everything on the EU (not that some still try hard) so it's of course the turn of the French.
As to the fail-safe strategy of shutting down everything on running into invalid data, it's hard to say whether that's appropriate or not without knowing a lot about the systems involved.
What certainly IS strange though is that a backup system (that is there precisely in case No 1 fails) has apparently been fed the very same crap... which produced the same result. Resilience?
What certainly IS strange though is that a backup system (that is there precisely in case No 1 fails) has apparently been fed the very same crap... which produced the same result. Resilience?
presumably though a backup would be useless without all current flight plans, this happening is probably less likely and preferable to a backup coming online and being next to useless because it's acting on information an hour old with planes flying around it doesn't know about
Uh?
So why bother with a backup if you can never use it?
Re: Uh?
In case the hardware dies
Systems I've worked on with disaster recovery have generally had the DR recovery point current or within a few seconds, usually using disk block level replication (e.g. EMC SRDF) or data transfer (e.g. Oracle Dataguard). As pointed out, you don't want to lose data when your primary goes down, but the problem is that the GIGO (Garbage In, Garbage Out) process takes both down at once, just the same as any other kind of data corruption would.
I've seen many comments on the Reg where people ask about disaster recovery/backup systems ignoring the fact that if the problem is the data, your DR system generally has the same problem as live.
In this case, the data absolutely should have been validated earlier and avoided the lockup that the system achieved. Where that flaw lies will undoubtedly result in a chunk of blame application, bus throwing and scapegoat discovery.
Maybe I'm old-fashioned, but if I have two identical systems fed the same data, then I'd expect them both to give the same result. Especially if it's something safety critical like making sure airplanes that have been carefully shepherd into close proximity to each other don't actually bump heads.
Or maybe there's a missing irony icon?
(Not that you'll find much iron in an airplane of course, or these days much metal at all given the preference for making everything out of araldite and burnt string.)
If Only ...
.. your old company was free of similar technical SNAFUs while on your watch Willie, but a lot of us know different. And then had to suffer the hassle of what passes for BA's refund "system".
@El Reg, do you have an icon for pot and kettle?
Re: If Only ...
And if I recall correctly, continually criticised NATS about their charges and blocking investment in the systems and processes that only exist to make sure his aeroplanes get to where he wants them to go so he can make money.
Le Brexit
I am incredulous that none of your regular commenters have blamed BoJo yet!!!
- De'guste' de Tunbridge Wells
Re: Le Brexit
Well it was a full Boris level shitshow.
Just checking to see if El Reg falls over if you use an apostrophe. O'Reilly.
Re: Le Brexit
BoJo brought Sunak who brought IR35 changes that culled experts from projects across multiple sectors.
This is why you don't put manglers in charge of anything important.
How many deaths did this "chaos" cause? How long would it take to count the bodies if the system were designed by a CEO?
How about unicode?
One of those characters that looks like a regular one, but isn't?
My new favourite euphamism
"Our systems, both primary and the backups, responded by suspending automatic processing to ensure that no incorrect safety-related information could be presented to an air traffic controller or impact the rest of the air traffic system,"
This guy is good. Very good. Sir Humphrey-level good. Chapeau, if that's too not inappropriate.
On encountering the wall, the car responded by suspending forward motion to ensure that no further processing of incorrect input data would take place and impact the rest of the vehicle
NATS should refund everyone - airlines and passengers.
And no %$%ing bonuses for NATS staff this year.
Expertise
Always assume that the form may be filled by a dog or someone sitting on the keyboard.
If someone submitted wrong data and managed to take down the whole system?
That is reassuring.
I think if we start paying engineers even less, then quality of work will certainly improve /s