News: 1634564710

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

EasyJet flight loadsheet snafu caused by software 'code errors' says UK safety agency

(2021/10/18)


An EasyJet flight to Edinburgh Airport took off with wrongly loaded passengers and baggage because of IT network congestion causing computer systems to interact "in a manner which had neither been designed nor predicted."

Last-minute aircraft changes followed by a critical but slow-running IT system meant the Airbus A321-Neo took off with a loadsheet intended for a different type of airliner. The loadsheet says where the aircraft's centre of gravity is – a vital safety calculation.

At the heart of the January 2021 cockup were "code errors" in EasyJet's departure control software suite, the UK's Air Accidents Investigation Branch said in a [1]recent report .

[2]

"The various elements of the IT system architecture do not 'talk' directly to each other but operate through a variety of interfaces," found the AAIB, adding this "makes errors and inaccuracies more likely."

[3]

[4]

An alert EasyJet cabin manager spotted the loadsheet discrepancy when passengers began taking seats in places the loadsheet said they shouldn't be sitting. If passengers (and baggage underneath) sit in different areas from what the loadsheet says, during takeoff the airliner might become unbalanced – leading to control difficulties for the pilots, or worse.

The cabin manager flagged up the loadsheet to the captain of the January 2021 flight, scheduled to run between Edinburgh and Bristol. Detailed investigations discovered that EasyJet's departure control software (DCS) had generated a loadsheet for an A320 and not an A321-Neo, meaning weight-and-balance calculations were wrong for that flight. The older jet had been swapped out for the larger A320-Neo at the last minute.

[5]

"Further investigation by the operator revealed that the discrepancy in information displayed between the aircraft management system and the departure control system was due to code errors in the Batch Interaction Layer operating outside of the original design specification," concluded the AAIB.

Every five minutes, said the AAIB report, the DCS runs an "internal validation process" to refresh its data sources for each flight. These include directly communicating with the aircraft's onboard computers to check that the aeroplane physically present to operate the flight is the same one as the DCS is expecting.

Two factors upset the DCS in this incident: the last-minute aircraft swap and the COVID-19 pandemic, which the AAIB blamed for "a high number of changes to [EasyJet]'s schedule" that day.

[6]

With a number (unspecified in the report) of changes being made to the DCS, that five-minute refresh job became slower and slower: "… changes made outside the five-minute window were not detected automatically by the system".

[7]Airline software super-bug: Flight loads miscalculated because women using 'Miss' were treated as children

[8]British Airways' latest Total Inability To Support Upwardness of Planes* caused by Amadeus system outage

[9]Amadeus! Amadeus! Pwn me Amadeus! Airline check-in bug may have exposed all y'all boarding passes to spies

[10]'It's like they took a rug and covered it up': Flight booking web app used by scores of airlines still vuln to attack – claim

Although gate staff had updated the DCS manually to input the A321-Neo, "the process did not consider this scenario and consequently the system had no mechanism to prevent the change of type being manually updated when boarding of the aircraft had started". The DCS used the aircraft registration letters as its validation for the aircraft, even though there was no cross-check linking the registration to the aircraft type.

"A type change registered in the [DCS] would prompt the seating algorithm to alter the [loading] figures, but the registration could match the previous aircraft causing confusion," found the AAIB.

EasyJet spokeswoman Holly Mitchell told The Register : "We are aware of the report and fully assisted the AAIB with its investigation. The safety of our passengers and crew is always our highest priority. We take events of this nature seriously and will always take action to ensure we maintain the highest standards of safety. While there were found to be no safety issues with this flight, we had already taken steps to implement actions to strengthen our procedures to prevent a recurrence."

Software-induced loadsheet dilemmas are not unknown in airlines. Back in April the UK offshoot of German airline TUI discovered it had been miscalculating passenger weights because [11]its check-in software recorded the title "Miss" as belonging to children instead of adult women. Meanwhile in 2018 British Airways' entire fleet was grounded worldwide after travel tech supplier Amadeus [12]suffered an outage of its loadsheet generation software . ®

Get our [13]Tech Resources



[1] https://www.gov.uk/aaib-reports/aaib-investigation-to-airbus-a321-251nx-g-uzmi

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YW2aOA7Y7WcOEoZ8ZGT59QAAAIU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YW2aOA7Y7WcOEoZ8ZGT59QAAAIU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YW2aOA7Y7WcOEoZ8ZGT59QAAAIU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YW2aOA7Y7WcOEoZ8ZGT59QAAAIU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YW2aOA7Y7WcOEoZ8ZGT59QAAAIU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[7] https://www.theregister.com/2021/04/08/tui_software_mistake/

[8] https://www.theregister.com/2018/07/19/amadeus_british_airways_outage_load_sheet/

[9] https://www.theregister.com/2019/07/16/amadeus_bug_light_pass/

[10] https://www.theregister.com/2019/01/15/amadeus_security_hole/

[11] https://www.theregister.com/2021/04/08/tui_software_mistake/

[12] https://www.theregister.com/2018/07/19/amadeus_british_airways_outage_load_sheet/

[13] https://whitepapers.theregister.com/



SkippyBing

'The DCS used the aircraft registration letters as its validation for the aircraft, even though there was no cross-check linking the registration to the aircraft type.'

They should get a subscription to Flight Radar 24, they have all that information. As do the CAA. As should Easyjet considering they own the aircraft!!

Yet Another Anonymous coward

Have they thought of painting it on the side of the plane, rather than just have a discrete little "320i" badge on the boot?

SkippyBing

You'd think that would help, one of the US carriers managed to fly the wrong A320 to Hawaii from LA. The right one had ETOPS* in large letters on the nose gear door, the wrong one didn't...

*Extended-range Twin-engine Operational Performance Standards to indicate that aircraft was fine to fly over the ocean without anywhere to divert to.

Ol'Peculier

Or "Engines Turn or Passengers Swim"

Yet Another Anonymous coward

To be far how many Americans would know that there is an ocean between LA and Hawaii ?

One of the usual idiot congresspersons was claiming that Hawaii's spike in Covid wasn't due to tourists but was all the fault of illegal immigrants - presumably building a wall would help.

WolfFan

It certainly would. If I can throw the idiots in question off of said wall. Optionally there might be hungry large land carnivores at the bottom of the wall. Komodo dragons for preference, but leopards, or larger cats would do nicely.

Weight of passengers

Yet Another Anonymous coward

Is this the system that uses an estimate of passenger's weight based on a very large, and therefore accurate, survey of bomber crews in 1945?

Re: Weight of passengers

SkippyBing

It may have been based on that originally but the current weights have been revised relatively recently. I think adults went up to 80kg at some point this century in response to the increase in mass of, well air passengers.

Re: Weight of passengers

Yet Another Anonymous coward

I was on a flight on some largish Boeing when they asked a dozen of us to move to the front of the cabin for takeoff.

Given that we were flying out of Houston and the other passengers all appeared to be either Sumo wrestlers or going to competitive eating event - I suspect the pilot took one look and thought; fsck what the balance software says

Re: Weight of passengers

wolfetone

It is, to an extent.

There was a crash of Air Midwest 5481, where the plane was overloaded and crashed in to the ground shortly after takeoff. The centre of gravity etc was pushed right back.

The NTSB worked out that the average weights used were more in line with what they were in 1936, and stated that carriers should use actual weights rather than averages. But (according to Wikipedia) 70% of the airlines in America still use averages.

I don't know if Europe uses different weights but yeah, you're not far wrong in your assumption.

Re: Weight of passengers

Yet Another Anonymous coward

> carriers should use actual weights rather than averages

I think that was specifically for these very small aeroplanes. I've been weighed on flights in the far North where you are wearing half of an REI store. It's done very discreetly so you don't see the actual number.

I don't imagine actual weight difference of 300 passengers on a 575 ton A380 makes that much difference.

Although I do think it is unfair that you get charged $1M/kg for overweight bags full of kit when Jabba the Hut's big brother is allowed to overflow into 3 seats for free.

Re: Weight of passengers

Ian Johnston

The weight of 300 passengers doesn't make much difference but the position of the weight of 300 passengers certainly does. 24 tonnes acting in front of or behind where the control surfaces expect can ruin your whole day.

Re: Weight of passengers

usbac

In light aircraft I always calculated weight and balance with actual weights of my passengers before takeoff. It's sometimes a delicate question, but you have to look at a person and deem whether they are telling you the truth.

I did know a pilot that carried a scale with him. I never went quite that far.

Re: Weight of passengers

Yet Another Anonymous coward

The checkin desk had a scale that you stood on with your hand baggage - so you could always claim it was a very heavy laptop

Not a trolling comment

Anonymous Coward

What factors determined that the DCS software performed so badly with a frequently changing schedule?

I can appreciate the calculations might be complex when working out optimum balance but we’re talking hundreds of data points, not thousands, or even millions

And once a loadsheet has been created which I imagine contains passenger seating and pallet loading plan - can’t be larger than a few kb? Maybe a meg? Can it? So there should have been no issue sending this to the loaders

And despite the chaos, are we talking about a dozen flights or thousands?

Ok ok ok, I get the 5 minute window / manual entry thing, but otherwise? We’re talking a few thousand data points per plane x a few hundred planes (max) x a few kilobytes, maybe megabytes, per plane. Computationally at least, a 386 could’ve handled that

See title. Not trolling, but as a developer, recently, of load scheduling as it happens, I’m curious as to why and would like to know what I’m missing

Re: Not a trolling comment

Yet Another Anonymous coward

I imagine a formal requirement.

Req: 3.141592654.... A load sheet shall be generated before boarding the aircraft

Therefore there is no mechanism for generating a new load sheet after boarding has started because that would be illegal. The formal requirements would be that in this event the flight is cancelled, a new load sheet is calculated, along with a new flight plan and a ripple effect that makes 1000s of flights delayed.

Re: Not a trolling comment

Anonymous Coward

> We’re talking a few thousand data points per plane x a few hundred planes (max) x a few kilobytes, maybe megabytes, per plane. Computationally at least, a 386 could’ve handled that

You're assuming that all the data is available locally. The systems I worked with at Heathrow often took data from a data bus, but that data was sometimes only put onto the bus at 5 minute intervals (usually because it came from one particular overloaded legacy system running on ancient hardware that no one dared try and upgrade because it was so critical).

So the calculation takes only a couple of seconds but the software has to wait several minutes for all the data to be marshalled. To make matters worse, if the software just queues up changes, without checking to see if the latest change invalidates one that's already in the queue, then it could wait up to 5 mins for the first change then another 5 mins before the second happens.

At the end of the day the pilot took off anyway - so it clearly wasn't that critical.

Re: Not a trolling comment

ortunk

Once debugged a problem like that, ayatems concerned were taking 15 minutes however truck did it in 13-14 and had to wait for data at the door.

Solution all syatems start triggering every 3 seconds :))

The score

Version 1.0

Humans working: 1

SWapps working: 0

GlenP

I recall an internal US flight into Rhode Island IIRC (but may have been somewhere in Michigan) where the pilot stuck his head out of the cockpit door and asked, "Can some of you move nearer the back?"

Passengers had been loaded solely from the front of the aircraft and it was fairly empty.

Batch Interaction Layer

ortunk

So cron jobs and fetch scripts as usual, a BOFH was there handling the auditor it seems

Walters' Rule:
All airline flights depart from the gates most distant from
the center of the terminal. Nobody ever had a reservation
on a plane that left Gate 1.