Lufthansa flights grounded due to major IT outage, 'construction work' blamed
- Reference: 1676464453
- News link: https://www.theregister.co.uk/2023/02/15/lufthansa/
- Source link:
The company confirmed to The Register : "Fiber-optic cables belonging to a telecommunications service provider were damaged during construction work in Frankfurt, causing an outage of Lufthansa's IT systems at the airport in Frankfurt. Flight operations are expected to stabilize in the early evening.
Deutsche Telekom has confirmed to us that it's cables were also cut.
[1]
"Late yesterday afternoon, a drill cut four fiber optic cables from Deutsche Telekom during third-party construction work on a rail line in Frankfurt. As a result, Lufthansa is also experiencing IT problems.
[2]
[3]
"Two cables have already been repaired overnight by our technical team and many customers are already back online. We are continuing to work at high pressure on the repairs. The situation is improving continuously.
"To ensure that customers can still be reached, they can request free call forwarding to their mobile devices from our customer service," the spokesperson told The Reg .
[4]
Lufthansa [5]first admitted to issues on Twitter at 1055 UTC: "Since this morning, the airlines of Lufthansa Group have been affected by an IT failure caused by construction work in the Frankfurt region. Unfortunately, this has led to flight delays and cancellations."
It [6]subsequently added : "We are working flat out on a solution. Until then, we ask all affected guests flying on LH domestic flights within Germany to book a train ticket and apply for a refund on http://lh.com."
It asked affected passengers to check the status of their flight on the company's app or website before going to the airport and told passengers with domestic flights they could "switch" their journey to a significantly longer one aboard national train operator Deutsche Bahn "until Sunday."
[7]
Lufthansa's global flight operations center is based in the outskirts of Frankfurt airport and it is here that disruption to the cables happened.
There's no insight into whether Lufthansa's presumed backup for that connection also failed, as was the case with [8]Aer Lingus in October last year. Aer Lingus execs claimed at the time that data was mirrored to two separate sites by their IT services provider, and that the lines had been replicated into both.
If indeed Frankfurt construction work alone is to blame for its troubles, then Lufthansa may want to reconsider resiliency of its systems, as one person on Twitter mentioned.
German engineering at its best! A single fiber cut causes a global outage! Where is the redundancy? — Mark (@FunkyBunch3112) [9]February 15, 2023
Ground systems that include check-in services were hit by the IT issue, and local air traffic control are sending planes off to other German airports including Nuremberg, Cologne and Dusseldorf.
[10]Southwest promotes internal IT executive to CIO in wake of that Christmas meltdown
[11]FAA grounds all US departures after NOTAM goes down
[12]Boeing bids the 747 a final, ultimate, conclusive farewell
[13]Airline 'in talks' with Kyndryl after failed network card grounds flights
There are reports that the IT issues are forcing Lufthansa to resort to pen and paper when boarding planes and that it is unable to process passengers' luggage digitally. And pictures are emerging on social media of the queues customers are forced to join.
Line to rebook or get an update from on the ground staff at Frankfurt Airport is ridiculous. Can I just get my bag from a cancelled flight [14]@lufthansa ?!?!?!?!? [15]pic.twitter.com/eUTl4cBtND — Krissie Vitasa (@kriseldav) [16]February 15, 2023
®
Get our [17]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y@0Pr8IFCMS3K3eBq0aOwgAAARI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@0Pr8IFCMS3K3eBq0aOwgAAARI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@0Pr8IFCMS3K3eBq0aOwgAAARI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@0Pr8IFCMS3K3eBq0aOwgAAARI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://twitter.com/Lufthansa_DE/status/1625811151899557888?s=20
[6] https://twitter.com/Lufthansa_DE/status/1625811153292075014?s=20
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@0Pr8IFCMS3K3eBq0aOwgAAARI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/10/10/kyndryl_aer_lingus/
[9] https://twitter.com/FunkyBunch3112/status/1625810448678547456?ref_src=twsrc%5Etfw
[10] https://www.theregister.com/2023/02/02/southwest_airlines_cio/
[11] https://www.theregister.com/2023/01/11/faa_grounds_all_us_departures/
[12] https://www.theregister.com/2023/02/01/boeing_747_final_goodbye/
[13] https://www.theregister.com/2022/10/10/kyndryl_aer_lingus/
[14] https://twitter.com/lufthansa?ref_src=twsrc%5Etfw
[15] https://t.co/eUTl4cBtND
[16] https://twitter.com/kriseldav/status/1625813763327934464?ref_src=twsrc%5Etfw
[17] https://whitepapers.theregister.com/
Indeed, even when redundant ducts are installed, often the contractors who then get tasked with pulling the fibre into them end up taking the lazy approach and pull both into the same duct and leave one empty.
The other classic one is you have redundant ducts properly taking geographically diverse routes to a facility, but they then come together where they enter the building presenting a lovely target for a JCB...
Yup. Entirely normal.
You buy service from two ISPs to get 'redundancy'.
Super-Duper ISP - Yes, you can have service from us between Frankfurt and Munich
Excellent ISP - Yes, you can have service from us between Frankfurt and Munich
Super-Duper ISP then calls Little-Horse Backbone fibre company - I say, can we have a path lit up between Frankfurt and Munich? Answer - of course!
Excellent ISP then calls Little-Horse Backbone fibre company - I say, can we have a path lit up between Frankfurt and Munich? Answer - of course!
So Little-Horse Backbone fibre company lights up a couple of wavelengths on their fibre between Frankfurt and Munich.
Little-Horse Backbone fibre company will not tell Super-Duper or Excellent which other customers are on that fibre (commercially confidential, obviously)
Along comes a JCB (backhoe for Americans), and Whammo!, your redundant solution fails.
***
You do not get guaranteed redundancy simply by buying from separate service providers, when the services are layered like filo pastry in baklava.
It's also surprisingly difficult to get from a single provider, but that's another story.
There is 'redundant' and then there is diversity and physical separation
The latter can be very hard to prove, particularly if you work with two organisations for the two 'redundant' routes. Sounds counter-intuitive, but having one organisation providing two physically diverse fibre links means that they (assuming their records are accurate) can actually prove that physical diversity. Otherwise you have the nested selling/reselling issue - often without the visibility you would require.
You need to know what you are asking for and then insist on evidence from the underlying fibre providers, regardless of your chosen service provider.
Ironic that this happened in the airline industry, which is all about resilience through carefully designed redundancy.
I used to work in IT in a US airport that LH flies into.
LH were one of the few carriers to bother getting two separate circuits into the airport, into two separate routers...in the same comms room. Where we then split one circuit to go over our internal backbone to our redundant data centre.
So you're left at the mercy of a single cable duct which could take out most of the airline connectivity at that airport.
They did look at the option of getting a circuit directly into the redundant data centre, problem was it was in the middle of nowhere about 3 miles away from anything, and the extortionate amount the telcos wanted to put a circuit for LH directly into that data centre probably would have wiped out their profit margin on that route......
Fiber... shred
Deutsche Telekom confirms too
https://twitter.com/deutschetelekom/status/1625587220785598464
https://twitter.com/deutschetelekom/status/1625824840924950528
German engineering
>Where is the redundancy?
As soon as the fibre was cut a team was dispatched to cut all the other fibres. For some reason this was in the operations manual and so 've vere only following orders'
Re: German engineering
Aye well, they've got a month to get it sorted before I'm flying :)
Re: German engineering
Don't worry: strikes at the airports will start on Friday.
Probably multiple "redundant" fibres in the same cable/duct, maybe even used by different ISPs. Very, very common I'm afraid and if you ask your ISP about this you will often not get far..