News: 1764075434

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Airbus: We were hours from pausing production in Spain

(2025/11/25)


Exclusive Airbus is overhauling its datacenter contingency plans after a ten-hour power outage across Spain and Portugal in April nearly forced a complete production shutdown.

The blackout – one of the worst in Iberian history – was triggered by voltage surges that overwhelmed the electrical grid. It knocked out traffic lights, forced metro evacuations, and impacted Airbus's primary Spanish datacenter in Madrid, part of its Campus Futura aerospace and defense hub.

Nationwide power outages knock Spain, Portugal offline [1]READ MORE

"We thought we were ready for an electrical issue, and in fact we were not," Catherine Jestin, Airbus Executive Vice President of Digital, told The Register .

Despite backup generators, Airbus faced a critical fuel shortage. The company scrambled to order more diesel, but fuel suppliers were overwhelmed with calls from other affected customers. "We were a few hours from a datacenter shutdown," Jestin said.

The stakes were high. Without datacenter access, production workers couldn't retrieve documentation or instructions. Even warehouse operations would have stalled if automated parts retrieval systems went offline.

[2]

"Probably we could have survived one day," Jestin added, noting that the company resorted to manual workarounds.

[3]

The incident has "triggered an improvement plan." Airbus is now ensuring adequate generator fuel reserves and priority supply contracts across its facilities in Spain, France, Germany, and the UK. The goal is to maintain operations for several days during a "massive blackout" without relying on external fuel deliveries.

[4]Cloudflare broke itself – and a big chunk of the Internet – with a bad database query

[5]Airbus okays use of 'Taxibot' to tow planes to the runway

[6]Azure stumbles in Western Europe, Microsoft blames 'thermal event'

[7]The race to shore up Europe's power grids against cyberattacks and sabotage

This may involve paying a premium for the fuel to power backup generators, yet pales into insignificance compared to the potential cost of halting a production line.

The outage in late April was a very rare event. Portugal's national grid operator said at the time: "Due to extreme temperature variations in the interior of Spain, there were anomalous oscillations in the very high voltage lines (400 kV), a phenomenon known as induced atmospheric vibration. These oscillations caused synchronization failures between the electrical systems, leading to successive disturbances across the interconnected European network." ®

Get our [8]Tech Resources



[1] https://www.theregister.com/2025/04/28/nationwide_power_outages_knock_spain/

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aSXgqhdGLNJsknA-B489hAAAANI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aSXgqhdGLNJsknA-B489hAAAANI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://www.theregister.com/2025/11/19/cloudflare_incident_report/

[5] https://www.theregister.com/2025/07/07/airbus_taxibot/

[6] https://www.theregister.com/2025/11/05/azure_thermal_event_west_europe/

[7] https://www.theregister.com/2025/11/03/europe_power_grid_security/

[8] https://whitepapers.theregister.com/



An Aviation Company Runs Out of Gas?

AnAnonymousCanuck

Maybe they should have had one of their pilots dip the tank once/twice a year.

YMMV, maybe not if the tank is empty.

AAC

Re: An Aviation Company Runs Out of Gas?

Snake

I would think that this teaches companies to plan mission-critical data centers with diesel fuel *and* access to [natural] gas feeds so that they can have both types of generators available. Even with large fuel supplies, unless you treat diesel fuel it can have problems with long-term stability *and* diesel generators do not like sitting long-term without use (why they have programmed exercise cycles); natural gas-fueled generators are much more tolerant of long-term standby.

Re: An Aviation Company Runs Out of Gas?

Anonymous Coward

"long-term stability"? Stuff starts growing in it.

Re: An Aviation Company Runs Out of Gas?

Lazlo Woodbine

Can confirm this.

Back at the turn of the millenium, I was working in a large retailer. We didn't really bother too much about the odd hour with no power back then, but one year, the shopping centre was having an extensive refit which would involve completely shutting down occasionally, including a whole bank holiday weekend. This would be a problem for the freezer department, which could only stay below -18°c for about 12 hours without power.

We decided to look at the generator on the roof, a big old Cummins diesel. Dipping the tank found it to be empty.

No problem, we'd get a barrel delivered and pump it up to the roof using the pump in the loading bay provided for that purpose.

Nope, the pump was dead, because nobody could remember it ever being used.

We rolled the barrel into the goods lift, but this only took us to the level below the roof, so we had to very carefully, and slowly, walk the barrel up a flight of stairs to reach the generator, where we used a hand pump to fill the tank.

Then we tried to start the generator, and it wouldn''t.

Luckily, my dad is a truck mechanic, so he came out and soon got it running.

We tested the generator properly by shutting the power off, it worked fine.

The next day the shopping centre people told us they wouldn't need to shut the power off.

At least we now had a working back-up generator...

What about connectivity

sal II

"Airbus is now ensuring adequate generator fuel reserves and priority supply contracts across its facilities in Spain, France, Germany, and the UK. The goal is to maintain operations for several days during a "massive blackout" without relying on external fuel deliveries."

They will also need to make sure that whoever provides their connectivity has similar provisions. Otherwise it will be a bit futile.

Re: What about connectivity

Roland6

The other piece of planning is to ensure during those “couple of days” they have the means to effect a full controlled shutdown.

Before getting the solar battery system. My UPS ran for 10~15 minutes, sufficient to ride out voltage fluctuations but totally insufficient for a major power outage. My thinking was if the power was out for more than 5 minutes it was going to be out for several hours and thus I had bigger problems, like no Internet/land line and mobile after circa 1 hour.

Re: What about connectivity

sal II

yes but looks like their goal is to continue production, not just give them more time for controlled shut down

Flawed business continuity plan…

Roland6

>” The company scrambled to order more diesel, but fuel suppliers were overwhelmed with calls from other affected customers.”

Saw this with Covid where DR/BC plans assumed:

a) only the single business was experiencing an outage

b) stuff (eg. SIMs, phones, laptops etc.) could readily be obtained from suppliers (because of assumption ‘a’).

Re: Flawed business continuity plan…

Korev

There was a fire at somewhere I used to work, the emergency plans mostly worked really well and the important people setup camp in the DR room in a different building. The only big problem was that people had gone from BlackBerrys and Nokias to iPhones and you can imagine which cables were in the said room. The big bosses were using their phones a lot for obvious reasons, so towards the end of the day getting hold of iPhone cables suddenly became one of the most important tasks.

They borrowed mine and never gave it back, the rude ****ers!

Re: Flawed business continuity plan…

An_Old_Dog

They borrowed mine and never gave it back, the rude ****ers!

For executives mentally-flying at 30,000 feet up ("big-picture thinkers") things like returning your cord are just "little details" which they quickly forget/don't care about.

That is one reason I never do "little favors" for such people.

Re: Flawed business continuity plan…

tip pc

Nokias & Blalckberrys would have been good for at least 2 days of constant calls, mails & texts..

was always amazed at how long my blackberry lasted despite the constant alert texts, emails and conference calls ii sued it for,

my iphone barely lasts a day even when i'm at home and barely touch it.

Re: Flawed business continuity plan…

Anonymous Coward

Strange - my iPhone 16Pro usually goes on my nightstand to get charged to 80% overnight - but, unless it's had some heavy use, it's rarely down to 50% by nightfall. After a year, its battery health reports 215 cycles and 100% maximum capacity. But then, I mainly use it for calls and texts, not as a portable game m/c or doomscroller. It gets used for directions when I'm driving to a new place, or a long distance (the latter usually to warn of delays ahead) - it then sits on charge. Even so, I still keep a battery in my EDC - just in case - BSS (Boy Scout Syndrome)!

Re: Flawed business continuity plan…

FirstTangoInParis

> Nokias & Blalckberrys would have been good for at least 2 days of constant calls, mails & texts..

The original Nokia’s would last 2 weeks on standby when fitted with the big battery. Those were the days.

Re: Flawed business continuity plan…

Lazlo Woodbine

Yep, supplies were a major issue during Covid.

I was working in a decently run boarding school at the time, and through contacts in China, we knew what was coming, so had started to make plans.

We ordered laptops for all staff that didn't have one, webcams for desktop computers, additional bandwidth, everything we could think of.

The problems started when webcams started disappearing, presumably so staff could zoom family from home. Our usual supplier had run out, our back-up supplier had run out and Amazon had more than doubled the price. We eventually managed to order a box of 24 direct from Logitec in Switzerland.

The same with laptops, when the new ones ran out, and jealous owners of older laptops wanted one the same as a colleague, we found HP had put prices up by over 50%.

BOFH-Excuse

An_Old_Dog

https://www.euronews.com/next/2025/04/28/what-could-have-caused-the-major-power-outage-in-spain-and-portugal-experts-weigh-in

"These oscillations caused synchronisation failures between the electrical systems, leading to successive disturbances across the interconnected European networks," Portuguese energy company REN reportedly told the BBC on Monday afternoon.

REN later refuted these claims in the Portuguese media.

https://www.livescience.com/planet-earth/energy/what-is-induced-atmospheric-vibration-and-did-it-really-cause-power-outages-across-spain-and-portugal

"The blackout that hit the entire territory of mainland Portugal today is the result of a significant voltage fluctuation in the Spanish grid at a time when Portugal was importing energy from Spain," REN said in a statement on Monday afternoon. "With this fluctuation, the control and protection systems of the Portuguese power plants, as expected in a situation with this configuration, shut down, causing the blackout."

The "induced atmospheric vibrations" explanation sounds like a line out of the BOFH's excuse list.

Re: BOFH-Excuse

MiguelC

Just this morning I posted a review of the disinformation spread about that event on another story's forum, you can check it at [1]https://edmo.eu/publications/disinformation-about-the-blackout-went-around-the-world-impersonated-media-rare-atmospheric-phenomenon-and-russian-networks/

[1] https://edmo.eu/publications/disinformation-about-the-blackout-went-around-the-world-impersonated-media-rare-atmospheric-phenomenon-and-russian-networks/

Wasn't Just a Power Outage

Anonymous Coward

I remember the outage because my wife and I arrived in Madeira (on holiday) that day and neither of our phones had coverage when we landed - my first check being to see if our luggage had arrived with us, or was still back at our departure airport! Nor was their internet access via the hotel wifi that evening. We discovered the reason the following morning, at breakfast. The island's power was uninterrupted (local generation) but communications went via the mainland...

When managers hold endless meetings, the programmers write games.
When accountants talk of quarterly profits, the development budget is about
to be cut. When senior scientists talk blue sky, the clouds are about to
roll in.
Truly, this is not the Tao of Programming.
When managers make commitments, game programs are ignored. When
accountants make long-range plans, harmony and order are about to be restored.
When senior scientists address the problems at hand, the problems will soon
be solved.
Truly, this is the Tao of Programming.
-- Geoffrey James, "The Tao of Programming"