Techies try to bypass damaged UPS, send 380V into air traffic system
- Reference: 1672750810
- News link: https://www.theregister.co.uk/2023/01/03/philippines_air_traffic_outage/
- Source link:
"At around 9:49 am local time, the Air Traffic Management Center (ATMC) which serves as the facility for controlling and overseeing all inbound and outbound flights and overflights within the Philippine airspace, went down due to power outage, resulting to loss of communication, radio, radar, and internet," [1]said transport secretary Jaime Bautista in an [2]online press conference.
The outage went on for eight hours, with normal operations resuming around 18.00 the same day. While NAIA [3]activated its crisis management team for a "multi-discipline approach to cushion the impact of the incident" and [4]ordered emergency response teams to implement irregular operations procedures, more than 65,000 would-be passengers across 360-plus flights were stranded.
LOOK: Stranded passengers jampacked at NAIA Terminal 3 as flights have been put on hold over “technical issues.” (🎥: [5]@NashNacion ) [6]pic.twitter.com/0R4TpTTgd5 — Michael Joe Delizo (@michael_delizo) [7]January 1, 2023
The glitch didn't just delay passengers coming in and out of Manila, it also disrupted the Manila Flight Information Region (FIR), which covers the entirety of the Philippines. This caused all commercial flights out of the country's airspace to be redirected.
There are currently no airborne commercial flights in the Philippines (MANILA FIR). [8]pic.twitter.com/xUhyPJarmE — Flightradar24 (@flightradar24) [9]January 1, 2023
MIAA general manager Cesar Chiong anticipated it would take approximately 72 hours to return to normal operations, with most flights coming in or out of Manila packed for the time being.
[10]Philippines logs on to Starlink for remote area internet services
[11]Southwest Airlines blames IT breakdown for stranding holiday travelers
[12]UK government in talks with datacenter operators over blackouts
[13]Tech supply chains brace for impact as China shifts from zero-COVID to rampant COVID
According to the [14]Civil Aviation Authority of the Philippines (CAAP), the loss in power originated in the Communications, Navigation, and Surveillance/Air Traffic Management (CNS/ATM) system's electrical network. Of course, uninterruptible power supplies (UPS) exist to prevent this exact scenario, but neither of the system's two backup power supplies worked.
One because its blowers "conked out," and the other simply failed to come online, according to CAAP director general Manuel Tamayo.
[15]
"The primary cause identified was a problem with the power supply and the degraded uninterrupted power supply, which had no link to the commercial power, and had to be connected to the other manually. The secondary problem was the power surge due to the power outage, which affected the equipment," said Bautista.
[16]
[17]
That power surge occurred when technicians tried to bypass the damaged UPS and sent 380 volts into the system, instead of the intended 220 volts, thus frying the terminals that receive satellite data from airplanes and air traffic management systems.
To add to the shame of not one but two failed UPSes, both the CAAP and Bautista recognized the CMS/ATM was already outdated before it was even fully operational in July 2019. Currently, in 2023, it has been outdated for a full decade.
[18]
"Although it's a system that was introduced in 2010, we implemented this in 2018, so the system is already in its midlife, so we really need to improve or modernize it. Maybe we can still use it, but we need to upgrade this to a better system," said Bautista.
The transport secretary added that the country's air traffic management system was at least 10 years behind that of Singapore.
The CAAP said it has made recommendations to President Ferdinand "BongBong" Marcos Jr to improve the country's air traffic management system. Marcos has already ordered a proposal on system upgrades. An investigation is underway by CAAP's Aerodrome and Air Navigation Safety Oversight Office and some senators have [19]called for heads to roll at the CAAP. Other politicians have urged to allocate funding for upgrades.
[20]
According to Batista, an upgrade would cost upward of $233 million (13 billion Philippine pesos), similar to what the 2018 system cost. That system was financed with loans from the Japan International Cooperation Agency.
Opportunists have already come in to offer assistance with the upgrade. Manuel Pangilinan, owner of the Philippines' largest telecom company, PLDT, suggested a colocation of multiple protective redundancies.
If our Group could be of any help to DOTr/CAAP, we’d be happy to participate - colocation of 2nd, even 3rd, redundancies in our nationwide data centers, required connectivities - fiber, satellite, wireless - robust even redundant power supply protection etc.Let’s all support. — Manny V. Pangilinan (@iamMVP) [21]January 1, 2023
Pangilinan himself was on a flight from Tokyo to Manila when his plane was turned around and sent back to Haneda.
"Six hours of useless flying but inconvenience to travelers and losses to tourism and business are horrendous. Only in the PH. Sigh," [22]tweeted Pangilinan.
Pangilinan is familiar with difficult system upgrades. PLDT experienced a P48 billion budget overrun after it [23]over ordered network equipment, forcing the exec into a shareholder mea culpa just last month. ®
Get our [24]Tech Resources
[1] https://www.facebook.com/100067651842656/posts/pfbid0311jFVMKnUf3jNLqDKyxidJp1XU1XKYnjnuR7xczoyZDVseSdjFWT8PGCT8bFMqtdl/?d=n
[2] https://www.facebook.com/PTVph/videos/543946364439710/
[3] https://twitter.com/MIAAGovPH/status/1609428116384141312
[4] https://twitter.com/MIAAGovPH/status/1609429197520531457
[5] https://twitter.com/NashNacion?ref_src=twsrc%5Etfw
[6] https://t.co/0R4TpTTgd5
[7] https://twitter.com/michael_delizo/status/1609455415045263360?ref_src=twsrc%5Etfw
[8] https://t.co/xUhyPJarmE
[9] https://twitter.com/flightradar24/status/1609412831962714112?ref_src=twsrc%5Etfw
[10] https://www.theregister.com/2022/07/28/starlink_philippines/
[11] https://www.theregister.com/2022/12/28/southwest_outdated_it/
[12] https://www.theregister.com/2022/10/18/uk_government_in_talks_with/
[13] https://www.theregister.com/2022/12/19/china_supply_chain_living_with_covid/
[14] https://pia.gov.ph/news/2023/01/02/caaps-statement-on-the-atmc-power-supply-failure
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y7RfMuZp0DIfHGWv69DM9wAAABg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y7RfMuZp0DIfHGWv69DM9wAAABg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y7RfMuZp0DIfHGWv69DM9wAAABg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y7RfMuZp0DIfHGWv69DM9wAAABg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[19] https://www.pna.gov.ph/articles/1191833
[20] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y7RfMuZp0DIfHGWv69DM9wAAABg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[21] https://twitter.com/iamMVP/status/1609700611741872129?ref_src=twsrc%5Etfw
[22] https://twitter.com/iamMVP/status/1609457082419544064?ref_src=twsrc%5Etfw
[23] https://www.philstar.com/business/2022/12/27/2233418/mvp-pldt-budget-mess-were-not-hiding-anything-not-playing-truth
[24] https://whitepapers.theregister.com/
Re: Critical
If they sent 380v into a 220v device that must mean they connected two of the three phases one to live and the other to neutral.
Was this a qualified electrician and if so in what?
To be honest I thought it was a who me a day late not a new story when I read it.
Re: Critical
I'm sure he was. I've seen quite a few electricians (and other "professionals") that had tons of qualifications.
Heck, they even had the NEMA 1-day "don't stick your fingers in the socket" course!
Re: Critical
Not sure if a fuse provides protection for over-voltage. By the time the current increases to the kill-fuse level, the damage may already be done.
I'm not arguing against fuses, they are a vital part of any protection regime just not sure if they would have helped much in this circumstance.
Re: Critical
Fuses won't stop frying, especially of the over-voltage variety
Future who me?
major fuck-up I reckon
Major UPS screw-up, by all means, but the one below freaked me out !
" ... ordered emergency response teams to implement irregular operations procedures, ..."
Like, REALLY ? I'd like to know exactly what those were, or should I just make sure I never approach PH in a plane at all ?
Re: major fuck-up I reckon
The procedures for when operations aren't operating as they regularly would
Get it while still moving
If our Group could be of any help to DOTr/CAAP, we’d be happy to participate...
Wow!
Translation: if we can earn a buck or two from this disaster, then we really want to be first in line to fill our bank accounts.
Picking on corpses while they are still slightly alive. That was a fast response. Never let a good disaster go to waste to earn another boat load of monetary exchange leaflets.
Been there but not the one who caused the fault
A few years back the company I worked for had finally decided to replace the batteries in the 3 phase UPS. Whoever installed it had done a bit of a 'quick and dirty' with the wiring and had not left much in the way of slack. The door that allowed access to the batteries was partially blocked by the output cabling but the service people could just about get the access they needed without the need to power down the server room.
About half way through the job the tension on the output wiring resulted in the neutral cable coming out of its terminal and falling against one of the live phase outputs. Amazingly only a couple of bits of kit let the smoke out before the monitoring shut the whole room off. Unfortunately some of the PSUs to the out of support but still critical SAN let the smoke out and it would not come back to life with the remaining PSUs. We managed to get some spares next day (phew!) and got pretty much everything up and running within 24 hours.
It took a few years off the IT managers lifespan though!
Aren't many APC UPS models made in the Philippines?
This incident doesn't shine a good light on local management of such devices.
Don't worry, be wiring.
When my university extended its campus, there was a stuff-frying incident with the electrical installation. A normal outlet was wrongly wired, thus delivering 380V instead of 220V.
It's so easy to make such a mistake, there are 4 or 5 wires in a cable: 3 phases, a neutral and ground/earth.
If you connect a phase and the neutral to a normal power socket, it delivers 220V.
If you, however, use two of the phase-wires, the outlet delivers 380V.
Normally, there is some color-coding of the wires to avoid this type of mistake, but stuff gets stuffed up nonetheless.
Re: Don't worry, be wiring.
Note also that one should test the output when connecting a new socket, just to be sure. In that case, a multimeter can help.
No Excuses
If you know there's three phase around you should always check with a suitable test meter. If you 'know' there isn't you should still check in case some numpty has been messing around. Also, especially check neutral continuity. The most lethal situation is a broken neutral that's just hanging there looking innocent because there's a star connected motor on it.
Re: No Excuses
Or as a wise man once sad: "If God tells you it's locked out, double check with Jesus, then test it dead yourself."
Strangers in the night
There's no power, what do we do?
747 coming down in the night
Tryin' to get a message though
What The People are Saying
“This delayed my holiday shipment of p.napoleonis galliformes from Palawan” Wile E Coyote, Restauranteur
“Maybe now they will consider moving to cloud.evil. Isn’t that right Mr. Bigglesworth?” Dr. Evil, Cloud CTO
“They shouldn’t have messed with my Pikachu” Ash K, Plebe Trainer
Re: What The People are Saying
> “Maybe now they will consider moving to cloud.evil. Isn’t that right Mr. Bigglesworth?” Dr. Evil, Cloud CTO
Well on-prem had too much Austin Powers going to it
A [1]couple of weeks ago in Bristol , National Grid somehow ended up raising the mains voltage to 266V. Incredibly, basically everything was fine and kept on working as normal.
We only noticed because the UPS feeding our comms rack was cheap and just passed on the 266V to the transfer switch. The transfer switch decided that 266 was too high and just turned off, so we only noticed when we lost internet access. The other UPS's coped just fine and conditioned the power, one recorded the initial spike at 272V! (Frequency didn't seem to drift any more than it usually does over a day).
If you'd asked me before, I'd have assumed that most consumer equipment would start to die past 260V, but everything from desktop computers to lightbulbs just shrugged it off.
[1] https://www.bbc.co.uk/news/uk-england-bristol-64056932
UK electronics have to tolerate 253v, don't they? 260 isn't that far off.
Critical
That power surge occurred when technicians tried to bypass the damaged UPS and sent 380 volts into the system, instead of the intended 220 volts, thus frying the terminals that receive satellite data from airplanes and air traffic management systems.
One could think that such a critical infrastructure would have additional circuits (sometimes being just a fuse) preventing any kind of frying in case a human decides to play games?
edit:
(sometimes being just a fuse)
That actually reminds me that I sometimes buy broken things on eBay to see if I can fix them. Very common thing is that the device has a fault and blow a fuse, so people tired of exchanging fuses just put a wire across and then device eventually gets fried. So for such a critical infrastructure, the fuse needs to be built in such a way, so that a human feeling clever wouldn't be able to easily install a wire to bypass it.