Techie traced cables from basement to maternity ward and onto a roof, before a car crash revealed the problem
- Reference: 1749195014
- News link: https://www.theregister.co.uk/2025/06/06/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Geoffrey," who told us about the time he was called out to fix a hospital's access control network – a setup that controlled physical access to different parts of the facility.
"Doors that should be locked were unlocked, cards that should unlock doors wouldn't register," Geoffrey told On Call. "It was a closed network, but absolute chaos, and the onsite engineer couldn't figure out what was going on," he added.
[1]
The job needed boots on the ground, and Geoffrey's boots were made for walking in to save the day.
[2]
[3]
On arrival he started in the basement and found a 16 port switch - part of the spine of the access control network.
"A couple of activity lights on the switch were blinking way too fast," Geoffrey told On Call. He decided to follow the cable that corresponded to the distressed port and found another switch and another pair of rapidly blinking activity lights.
[4]
Following another cable led Geoffrey into a maternity ward that incongruously housed a small wall-mounted box within which he found a small switch, power-over-ethernet injectors, and yet more cables that he learned were connected to a pair of wireless bridges on the hospital roof that enjoyed line-of-sight to a carpark.
Because of course it made sense for a switch in a maternity ward to connect to a roof and a car park.
Geoffrey unplugged an injector and the network returned to perfect health.
[5]
Which suggested the bridge was the source of the borkage and meant a visit to the roof was in order.
Once he ascended, Geoffrey gazed at the carpark and noticed that one of its exit barriers was working fine but the other was permanently raised.
The onsite engineer explained that someone recently drove their car into the barrier and damaged the wireless bridge, which had been replaced … just before the access control system went bad.
Geoffrey had a hunch that the access control system vendor shipped pre-configured bridges, usually with the same SSID and password.
[6]Techie fixed a ‘brown monitor' by closing a door for a doctor
[7]User unboxed a PC so badly it 'broke' and only a nail file could fix it
[8]People find amazing ways to break computers. Cats are even more creative
[9]Need a Linux admin? Ask a hair stylist to introduce you to a worried mother
A little testing proved his hypothesis: the bridges on the roof both shared a name and password. After the accident in the carpark they'd dropped the connection to the Wi-Fi kit down there and connected to each other. The resulting loop flooded the network and caused the access system issues.
Installing new bridges with unique SSIDs and connecting the right cable to the right box soon sorted things out.
Have your debugging efforts taken you on strange journeys into unlikely places? If so, start your trip towards becoming an On Call contributor by [10]clicking here to send us an email . ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aEK8J57sa6JUvdGChK1NaQAAAEA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aEK8J57sa6JUvdGChK1NaQAAAEA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aEK8J57sa6JUvdGChK1NaQAAAEA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aEK8J57sa6JUvdGChK1NaQAAAEA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aEK8J57sa6JUvdGChK1NaQAAAEA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/05/30/on_call/
[7] https://www.theregister.com/2025/05/23/on_call/
[8] https://www.theregister.com/2025/05/09/on_call/
[9] https://www.theregister.com/2025/04/18/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
He was pretty zippy at figuring it out.
> Installing new bridges with unique SSIDs and connecting the right cable to the right box soon sorted things out.
You mean nursed it back to life?
aborted...because it spawned packets...in the maternity ward.
Ah, the frantic blinkenlights...
It's always so easy to detect a network loop, but solving it is a whole different matter!
My first question is always "has someone installed a Sonos speaker?", because those bloody overpriced pieces of junk cause network loops by default if you have the audacity to use them with wired ethernet.
I've also had issues with certain mesh-capable access points. A very specific scenario where a cable was bad in a way that allowed PoE to power the device, but the link took ages to negotiate (eventually it did do either 100m or 1g. The device went into mesh-mode because it saw no uplink, then the link came up causing a loop, the device rebooted to return to normal operating mode. On boot the network was down, so it booted back into mesh mode, ad infinitum. That one took a while to figure out.
Also, bless switches with STP, even if they don't fix the network loop, they help you identify the problematic ports saving you *a lot* of troubleshooting with potential downtime.
Re: Ah, the frantic blinkenlights...
Fortunately SONOS has started maliciously breaking their products EXACTLY when the warranty expires....remote bricking of even their newest models with "get lost loser" and "well buy a new one then" being the built-in customer service responses. Their popularity is roughly the same as a job unblocking mcdonalds toilets using a rubber glove....
Company is currently going down in flames internally, and trying desperately to hide its collapse from investors with vague statements and (potentially illegal) fudged financials.
vlans & other segregation config might have hidden the issue
Vlans and other appropriate (broadcast rate limiting springs to mind) segregation config would have alleviated many of those performance issues, and thus Geoffrey roof visit might have never happened.
i'd be uneasy about running unrelated stuff through a switch in a maternity ward, especially without at least vlan segregation but i'm told i have old fashioned out dated methods of working so........
Re: vlans & other segregation config might have hidden the issue
By the time you get to the maternity ward it's a bit late to worry about barrier methods.
Re: vlans & other segregation config might have hidden the issue
Never underestimate the data bandwidth of a teenage boys favourite sock! That helical data storage capacity is huge, and they seem to produce it so regularly!
It's only once they start working that they can afford proper latex barriers!
It is said that 90% of people are caused by accidents!
--------> Yup, the latex one please!
Re: vlans & other segregation config might have hidden the issue
You are to be commended for not lowering the tone TOO far by forming your "arrow" without the leading 8 and with emdashes instead of equals signs.
Re: vlans & other segregation config might have hidden the issue
" It is said that 90% of people are caused by accidents! "
Explains a lot. Sadly almost all were preventable.
Re: vlans & other segregation config might have hidden the issue
There is a new Thanos condom that prevents 50% of them!
Re: vlans & other segregation config might have hidden the issue
Don't rely on ancient mariners. They only stoppeth one in three.
Clearly the car crashing into the barrier had some serious knock-on effects
I'll get me coat
Thank you for that lift. It brightened my day
I'm pleased Geoffrey didn't Bungle it...