The future is now, old man: Let the young guns show how to properly cock things up
- Reference: 1621236609
- News link: https://www.theregister.co.uk/2021/05/17/who_me/
- Source link:
"Al", for that is not his name, was looking forward to a well-earned retirement after a career spent at an IT giant working on everything from compilers and operating systems to firmware and networking.
Faced with a future revolving around daytime television, Al decided to keep his hand in by taking on the role of a part-time IT manager at his local GP practice (usually the first port of call for Brits seeking healthcare).
[2]
[3]
[4]
The practice also had a full-time support person dubbed by Al as the " [5]PFY " who we have Regomised as "Ernie."
Al and Ernie were tasked with replacing the telephone exchange, "something which I knew little about," Al admitted. However, thanks to the assistance of the supplier and an engineer, all went well. Perhaps a little too well, instilling possibly a little bit of over-confidence.
A few weeks later a small change was needed. Rather than thrash blindly into the carefully crafted system, Al and Ernie consulted the documentation. It seemed straightforward enough: "We decided we could do it ourselves although the process was fairly intricate."
[6]Accidentally wiped an app's directory? Hey, just play the 'unscheduled maintenance' card. Now you're a hero
[7]Terminal trickery, or how to improve a novel immeasurably
[8]Don't cross the team tasked with policing the surfing habits of California's teens
[9]You want a reboot? I'll give you a reboot! Happy now?
Age and experience on his side, Al was cautious. There were no test rigs on which to validate the change. "I was nervous of making the change to a live system and wanted to give it more thought," he said.
Ernie, with the misplaced confidence of youth, had no such worries and had already charged in like a bull in a data centre.
[10]
Readers will be unsurprised to learn that seconds after the young chap had pressed the go button, every phone in the practice stopped working.
"Turns out," said Al, taking partial ownership for the cock up, "we'd replaced the entire programming with our small change!"
Thankfully, the surgery had a backup plan for handling incoming and outgoing calls, but how were Al and Ernie to reprogram the entire exchange? Particularly in light of the catastrophe created by their tiny little change?
[11]
There may have been a plan for the physical phones, but no backup existed for the programming. The supplier hadn't taken one either. One would have hoped that someone with Al's experience might have sorted such a thing, but we can imagine any suggestion was short-circuited by the enthusiasm of the PFY.
In desperation, the original engineer was contacted and... yes! Likely as a result of long experience with idiot users, the engineer had thought to take a copy of the configuration during the installation and still had it on a diskette. He may have been 200 miles away, but thanks to the marvel of modern communication was able to connect to Al's exchange and upload it.
"Crisis over," signed Al.
As for the fallout, he and Ernie were given a telling-off by the practice manager and made to promise never ever to do it again. The supplier would deal with changes in the future. Ernie, the PFY of this story, moved on a few months later.
"It wasn't until years later," said Al, "it occurred to me that the engineer interface into the exchange would have been a prime hacking target for obtaining free calls and anonymity on the phone network."
Times were a bit more innocent 20 years ago.
We've had accidental file deletion. Now we've had accidental phone deletion. Can you go one better? Maybe you accidentally [12]wiped a bunch of records from the Police National Computer ? Confess all in an email to [13]Who, Me?
Regomised anonymity assured. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YKI@wuT8WQlz7@X3lFYF1AAAAIg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKI@wuT8WQlz7@X3lFYF1AAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKI@wuT8WQlz7@X3lFYF1AAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/offbeat/bofh/
[6] https://www.theregister.com/2021/05/10/who_me/
[7] https://www.theregister.com/2021/05/03/who_me/
[8] https://www.theregister.com/2021/04/26/who_me/
[9] https://www.theregister.com/2021/04/19/who_me/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKI@wuT8WQlz7@X3lFYF1AAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKI@wuT8WQlz7@X3lFYF1AAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2021/01/15/pnc_records_deleted/
[13] mailto:whome@theregister.com
[14] https://whitepapers.theregister.com/
Re: When network becomes notwork.
Sounds like his experience came at your expense. :-)
Police Computer
Since you mentioned the Police National Computer ... years ago I worked on a project to support access to the system from some new boxes. Why that project was needed is a story in its own right, thanks to a salesman "misunderstanding" the requirements, selling something totally wrong for the network, and then we had to bail him out.
But in order to test and demo our work, we had to send queries to the real PNC. But because of security/privacy concerns, the only query I was allowed to run was to lookup my own car registration.Unfortunately for readers of the Reg it didn't show as stolen or written off. But I did wonder if analytics (even in those days they did some, although they might not call it that) would show a concern about repeated queries for a particular vehicle.
Re: Police Computer
"thanks to a salesman "misunderstanding" the requirements, selling something totally wrong for the network"
You mean unloading an obsolete bit of kit that provided him with a large commission for shifting the archaic boat anchor?
Seen it all to many times ... Yet another reason why Management and Marketing should have absolutely no hand whatsoever in purchasing new networking kit.
Re: Police Computer
I used to do support and installation work for a company writing HSE training software. Sold to big councils, police forces, fire services, banks, etc.
Usually my first task on site with a customer was finding out what special extras the salesman had promised but forgotten to mention to the programmers. The little changes like promising a Welsh Police force to have everything translated into a bi-lingual edition. So on the Go Live \ Install day I was usually explaining to the devs of the changes that had to be hacked into the code ASAP.
As to security - haha, don't make me laugh. This was the kind of developer who wanted to always leave every folder fully writable to all on the web application he was writing.
Re: Police Computer
This is, sadly, rather common.
I've always tried to make developers develop on systems that are configured as they would be in production.
Salespeople just chase the money. Users want the latest wizzy crap, regardless of whether its actually supported or not. Developers just want to make it work. Security want it to work securely. Support teams just want it not to fail.
Re: Police Computer
I worked for the vendor who used to supply the PNC and one of our salesmen on the account was likeable enough but managed to find himself on the front page of the Sun for walking out of the PNC with a removable disk pack full of data under his arm. Without going through the formalities of seeking any permission to do so. During the Falklands war.
Proably the same bloke as the one who sold the unsuitable kit.
An age ago. Or two.
"it occurred to me that the engineer interface into the exchange would have been a prime hacking target for obtaining free calls and anonymity on the phone network."
Yes, it was. That's why sensible people had it locked down unless it needed to be opened for a short time ... and then locked it down again immediately the need passed. Sometimes this was as simple as unplugging the modem attached to the executive port ...
"Times were a bit more innocent 20 years ago."
No, no they weren't. That kind of innocence and naivety was lost by the early 1980s ... The Morris Worm of 1988 was a boot to the head of those who hadn't got the picture quite yet. Any hold-outs with this kind of open system by the year 2000 were idiots whistling past the graveyard.
YADM ... Yet Another Doh Moment ...
I had recently managed moving our Siemens PBX, Telephony servers & call centre servers into a shiny new server room. I had purchased a network card for the system's UPS so we I could monitor it along with all the other services (knowing something is misbehaving 30 seconds before the users being a Good Thing (tm)).
So I rang the suppliers techies and quized them on procedure, did the change thing, as no downtime was needed I proceded on a Monday morning with a Sunday hangover. I put the UPS in to pass-thru mode prior to fitting the card ... Ker-Thump!!! ... both racks went down. It took me most of the day to recover everything.
Basically our sparkies had not wired in the pass-thru. I was quized by the IT Director and the Finance Director and while they agreed the outage was not my fault I was unfortunately the PM on the system move so I should have checked the sparkies work. Obviously the dept thought it was hilarious.
A bad day ended better, went to the pub.
One tiny change
Worked on a very large product with a very large number of users. One developer decided to sneak in 'one tiny change' as we were finishing up a service pack. Cue 220k users having to uninstall the service pack and an emergency service pack released the next week.
Regomised
Shame the Regomiser didn't pick "Al" as "Bert" then the story would have been Bert and Ernie
Backup always
Or pay the price. We've all been there. But you should never go there more than once.
Back door in to the comms systems
I do recall when I was Application Support Manager for a sizeable telecoms company and found, quite by accident, that the installation engineers had deliberately installed a support line in to the main switch. The switch it self was HUGE, thing mainframe and add extra cabinets. These forward thinkingguys had put this modem in so they could remote dial in to the system and patch any faults. But they hadn't thought of any security. None whatsoever. You dialled in to the number and you had a console, albeit at 9600 baud, directly in to the internals of the switch operating system. From this you could do anything you wanted: Want a new number? Easy. A free-for-life phone line? Simples. Whats more, this went in at such a high level nothing was even logged so we never knew if this was hacked.
Re: Back door in to the comms systems
That's why we used dial-back modems.
Cognizant Engineer dials in, enters "password" for modem. Modem hangs up. Modem looks up "password" in a list, and dials back the appropriate number. Cognizant Engineer's modem answers, makes the connection, and is then presented with a login for whatever bit of gear needs twiddling, this time (hopefully!) with a proper login/password pair. Once set up, it's easier to do than to describe. Side benefit is the owner of the bit of kit being worked on pays the long-distance bills.
Just move some disks...
I heard that some young chaps wanted to move some solid state disks about 6 inches without powering it down. These were about 5ft high and 3 ft square and heavy. So the lads following the best health and safety procedures put their backs against it, and pushed with their legs. This was fine, till one of them found he had caught his belt under the recessed emergency power-off "pull knob" and did an emergency power off as he stood up!
Re: Just move some disks...
If you knew how, you could program them to walk by themselves.
When I was at DEC, one of the guys learned to make the washing machine sized disk drives "walk" across the floor ... I had to fire him when he did it in front of Ken Olsen, who was visiting our lab. Was very hard on the hardware.
Re: Just move some disks...
I've seen something similar happen on an ICL2900 mainframe many, many moons ago. Guy managed to catch a reboot switch with the cuff of his suit jacket while moving a terminal around. 3 days it took to get the server back up!
I never wore a suit jacket or had my sleeves down in a machine room after that.
I remember it like it happened yesterday - but that was over 35 years ago. I feel *old*!
When network becomes notwork.
Had a very experienced network engineer make a "small" change that took out half the network (one that he mostly built).
Fine - we all make mistakes. His was not having any change paperwork raised (despite a company wide email stating that no paperwork meant no change).
But rules are rules, right? So we lost an experienced network engineer....