Techie wiped a server, nobody noticed, so a customer kept paying for six months
- Reference: 1678696327
- News link: https://www.theregister.co.uk/2023/03/13/who_me/
- Source link:
This week, meet a reader who asked to be Regomized as “Sam” and once worked at a company that, thanks to a series of acquisitions, had a diverse portfolio of infrastructure and apps it was migrating into a consolidated datacenter.
One of the apps handled what Sam described as “billing for inspections.”
[1]
“Plans were drawn to migrate the system into existing billing platforms,” Sam recounted. But as often happens when companies consolidate, staff with important knowledge fall were let go without care.
[2]
[3]
That mean the billing app’s sole developer was surplus to requirements but had already been hired as a contractor to assist with " some emergency temporary repairs" while the larger development team worked on the migration.
Two years later the app was still running on its original server, and Sam got word it was time for that box to move to another role.
[4]
“After a few emails confirming with the rest of IT that this server was no longer in production - because we surely did that migration - I removed it, wiped it, un-racked it, physically moved buildings, and repurposed it for an internal system,” Sam told Who, Me?
And then he thought nothing of it for six months, until he received a message about not being able to connect to the long-since-repurposed server.
“A scramble ensued,” Sam said, as he and colleagues checked the app had indeed been migrated.
[5]Don't worry, that system's not actually active – oh, wait …
[6]Backup tech felt the need – the need for speed. And pastries and Tomb Raider
[7]A tip for content filter evaluators: erase the list of sites you tested, don't share them on 100 PCs
[8]Learn the art of malicious compliance: doing exactly what you were asked, even when it's wrong
It had not, and nobody knew why.
Nasty questions followed.
[9]
Why wasn’t it migrated?
Do we have backups?
What do you mean when you say “There are no backups”?
Why did it take six months for someone to report this app was down?
The best question of all was: “How long have we been billing clients for this?”
The sole developer who understood the app was again asked to help, and according to Sam “told the company to buzz off and that he would not help us for any amount of money.”
At which point the migration project shot to the top of the development team’s to-do list and was given all the resources needed to get the job done fast.
“It was integrated into the existing system in just a few months, agile timelines be damned,” Sam said.
Sam told us he was not blamed for this mess. “I had done all my checks first,” he said.
But he’s still a little down on himself for not quite doing all he could to ensure the server was not indispensable.
“I guess it goes to show, no matter how many times you check, always rely on that powered-down ‘what does it break’ analysis before pulling something out, no matter how many people say you're in the clear,” he told Who, Me?
If you have a similar tale of a resource being removed without anyone noticing, [10]click here to send Who, Me? And email and it might be your story kicking of another week of Reg fun. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZA8CVlUHml2c4XFEFnZpOAAAAFY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZA8CVlUHml2c4XFEFnZpOAAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZA8CVlUHml2c4XFEFnZpOAAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZA8CVlUHml2c4XFEFnZpOAAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/03/06/who_me/
[6] https://www.theregister.com/2023/02/27/who_me/
[7] https://www.theregister.com/2023/02/20/who_me/
[8] https://www.theregister.com/2023/02/13/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZA8CVlUHml2c4XFEFnZpOAAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Who ate the editor?
ChatGPT ate the editor, of course.
Re: Who ate the editor?
ChatGPT ate assimilated the editor, of course.
Resistance is futile.
It was migrated!
Wow, similar situation, the application previously used a small executable on the client site. It was migrated by the vendor to an IIS(eww) server with a banal webpage. The trainer came, the users were trained. It was out of our hands. Sometime later(months?) the server was decommissioned, but not wiped!* A few days later the roars from the finance department were heard all over the office**... We, as in, me and my manager marched in, pulled up the webpage for the new banal IIS webpage, and some strange looks were exchanged between the finance department plebs(I'm sure coins were dropping). During the following days, it was discovered that during the training the cut over date wasn't really defined. So the users kept using the old system. In the end, the vendor was able to extract the months? of work from the old system, and massage it into the new system.
* One colleague wanted it for a test bed, but was too busy to pull the trigger!
**I'm pretty sure that was the day my hair started turning grey.
I feel older just talking about it!
Turning off an waiting...
...is a great tactic but I doubt it would have worked given no one noticed for six months.
Obvious which company this is
"staff with important knowledge fall were let go without care"
It's Twitter, isn't it?
Re: Obvious which company this is
SOP at the company I work for. Fire anyone with specific knowledge, some weeks later "lift and shift" migration, then some weeks later have the offshore support team start the support.
My advice to colleagues, "don't lift a finger, don't go the extra mile. The upper management arsed this up, let them wallow. The headline money saved won't reward you, but will help them earn their bonuses and the expense of your unpaid overtime". As a professional it stank, as a lesson learning exercise, it was gold.
Who ate the editor?
"staff with important knowledge fall were let go without care"
"That mean the billing app’s sole developer was surplus to requirements"
Almost as bad as my first emails of the week.
And OT, that's why I always take a backup before deleting *anything* (GDPR - whassat?). LTO FTW.