Life lesson: Don't delete millions of accounts on the same day you go to the dentist
- Reference: 1736757073
- News link: https://www.theregister.co.uk/2025/01/13/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Blair" who once worked as a Unix sysadmin for a company that ran a colossal PABX that underpinned a mid-sized nation's entire telephone system.
When Blair started the job, the company and the network were in a terrible mess.
[1]
"The datacenter cabling was a rat's nest, the UPS was not networked, there was no monitoring, no documentation, and the list of passwords was on a scrap of stained grid paper," he told us.
[2]
[3]
One thing did work: backups of Oracle databases.
Blair tidied things up, made them more resilient, and eventually reached the point at which he could turn his attention to a documentation system.
[4]
"I created a mediawiki and used LDAP for authentication." This worked well. So well that the company's software engineer asked him to build another to replace the shadow IT wiki he was using.
"He didn't just like the new wiki but also appreciated being able to use the same login as his PC and email," Blair wrote.
The request was then extended to applying LDAP to the customer ticketing system.
[5]
Blair decided that was possible after a simple change that he scheduled for the same day he planned to spend an afternoon with his dentist.
It turned out the change wasn't so simple as switching over to LDAP authentication because the ticketing system deleted all existing users.
[6]Brackets go there ? Oops. That's not where I used them and now things are broken
[7]Coder wrote a bug so bad security guards wanted a word when he arrived at work
[8]Panic at the Cisco tech, thanks to ancient IOS syntax helper that outsmarted itself
[9]NetAdmin learns that wooden chocks, unlike swipe cards, open doors when networks can't
Remember how we said this was a mid-sized country? Blair had deleted millions of users.
He felt something might be awry as while he lay in the dentist's chair, his phone scarcely stopped buzzing.
Blair bolted back to the office and gave himself a crash course in Oracle database recovery.
"I fixed it in the wee hours of the next day, and I confirmed that all users were present and accounted for," he wrote.
A little later, he learned the full horror of his error by finding documentation that warned his plan was dangerous.
"It wasn't in bold or caps, there were no warning symbols or information boxes," Blair argued in his defense. "It was just another sentence in the paragraph: 'Enabling LDAP authentication will delete all existing users'."
Have you ever been unable to escape and fix a mess of your own creation? [10]Click here to send a confession email to Who, Me? so we can issue a fresh life lesson on a future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z4TyVh54Ytz0ztFCF7UqAgAAAAo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z4TyVh54Ytz0ztFCF7UqAgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z4TyVh54Ytz0ztFCF7UqAgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z4TyVh54Ytz0ztFCF7UqAgAAAAo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z4TyVh54Ytz0ztFCF7UqAgAAAAo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/01/06/who_me/
[7] https://www.theregister.com/2024/12/16/who_me/
[8] https://www.theregister.com/2024/12/09/who_me/
[9] https://www.theregister.com/2024/12/02/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: "It was just another sentence"
And test it too. Spin up a test version of the system, ideally with a copy of some real data and then implement the change on the test system to see if it works properly. This should be the case for changes on any critical systems really, always have a test environment you can break and rebuild without impacting production.
Re: "It was just another sentence"
Yeah. RTFM. All of it. And understand it.
With LDAP? All and Understand is at best a tall order either separately or together. Complex, convoluted and in places just plain daft.
But in this case I guess it was the ticketing system that was at fault as it would appear to support only one authentication service at a time (clearly didn't use pam and nss.)
Presumably the fix was to export the user accounts from the Oracle RDBMS, massage into ldif files and import into ldap or alternatively knock up a gateway from ldap to the existing Oracle system
Re: "It was just another sentence"
I couldn't agree more. But... the warnings come after the spell!
Re: "It was just another sentence"
The corollary of this is: When you write documentation, assume that if anyone even looks at it, they probably won't read it all, so at least highlight important points.
Re: "It was just another sentence"
"And missing one sentence is an easy thing to do when what you are reading is about as exciting as watching paint dry."
My job once involved watching paint dry - and it was actually quite exciting. Unfortunately, I'm not allowed to say where it was...
Life lesson...
"The request was then extended to applying LDAP to the customer ticketing system"
if the Ticketing System was home-made, he should've asked the developer(s) on the best way to do it, if it was commercial (as it seems the case, since there was a documentation) he should've asked the company that made that thing and was also providing support (I assume somebody that run a "nation-sized" system does also pay for support for his wares).
So life lesson: ASK THE F*ING QUESTION before "assuming".
Re: Life lesson...
"So life lesson: ASK THE F*ING QUESTION before "assuming"."
Sometimes that's the case, but life and most work is based on assumptions and generalisations. As a primitive form of learning they are arguably these are the very foundations of civilisation. Logically, would you expect or even harbour reasonable doubt that implementing better authentication to delete all existing user records? Really? Somebody, somewhere thought "if a sysadmin implements LADP, then the system should delete all user records". Now there's the fuckwit responsible here, not "Blair".
And the author of the shitty manual that doesn't make clear the huge consequences. Hiding something of huge consequence as though it's trivial is a grade A failure.
I suspect that the stupid, stupid idea of deleting all users was only ever tested on a small dev system that it was trivially easy to roll back and forward, and where there were no operational consequences or angry managers.
Auto-Account Deletion
Ut ullam impedit amet et. Harum voluptatum debitis itaque corrupti libero dolorum. Doloribus nihil sit laborum. Laborum amet est est. Qui quo in voluptas maxime doloremque cumque voluptas. Officia sequi voluptates placeat. Impedit non et minima. Veniam doloremque qui dolorem excepturi autem debitis. Minima voluptatibus magni aperiam temporibus atque ullam iure. Aut dolores unde enim odit cupiditate non numquam et. Praesentium ut veniam quis ut voluptatem inventore at. Voluptate voluptas omnis a qui temporibus. Tempore facilis tenetur consequuntur. Quod voluptates sed inventore blanditiis. Sint voluptatibus optio enim incidunt deleniti nostrum. Molestias corporis et corrupti velit eius natus. Sint delectus rerum praesentium nemo. Et dicta odit dolor. Quos repellendus et saepe consequatur rem molestiae nesciunt. If you switch to ldap all accounts will be completely deleted because this is a feature. Soluta beatae perferendis sed. Est temporibus natus ab odio tenetur nemo alias. Ex sed consectetur possimus quia animi. Doloribus nihil cumque commodi recusandae id dolores quisquam cupididate.
Re: Auto-Account Deletion
I should try this with my next annual performance review
Lorum ipsum I deserve a one hundred percent payrise dolor sit amet
Re: Life lesson...
Or maybe an "are you sure you want to nuke your user records?" popup? (or y/N prompt in CLI).
They get abused for trivial actions too often, but i feel like the possible consequences here are severe enough for that...
Along with the fact that configuration changes generally shouldn't be deleting user data in the first place. Is it really necessary to delete all user records instead of creating a users_ldap table, or creating a "authentication source" column? oh my, did i just accidentally create a feature to use multiple auth sources simultaneously too??
He felt something might be awry as while he lay in the dentist's chair, his phone scarcely stopped buzzing.
Blair bolted back to the office and gave himself a crash course in Oracle database recovery.
So he went back and gave the database a filling?
RMAN, that was a good story...
Getting Oracle to work properly is like pulling teeth
Getting Oracle licensed properly without overpaying like they want to trick you into is like pulling teeth.
According to Oracle, there is only underpayment they can prove, and underpayment they are currently unable to prove (don't worry, they will at some point).
What is the 'overpayment' you speak of?
I have been in the situation of trying to understand Oracle restore, and get it working for a system migration ...... and I'd rather have been in the Dentist's chair .... having root canal work .... without anaesthetic!
----------> The icon for root canal work!
I am currently dealing with a supplier who uses MFA ( the TOTP kind) for me to logon and create certificates for our machines to access our data in their "lake"
The MFA has out of the blue stopped working despite me having a record of the initial secret key .
Their only solution to this is to delete my account and recreate it , which will invalidate the certificates i've crated and validated and signed etc for our servers to talk .
As a password reset solution I find this less than satisfactory.
They are also unable to answer any questions on why this happened , or why they cant just reset the MFA , or how exactly the MFA works as "a third party is providing the MFA"
This is a very large NHS healthcare provider .
sorry for the off topic rant , this is pissing me off no end - the link is "authentication" I guess .
LDAP?
Is this the thing that lets me login on 3 different apps my employer uses with the same password, but when I change my password, Google pw manager will ask me to type the new password on all of them?
Is this how any of this is supposed to work? For my employer, only one change and all the apps obey the new password, but for me, I have to type it again on every app, it just doesn't read my new credentials and let me through?
Isn't it supposed to skip every login prompt down the road once I'm logged back in with the new pw? What's the point of unifying all the passwords if you have to type them (or Google asks to update it for me) every time?
I would use the Windows user icon, but you know...
TLDR I have a grievance about typing passwords over and over on several places, if they are synchronized. Yeah, token and man-in-the-middle and all that.
"It was just another sentence"
Yeah. RTFM. All of it.
And understand it.
At least he was able to recover the issue. Good on him for that. And missing one sentence is an easy thing to do when what you are reading is about as exciting as watching paint dry.
But as an admin, what your doing is important (at least, the consequences may be). So read that manual.