DBA heroes don't always wear capes. Sometimes they just have a bunch of forgotten permissions
- Reference: 1612767486
- News link: https://www.theregister.co.uk/2021/02/08/who_me/
- Source link:
"Joff" submitted today's story, which takes us back to the mid-2000s and the divesture of his company from a US multinational. He was running a team tasked with producing interface processing systems that needed to collate data from a multitude of information sources. That data, he said, "then passed to various other internal and external systems for processing once the finance departments in multiple countries had done their bits."
[2]
Data wrangling like that can be such a joy and, sensibly, "We determined that the optimal design was a database to verify all information was correctly pre-processed before any onward transmission."
A logical decision when faced with massaging multiple data sources into something that made sense. As is so often the case, timelines were tight but Joff's team managed to get through the design, build and test phases on schedule. Doubtless with the assistance of caffeine and the odd pizza or two.
[3]
"As an ex-techie now in management," Joff added, "I could appreciate the work they did and tried to be as involved in the design as much as I could without disrupting them too much."
A rare breed in the boss stakes, for sure. Then again, this is his story after all.
Development and validation complete, the team were ready to put the database on the production servers. While they might have been DBAs on the build and test environments, the reins for production were firmly in the hands of the corporate IT department.
Still, there were weeks to go until go-live. A request to create the database, setup the backups and so on was submitted.
"And we waited ... and waited ... and chased ... and waited..."
The clocked ticked down to the 48-hour mark, and the database had still not been created. Joff and all bar one of the team were at their wit's end. All bar one? There was an individual, however, who could pull on the cape of DBA power and save the day.
"One of my team," explained Joff, "came and admitted that during a major debugging exercise some months before he had been given extra permissions."
The team member claimed (to Joff's satisfaction) that he had both not realised he still had the permissions and also that he had not abused them.
However, he could use these permissions to create the needed database, although was a bit worried about how their then corporate overlords might react.
Joff pondered. Should they keep waiting and risk delaying the go-live, or proceed? He gave it another 24 hours before hitting the go button and agreeing to take any flak that might come the team's way.
All went well. The database was created ready for go-live the following day and Joff's team got on with the required pre-population work ahead of the data floodgates being opened.
Two hours later the phone rang. The corporate DBA team were incandescent with rage. They had tried to create the database, as requested two weeks ago, and it was already there.
All permissions were to be immediately revoked. This would also slam on the brakes for the go-live.
Joff took the phone and, we like to think, held it arm's length while the ranting went on. When he was finally able to sneak in some words, he told the caller: "look we are in the middle of a major go live at the moment. I understand your concerns, but can we talk about this after the weekend when we are done with this important work..."
The anger-froth continued to spout for a few more minutes until eventually agreement was reached and Joff's team could proceed.
The inevitable follow-up did indeed come in the next week. The corporate DBA team were back ranting on the phone and told Joff that all access was to be removed from his team.
Except... after that weekend the ownership of the server had shifted. (Remember that divesture?) So.... any changes would need to be decided by both parties and Joff certainly wasn't about to relinquish any permissions.
"In fact," he told the corporate team, "from now we would be needing to pre-vet any actions that they were recommending."
Ouch.
"I never heard anything back on the matter."
[4]
Ever found yourself taking matters into your own hands in order to save the day? Or been on the receiving end of fury-spewing after you got found out? Share your confession with an email to [5]Who, Me? ®
Get our [6]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_bootnotes/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YCEZy-zgzmfWtohhDr-7dAAAAA4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_bootnotes/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YCEZy-zgzmfWtohhDr-7dAAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_bootnotes/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YCEZy-zgzmfWtohhDr-7dAAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] mailto:whome@theregister.com
[6] https://whitepapers.theregister.com/
As an ex-techie new to management it may even be true.
Ouch. Memory trigger
Mentioned it before I think.
The great Manchester Comms fire.
I was first in that day and couldn't reach anyone on the DR teams.
So took the decision myself to shift all linklines to other offices then waited for the fallout.
Luckily we were trialling Sametime so at least had some semblance of real time Comms given even mobile networks were down.
All worked out in the end.
By lunchtime I had a single working phone line courtesy of Otis lifts the floor below who were on what's now Virgin with a cable strung out the window to the floor below.
2 days later my colleague at our main site in Bournemouth had to crash a DR meeting to explain that just because the Manchester office could call ourselves didn't mean we could move everything back!
Happy days. Not...
Dave
Re: Ouch. Memory trigger
Although...
Having the only working phone line does lead to your desk becoming very very popular.
I remember the day our Security team lectured us about the importance of system security (can't remember the exact kerfuffle at the time, but it turned out to be a non-issue anyway).
Queue 2 days later and they are in a panic. A change was made on one of their systems (in relation to aforementioned kerfuffle) and they had locked themselves out of the root account - which they needed access to.
They call me out of desperation, and it takes me all of 5 minutes to identify a root owned 777 permissions file in init.d which I used to hack into the system....
They were actually really good guys though, and we all had a good laugh about it.
Cue
Nice solution to the problem, but things like this have a tendency to backfire the next time. I would have documented the request and every communication to the DBA team to get the database up in time (with proper date and time) and let the deadline whoosh by. In the subsequent auto da fé the DBA team would be for the pyre.
Better than letting a developer create a DB in the wrong place or with the wrong options which then brings the whole server down. OK, perhaps it was all done correctly but escalation seems a much better approach than bodging.
Not just document, but also start raising it up the command chain.
So as the days go by without a response - you pass a message to your management that you've had no response. And you escalate the tone of those messages - so a few days beforehand, you are telling your management that you will not be able to go live "because this other department are not co-operating".
That way, you have evidence that you did all you could to make things happen, AND you can document that your management knew there was a problem, AND you can document that you did everything within your power/authority to make things happen. If the brown stuff does hit the air movement device, then you should be out of the firing line - or at least you've made yourself a good umbrella. But hopefully, if your management is any good, they'll have made things happen and the problem will have been resolved before that happens.
It's not mentioned in the article, but it's also good to make sure that in your request it is clear what the timescale is - you don't want the other side to be able to use the excuse that they weren't aware of when it was needed/it was urgent/whatever.
I've found that works in some companies, but not in others. One company I used to work for was so good at that approach that very little got done in any decent timescale because of the sometimes-circular blame-chains that developed (some links justifiable, some just well-worded excuses).
Another company almost had the opposite problem. If something landed in your lap and there wasn't enough time to get the appropriate people involved, it was your job to fill the gaps to the best of your ability. No excuses. I've set sites live before with no input from the persons who usually supply content, imagery and design, nor any legal oversight on the various policies, terms and conditions that were required (all requested several times over a two-week period). At point of set-live, I made it quite clear to every senior manager involved in the project what I had done, why I had done it, and what really needed to be reviewed and by whom at the earliest opportunity.
A month later, still no changes.
Created already
Don't leave us hanging - why were the DBA team able to detect abuse of permissions nearly instantaneously, but not able to detect multiple emails asking when the database would be ready?
I've had similar in the past: you ask for something to be created and the team seem to think that creating said item is the end of their responsibility and that telepathy or osmosis will serve to get the information needed to access the new resource to you.
Re: Created already
I got the impression from the article that they'd finally decided to act on the two-week-old requests and create the database. Then found that it had already been created and got stroppy about being asked to do something that had already been done.
> A rare breed in the boss stakes, for sure. Then again, this is his story after all.
Very nicely put ... well done