BOFH: Well, we did tell you to keep the BitLocker keys safe
- Reference: 1721982609
- News link: https://www.theregister.co.uk/2024/07/26/bofh_2024_episode_14/
- Source link:
"Still bad," I say.
"But haven't you done the fix?"
[1]
"We're still working on it," I say, not looking up from my screen.
[2]
[3]
The PFY, on the other hand, is wordlessly focused on scribbling away at a sheet of paper.
"How long do you think it will be?" he asks.
[4]
"As I told you before, this level of recovery – on so many systems – it could take days."
"Yes, but how far through the process are we?"
"It's too soon to tell."
[5]
"Well, can't you give me something? The board are wanting to know when things will be back to normal."
"I don't know – but the more distractions I have, the longer it's going to take."
"But it's been days!" the Boss says.
"It has – but it would have been considerably shorter than that if someone hadn't lost the BitLocker keys."
"I didn't know they were important. You give me lots of bits of paper!"
"Yes, but I don't give you lots of pieces of paper and ask you to put them into safe keeping."
"I... I'll check back later," the Boss says, scurrying off.
It's true, shutting down all your servers and blaming it on a global outage is a shameful thing to do – but it was too good an opportunity to miss. Obviously, we had to invent an "internal update cache" to explain why we'd not been taken down at the same time as everyone else, but it seems to have played out OK.
And because the Boss has "lost" our "BitLocker keys," we have to do "recoveries." At the time, giving him a bunch of BitLocker keys and telling him to "keep them safe" was just some spade work for an as-yet-unknown future event. We knew he was bound to lose them but stealing them off his desk later that evening was just the belts and braces approach of a true professional...
The event does, however, highlight how little anyone knows about our IT services. No one seems to know the software we're using to protect our "servers," what "servers" we have (and how few of them there now are), nor how many of our services are SaaS.
Every cloud service has a gold-plated lining, and we intend to fully mine that lining starting with meal allowances, and ending with an overtime bill that will likely fund a holiday tour of the breweries of Belgium...
...
"Success!" I say to the Boss on his next visit. "I've brought up our accounting server!"
"Oh, the finance department will be pleased."
"No, no, I meant OUR accounting server. The one we use for tracking overtime."
"What about the financials system?!"
"Oh, that's a while away yet – I'm bringing them up in alphabetical order."
"NO!" the Boss snaps. "We need finance up FIRST, before anything else."
"But wouldn't you rather we starte-"
"RECOVER FINANCE!" he snaps.
"But I'm halfway throu-"
"FINANCE!"
"I... OK," I respond.
...Three hours later...
"It's up," I gasp, appearing at the Boss's door after cloaking myself in the demeanor of someone who's spent the last three hours of his life digging rock from a quarry by hand, rather than watching YouTube videos on the current crisis to get some believable buzzwords.
[6]BOFH : It's not generative AI at all, it's degenerate AI
[7]BOFH : Why's the network so slow?
[8]BOFH : An 'AI PC' for an Acutely Ignorant user
[9]BOFH : Come on down to the dunge– erm … basement
"Excellent. I'll let them know," he says.
"Righto," I sigh, wandering "wearily" back to Mission Control.
I'm barely there when the Boss turns up again. "Financials isn't working," he says.
"Sure it is, I checked it out, it's up and running."
"No, no one can get into it."
"Well, they wouldn't, would they?"
"Why not?"
"Because I need to recover the authentication server. But you made me stop that and start on finance."
"What?"
"The authentication server. Active Directory. Comes after accounting."
"You should have told me that!" he snaps.
"You wouldn't let me."
"This is bad!" the Boss says.
"For you," the PFY adds.
"What?"
"It's bad for you," the PFY says. "I mean you lost the BitLocker keys; you've delayed the recovery of our systems; and any time now someone's going to be looking at who signed off our server security policy."
"Well, I didn't sign it!" the Boss blurts.
"Sure you did!" the PFY says. "I've got the contract right here. Careful, the ink's still wet."
You've got to hand it to the PFY – any opportunity to perfect his forgery skills.
"Give me that!" he snarls.
"I could, but who knows what the next contract might say."
"What do you want?" he seethes.
"For starters, I think our overtime covered."
"And?"
"I think we need to learn more about recovering critical systems. I hear there's a great course in... uhm... Atlanta. And I think Simon's quite keen on attending a week-long series of recovery workshops in, uh... Belgium?"
[10]BOFH: The whole shebang
[11]The Compleat BOFH Archives 95-99
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqNzxpU7C0V72M0l2qBb6wAAANE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqNzxpU7C0V72M0l2qBb6wAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqNzxpU7C0V72M0l2qBb6wAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqNzxpU7C0V72M0l2qBb6wAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqNzxpU7C0V72M0l2qBb6wAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/07/12/bofh_2024_episode_13/
[7] https://www.theregister.com/2024/06/21/bofh_2024_episode_12/
[8] https://www.theregister.com/2024/06/14/bofh_2024_episode_11/
[9] https://www.theregister.com/2024/05/24/bofh_2024_episode_10/
[10] https://www.theregister.com/data_centre/bofh/
[11] http://www.bofharchive.com/
[12] https://whitepapers.theregister.com/
Re: recovery workshops in, uh... Belgium?"
Belgium is a win all round - wine, beer, excellent local food. And chocolate to bring back to keep the folks back home sweet too.
Bring it on!
I'm sure none of us have ever blamed our own fuck-ups on a well publicised but unconnected problem right?
;)
I'm sure none of us have ever blamed our own fuck-ups on a well publicised but unconnected problem right?
Of course not! Unfortunately the proof of my excellence and diligence is currently on a train, stuck just outside Paris. There's been an attack on TGV system - but I'm sure it'll be here shortly.
Would you mind paying the invoice in cash? Unfortunately my bank are currently unable to accept payments due to CloudStrike.
More reliable than buses
It is indeed fortunate that 'unconnected problems' have a feature of being regularly available. Like buses, often in multiples at the same time.
And on a Friday! ----->
I'm sure none of us have ever blamed our own fuck-ups on a well publicised but unconnected problem right?
Early-/mid- 2000s I was doing some acceptance tests for a system. To be honest, it wasn't as mature as it should have been and I would have liked for us to have had a bit more time to work on it before subjecting it to testing in front of this customer. As I feared, during testing some errors occurred, commensurate with one of our components not processing data sent to it over the network.
Around this time there was a period of fairly intense solar activity - enough to get a bit of coverage in the press because aurora sightings were predicted for England. I took a punt and decided to blame the problem as being due to loss or corruption of data over the wireless network due to cosmic rays.
Unbelievably the customer bought it, and we agreed to reconvene for testing in a few weeks' time, once those pesky sunspots had calmed down. That bought us enough time to fix our software, and it passed the rescheduled test with nary a hitch.
We're currently nearing solar maximum, just saying. Don't waste those precious opportunities (and not just the ones to fill social media with blurry photos of the sky at night, again).
@phuzz
Still working occasionally on a particularly intractably Y2K issue... Fix expected by 2038.
Moving to cloud?
Even the most obscure clouds can be mined for their silver lining - with careful planning :)
Re: Moving to cloud?
[1]Obscured by Clouds
[1] https://www.youtube.com/watch?v=B0Fq_22cfFA
Balanced view
I've looked at clouds from both sides, now.
Executive summary: resilience similar to ice cream castles in the air, full report for the board follows.
Genius - Ocean's Two
We knew he was bound to lose them but stealing them off his desk later that evening was just the belts and braces approach of a true professional...
recovery workshops in, uh... Belgium?"
I'm sure the implied meaning is breweries but I've had occasion to attend some conferences of the Belgian Digital Infrastructure Association and the wine flowed freely. I recommend.