BOFH: There's a fatal error in the blinkenlights
- Reference: 1744367232
- News link: https://www.theregister.co.uk/2025/04/11/bofh_2025_episode_7/
- Source link:
Digging deeper we find that it's a personal favor, not work-related in any way.
"Five Pints," the PFY says.
[1]
"Each," I add.
[2]
[3]
The Boss makes a show of pretending to think this over carefully, then reluctantly agrees. The company concerned is an investment outfit that the Boss has his retirement money with. Whilst the returns are good, their technical problems have rapidly eroded the Boss' confidence to the point that he wants us to get them running long enough for him to get his cash out. Today.
"Oh, right!" I say. "Totally understand. Ten Pints."
[4]
"We agreed Five!" the Boss quibbles.
"You know, I think we have that six-week long I.T. inventory to do," the PFY suggests. "It's quite urgent too. Perhaps we should get onto that instead?"
...
[5]
A deal is struck, and the PFY and I grab a cab to the company. Their resident geek, told to expect us, is waiting at the door.
"It's the main server we're having problems with," he says, as we ride up in the lift with several power-dressed investment types.
"Just through here," he says, leading us down a maze of twisting corridors, all alike.
The server room is small but amazing. Four racks of gleaming equipment humming away, connected in cable via streamlined cable management.
...
"So this is all a front?" I ask, after a minute or so of looking around.
"What?!" he asks.
"The racks, the servers - it's all a big fake?"
"I don't know what you're talking about," he says, confused.
"Please. I mean look at that cabling. It looks like it was put in by an OCD architect."
"And?"
"And that never happens. I mean it might look like that on Day One, but after a couple of years it looks like an explosion at a spaghetti factory. And look at your hardware."
BOFH : Don't threaten us with a good time – ensure it [6]READ MORE
"What about our hardware?" he gasps.
"None of them have their covers half off. Besides, you've got four G4 DL320s in there. Sure, the front bezels have been polished, but they're over 15 years old."
"There's a proliant 3000 up there," the PFY chips in, pointing at another old piece of kit with a red LED glowing angrily.
"Oooh look!" I say. "An aftermarket LED 'upgrade'."
...
A minute of silence passes, then finally the geek cracks. There's no server hardware. Nothing. Over the last five years the entire company operation has moved into online services - theoretically leaving our geek with no job.
"So what do you... do all day?" the PFY asks.
"SOME days, I'll take a complete snapshot of our cloud infrastructure," he says.
"Once a month you mean?" the PFY surmises. "So what do you do with the rest of your time?"
"I, um, manufacture outages," he admits.
"Manufacture outages?"
"Yeah, I'll light up the RED lamp on a server and, uh, take a cloud service offline."
"Why?"
"Because then they'll call me and get me to fix it. I'll bring them in here, fire up a linux laptop with the Matrix screensaver, edit a JPEG with a Hex editor, pretend to find a virus signature or an internal consistency error, then 'fix' it and bring the service back online again."
It seems so simple now that he says it.
"And no one's rumbled you?"
"They're too busy trading. None of them knows where the data lives, what services they're subscribed to, or what the underlying infrastructure is. Two weeks ago I told them the enterprise license for the English Language keyboard was about to expire and they'd either have to pay the £2,000 licence fee or they'd automtically be switched to the open source Turkish keyboard."
"And they okayed two grand?"
"They okayed FIVE grand. For a three-year license."
"So... this is all you do all day is it?"
"Most days. Other days Carl or Peter does it."
"Carl or Peter?"
"Yeah, we work shifts - because the market never sleeps."
"So let me get this straight. You don't have any servers, you don't have any real work - AND THERE ARE THREE OF YOU - so you just make problems to keep yourself in a job?"
"Yep, That's pretty much it."
...
"I feel like I've wasted my life," I admit sadly, on the way back to the office after the "server" is back online.
"Me too," the PFY says.
Half an hour later I'm back in Mission Control when the Boss enters nervously.
"How did it go?" he asks.
"Well it looks like they're up and running, but to be honest their hardware has some real issues, so if you were going to be getting your money out, I'd be doing it today."
[7]BOFH : Have you tried forcing an unexpected reboot?
[8]BOFH : The USB stick always comes back – until it doesn't
[9]BOFH : Printer's festive bips herald a merry mystery for the Boss's budget
[10]BOFH : The Boss pulled the plug on our AI, so we pulled the pin on him
"Really?" the Boss gasps.
"Really," I say. "The system management is appalling and I don't think their IT guy has a clue what's going on."
The Boss almost knocks the PFY down on his way out the door, but he's back again inside a minute.
"There's something wrong with my computer!" he says, waving his laptop around madly. "I need you to look at it now."
"What seems to be the problem?" I ask, taking the laptop from him. "Let's see, all looks fine to me... oh..."
The PFY looks over, and shakes his head.
"Has someone forgotten to pay their enterprise English language keyboard licence?" he asks. ®
[11]BOFH: Previous episodes on The Register
[12]The Compleat BOFH Archives 95-99
Get our [13]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z_k8njzVZggAx8dtVS6b2gAAANQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z_k8njzVZggAx8dtVS6b2gAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z_k8njzVZggAx8dtVS6b2gAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z_k8njzVZggAx8dtVS6b2gAAANQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z_k8njzVZggAx8dtVS6b2gAAANQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/11/08/bofh_2024_episode_21/
[7] https://www.theregister.com/2025/03/21/bofh_2025_episode_6/
[8] https://www.theregister.com/2025/02/22/bofh_2025_episode_4/
[9] https://www.theregister.com/2024/12/20/bofh_2024_episode_24/
[10] https://www.theregister.com/2024/10/25/bofh_2024_episode_20/
[11] https://www.theregister.com/data_centre/bofh/
[12] http://www.bofharchive.com/
[13] https://whitepapers.theregister.com/
CI/CD in full effect.
I stand here in complete admiration. That was more devious than the contract negotiations I am currently sleeping through.
One for the weekend.
Beat me to it
Yeap, already giving me ideas on how to keep myself "occupied" when things are running too smoothly.
Everything works smoothly, they think we are not doing anything (when we do all the work in the background), need just enough hiccups in the system so that they see us as being needed.
Lessons of inspiration
There are always colleagues and peers who can inspire true improvement. Cheers!
As i read the passage
'... a maze of twisting corridors, all alike.' I couldn't help but wonder, and fear, if this was a suicide mission.
I'm so happy that it wasn't.
Re: As i read the passage
xyzzy
Re: As i read the passage
Nothing happens.
Re: As i read the passage
Twice as much happens :-)
Re: As i read the passage
A hollow voice says "plugh".
Re: As i read the passage
Thorin sits on a rock and starts singing about gold.
Re: As i read the passage
You are at Y2.
Never thought I would see the day when they were...
Out BOFH'd by Uber-BOFHs.
Re: Never thought I would see the day when they were...
I wouldn't say the were out done, more that they were schooled a bit in the finer points of BOFHing. We can all get complacent when we think we have it all dialed in.
Re: Never thought I would see the day when they were...
Yes, all apart from having to work in shifts.
I wonder if there will be a sequel where we get to meet Carl…?
Re: Never thought I would see the day when they were...
There is no Carl or Peter, he's getting 3 salaries - or he's an amateur!
Re: Never thought I would see the day when they were...
> . . . more that they were schooled a bit in the finer points of BOFHing.
I believe you meant to say "schooled a bit in the finer pints of BOFHing."
Cheers!
"English language keyboard license"
Hilarious. Sheer bleeding genius. Great start to the weekend
Re: "English language keyboard license"
Well, it's clearly expired or else it'd have been a keyboard licence
Re: "English language keyboard license"
Well I'm guessing that your kb would come with a Kernow licence, Korev.
Decent lager, I can buy it in my local Asda in E10, but it doesn't seem the same when you have not spent a day by the sea.
No defenestration but one of the best
I should have hung onto some old grey Sun boxes and done the same thing!
This is bittersweet. Loved the article, but also discovered that I can no longer watch Star Wars over telnet...
It's Been Done
I saw investment company with pretend IT equipment and thought Bernie Madoff.
https://www.zdnet.com/article/how-old-machinery-empowered-bernie-madoff/
Re: It's Been Done
Jean-Pierre Van Rossem ran a company called Moneytron (old F1 fans will know this name well!) and claimed to have a computer that could predict the stock market. It was obviously all bollox.
https://en.wikipedia.org/wiki/Jean-Pierre_Van_Rossem
Even and old BOfH can learn new tricks.