New IT boss decided to 'audit everything you guys are doing wrong'. Which went wrong
- Reference: 1674203409
- News link: https://www.theregister.co.uk/2023/01/20/on_call/
- Source link:
The source of this story is a reader named "Scanlon" who once worked for a company that hired a new operations director. Not long after taking the gig, the new guy declared his intention to "audit everything you guys are doing wrong."
Which rather rubbed Scanlon up the wrong way because, as he told On-Call, "I don't pander to someone who has more ambition than technical nous."
[1]
Ouch.
[2]
[3]
Scanlon couldn't stop the new ops director running his eye over the company's tech, which at the time was mostly midrange systems as it moved away from mainframes.
When Scanlon arrived at the company, he found several IBM P570 systems in place. Curiously, they were being used in bare-metal mode rather than running virtual machines.
[4]
"The result was expensive tin that wasn't performing anywhere near capacity and there were plans to buy even more kit," Scanlon scoffed.
Our hero was sorting that out as the new ops director arrived. He had one remaining issue to address: a tricky SAN driver upgrade to get a third-party vendor's storage kit playing nice with the IBM servers.
Because the upgrade was not complete, it was included in the audit.
I didn't receive a word of thanks and I was never compensated for my lost holidays
"I had an initial meeting with the new chap where he introduced me to an external consultant that he had engaged to audit our systems," Scanlon recalled. "I ran through the configs and pointed out that we were still dealing with a vendor to resolve the driver updates and that as a result there was a patch that we were holding off applying."
Scanlon reckoned other work could wait until the SAN was sorted, pointed out he was taking holiday the following week, and asked if the audit could wait until he returned. He left the room feeling his plan had been accepted, so headed off for his break.
[5]
On the first Tuesday of his holiday, a colleague tracked him down with news: the consultant had been given root passwords, commenced the audit in Scanlon's absence, and had run some scripts to commence his probe.
Those scripts took a while to run, which the consultant explained, naturally, on his way out the door for the day.
Not long afterwards staff noticed an eruption of alerts, followed by clients complaining they could not process transactions.
Which was why colleagues called Scanlon, who tried to triage the situation from afar.
Before long, the new ops director demanded he come to the office.
Happily, Scanlon was only five hours away by road and was decent enough to answer the call.
When he arrived, in the wee hours of Wednesday, he noticed all the servers had surrendered to a kernel panic.
A glance at log files quickly showed the script run by the auditor was the cause of the crash.
The script was buggy, but when the first server became unresponsive the consultant dismissed it as a mere pause while the script did its job.
So he ran it on more machines. Which also crashed.
Scanlon's assessment of the situation was that the script disconnected all the virtual servers from the network and storage, so the servers promptly hung.
[6]This can’t be a real bomb threat: You've called a modem, not a phone
[7]Cleaner ignored 'do not use tap' sign, destroyed phone systems ... and the entire building
[8]Don't lock the datacenter door, said the boss. The builders need access and what could possibly go wrong?
[9]Server broke because it was invisibly designed to break
Once he restored things to working condition, the SAN and network paths came back online, databases recovered, and business resumed.
Scanlon got back in his car and drove another five hours to resume relaxation with his family.
Is he still a little bitter about the incident? Maybe just a smidge.
"I didn't receive a word of thanks from the company, and I was never compensated for my lost holidays," he wrote.
"The one saving grace is that I remained with the company for several years after that, while the tenure of the ops director was significantly shorter."
"And my response to the company was quiet quitting for the next 12 years" – showing up to work without giving it his best effort.
Have new managers and their minions impacted your work, and seen you called out to manage their messes? [10]Click here to email On-Call with your story, which it would be our privilege to consider for a mention in this space on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y8p0UVzJDC0LPx@bCYA3GQAAAMw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8p0UVzJDC0LPx@bCYA3GQAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8p0UVzJDC0LPx@bCYA3GQAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8p0UVzJDC0LPx@bCYA3GQAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8p0UVzJDC0LPx@bCYA3GQAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/01/13/on_call/
[7] https://www.theregister.com/2023/01/06/on_call/
[8] https://www.theregister.com/2022/12/23/on_call/
[9] https://www.theregister.com/2022/12/16/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
"I didn't receive a word of thanks from the company, and I was never compensated for my lost holidays," he wrote.
Oh dear, but typical of most companies who see employees as glorified lightbulbs.
"And my response to the company was quiet quitting for the next 12 years" – showing up to work without giving it his best effort.
Good! Have a pint. Workers of the world unite etc.
Upvoted and agree.
I really hate that phrase “quiet quitting”. A person has a contract, hours worked money in return. Working those hours are all someone has or should do. What’s the phrase for when the boss only pays what your contract states? Why is it not an issue that they don’t add a few hundred extra at the end of the month??
Do you add a dollar to the price of your pint ?
I agree wholeheartedly with this.
I've said it on El Reg many times but I am contracted for 40 hours a week. You are paying me for 40 hours a week. You get, as a result, 40 hours a week.
There are rare instances (planned work that must be done out of hours or a P1 incident) where I waive this but only for time off in lieu (at the correct 1.5 or 2x rate).
The power of "no" is very freeing and more people should try it.
And indeed - when you point out the hours worked vs contracted, the responses vary from "ok" to "we can find someone else". Go on then.
I worked at one particular place and the manager was a sociopath. Amongst his many bad traits was the he would work most days from 07:00 to 23:00 and boast about it as though it was a good thing. He once accused me of not being committed because he expected all the team to put in those extra (unpaid) hours and was not overly impressed when I pointed out that one of us was an idiot and it wasn't me.
I suspect he's now overall head of HR for Elon Musk's companies
Professionalism
Absolutely agree with all of this - to me, if I'm doing work but I'm not being paid for it, my work has become a hobby. It is literally unprofessional to donate your work to your employer for free.
I have plenty of hobbies, I don't need my day job to be one of them.
Same here. It's pretty usual for a company to do the bare minimum to keep it's staff around. I've worked for several that do things like give out below-inflation raises (if any) for years in a row. They consider it good business. But suddenly it's a big issue when an employee doesn't show up to work and give it every ounce of their soul? No. Treat me like that, and I'll treat you the same in turn. I'll show up and do the bare minimum I need to do to get paid. It's not quiet quitting, it's treating the contract the same way the company is.
I would point out that I have also worked for some companies that did better than that. Generally, they got a much better result from me and the rest of their staff.
I've always liked "Acting your Wage"
It depends on what you want from your job. I was always happy to give a little more than required when it was necessary. It paid off handsomely over the years in terms of pay rises, bonuses, promotions etc. although I still made sure I kept a good balance between work and life.
Other colleagues whinged about being "passed over" or unappreciated, but they were usually the clockwatchers who never did one minute more than their contact required.
If people want to do the strict minimum, and are happy with no career prospects, good for them. Some of us want more from our lives.
Minima and maxima...
You've been lucky. Most companies are really happy to have people that go over and above, but less happy about rewarding them beyond a pat on the back during the next review.
pah
I've often found the arse lickers who get these, not the staff who do the real work
Quiet quitting for TWELVE YEARS! I bet that showed them. You could have put in some extra effort and either been promoted or moved company six times or more and had a more rewarding and interesting career.
What a Pratt..
The correct term for "quiet quitting" is "work to rule", unfortunately it's always been seen as a bad thing, even if it's doing exactly what your employer is paying you to do...
If you want to be a plodder that's a perfectly valid choice, but it seems like a very boring way to waste 1/3 of your life. You might as well at least do some interesting things and add value for 8 hours a day. Or sometimes if it's particularly interesting a bit longer.
This is it.
The term "quiet quitting" is portrayed as doing 35 hours in a 40 hour week. But that's not right. You're doing exactly what you're meant to do. It's just the enthusiasm, going the extra mile, not doing the unspoken things - that's what's meant by it. Which is fine.
What that company did wrong: rely on a single critical resource to have its IT working.
'Scanlon ' was very nice to answer that call. Seen how the things went later, he should not have been.
""And my response to the company was quiet quitting for the next 12 years" – showing up to work without giving it his best effort."
People like this is why the economy is doing shit and wages have flatlined. All people like this achieve is to make their fellow workers poorer.
So individuals are important in your view and individual performance is a determinant of economic outcomes?
Why then do HR folk and various directors insist in using language like 'resources' and 'resource action' that tends to treat their workers as standardised cogs?
"People like this is why the economy is doing shit and wages have flatlined."
No wage growth has slowed but that may have more to do with shareholder value engineering than the merits of individual employees.
The niavity of your comment implies that you haven't reached you 30th birthday yet.
"Quiet quitting" is the symptom, not the disease.
"Quiet quitting" is a horrible term by a management consultant, describing an employee doing what they are contracted to do. Oh, the horror.
I've done overtime for employers, and have gone over and above my job description with the reasonable expectation that it'd be paid for either directly (via overtime pay) or indirectly (via promotion, pay rise etc).
If it is, then that's all very well and good. Under those circumstances an employer can reasonably expect people to keep doing things over and above the contracted requirements of your job description because doing so is in the self interest of the employee.
If an employer decides to deliberately not reward employees for doing anything above their minimum contracted requirements via not paying overtime, not handing out promotions for hard work etc (in the name of equal pay for every worker) then i'd like an explanation from said employer as to precisely why they think that any rational person would do anything over and above the contracted requirements.
"...People like this is why the economy is doing shit and wages have flatlined. All people like this achieve is to make their fellow workers poorer...."
So let me get this right - to you a contract is meaningless? A person's work/life balance is meaningless? I imagine you are one of those people who think it's ok to expect 80 hour working weeks and paying for 40? You also come across as the kind of person who puts themself before anyone else and will happily exploit others to get ahead/make an extra dime.
Fuck. Off.
People are employed to do a job. They bring skills to that role for an agreed number of hours at an agreed salary. By your logic, I can say I will work 20 hours a week but expect to be paid for all 40. You'd soon scream blue murder.
So please explain why you think it's ok to abuse a contract one way by expecting free hours but not the other by expecting to only work half of them? Both, per the above, for the same pay.
We've all worked with people who get away with doing the very minimum possible without getting sacked. Plodders. Also people who work their socks off, and never have time for family or social life.
I look for something in the middle. Work hard and go the extra mile for the 8 hours I'm supposed to be working. Sometimes work later when it's needed for an emergency.
Sometimes take a long lunch to get that back.
It's called balance, whereas "quiet quitting" means doing the minimum.
I never said that people should do more hours then they are paid for, try to read what people actually say.
When you are paid for 40 hours of work you are paid to give it your full effort for 40 hours and not slack off, people who decided to half arse everything they do are the ones exploiting others and making things harder for their colleagues.
Obvious troll is obvious...
I'll talk to you again after your inevitable burnout after putting in way more hours than contracted and your employer getting rid of you for it. (Or if you're self employed, after your inevitable burnout that caused your life's savings to evaporate).
I do for my employer what I'm contracted to do. And on average not an hour more. Tell me, why should I do anything over what I'm contracted for?
People like this is why the economy is doing shit
No, managament like that is why economies flatline. If people were honest about the full cost picture of staffing they would see that management treating people that work for them as human beings comes with significant financial upsides because a member of staff is not just a number on a spreadsheet, they're also a collection of expertise and institutionalised knowledge that you somehow magically have to recreate when you lose them and that is especially the case for the kinds of people you can call up in the middle of the night and can fix things with merely thinking their way through the issue and pulling in the right resources to fix it there and then.
Being nice does, of course mean that you start with not employing entitled f*ckwits or mates but people who can genuinely lead a company but it is possible. [1]There are some examples out there.
There is another factor at play here: plain greed. If you do not squeeze the absolute last drop out of a company you also end up with some safety margin that will help when things get tough. Squeeze staff, and you will pay eventually.
[1] https://www.bbc.com/news/stories-51332811
Are you Elon ?
name check out
That's it, that's the post
I've related this one before but it's due another airing.
We were running a 486 SCO Unix box for our manufacturing/business systems, with about a dozen terminals. It had a DAT tape drive for backups which, with a sensible rotation & off siting policy, was about as good as things got then. I went away for a week but took the office "mobile" with me (a Motorola car phone attached to a big battery). On about the Wednesday the office called to say the tape wasn't ejecting. My response was, "OK, it's not critical, just leave it alone until I'm back next Monday!"
That wasn't good enough for the parent company though, they sent an engineer from their PC support from Hampshire to Northants to have a look, without bothering to consult me or at least say they were doing so. As the engineer had never worked with Unix he just switched the box off, replaced the tape drive and turned it back on, then he wondered why he was getting multiple error messages about corrupted file systems and a failure to boot.
The end result was exactly what I'd been trying to avoid, once I got back in on the Monday (it wasn't physically possible for me to get back sooner) and then rebuilt everything and restored back to the previous Monday (the last usable backup) the company had lost, or had to repeat, over a week's work. The parent company later tried to get me to go down south and work for them when they closed the Northants operation but I refused!
VPO story
Years later, a big WW DC was entirely monitored by HP VPO (or whatever names it was back then, since it was changing on a yearly basis).
A lot of alerts were pushed daily to it from every layers of OS or apps, and a 24X7 operators shift was in place.
So, VPO was really critical, like the radar of an airplane.
One day, manglement sent to us an HP expert to audit the thing.
He sat in the open space for multiple days, without mentioning to us, the ops teams what he was doing at all, and went back home (early) the friday end of the afternoon.
We had on call L2, and the guy had a completely hellish week-end, that week: VPO was crashing, the DB was hanging. He basically spent the whole week-end trying to get to the issue.
What happened was: the HP guy discovered multiple hundreds of thousands uncategorized alerts, stale alerts. What did he do just before leaving for the WE: yep, select the first, scroll down to the last, select all, then acknowledge the alerts. Without telling anyone, then fucked off.
VPO went on working full stream acking every one of them, creating MASSIVE numbers of DB transactions, causing the DB to switch logs almost each second, pilling all archived logs in the archive FS, filling it up again and again !
Peace was only restored when all alerts were all acked, with the L2 guy frantically emptying the archived logs FS as fast as possible (and no, we were not in the position to back them up so fastly, so had to delete them) for the whole WE.
Gosh, this HP guy was never allowed again in our buildings.
someone who has more ambition than technical nous
I always wondered how "nous" was spelled.