News: 1610093788

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Pizza and beer night out the window, hours trying to sort issue, then a fresh pair of eyes says 'See, the problem is...'

(2021/01/08)


On Call The second [1]On Call of 2021 is a warning not to mix pizza and hashes.

Today's story comes from "Jon", who describes himself as "a manager of a perpetually understaffed IT team." As many will be all too familiar, this put him directly in the escalation path for pretty much any issue that could be daubed with the "it's IT's fault" brush.

[2]

On the night in question, Jon was preparing for an evening of pizza and beer with his spouse and friends. Of course, it being a precious Saturday, his work phone rang (much, we imagine, to the unalloyed joy of those around him).

"On the other end," he explained, "was one of my team members, who had become stumped by an issue with our VMware environment and was calling me for advice."

He logged in, looked around the environment, and found that authentication into vCenter "was completely b0rked."

I built a shed once. How hard can a data centre be? [3]READ MORE

vCenter is a key part of a VMware setup. Administrators use the utility to manage virtual machines and ESXi hosts. It being broken would represent a bad day for all concerned.

Noting that his team member was in a later time zone, Jon did the decent thing and let him off the hook, taking responsibility for the issue himself. After all, how hard could it be?

As it turned out, very hard. The hours ticked by and Jon was abandoned as spouse and pals set off to enjoy the pizza and beer he'd been looking forward to. He bit the bullet and opened up a Priority One case with the vendor.

"As tradition demands," he sighed, "the support agent instructed me to retry all the steps I had already tried, to no avail…"

Jon and his new friend began to pick through the configuration with a fine-toothed comb in search of what had upset vCenter so, but turned up nothing.

It took several escalations within the vendor's support organisation before Jon found himself speaking to a new technical support person.

A fresh pair of eyes spotted the problem all too quickly and a configuration file was flashed up.

"What's the problem?" asked Jon.

"See all these hashtags?" the tech patiently explained.

"Hashtags? What the...?"

The penny dropped. "Someone," Jon explained, "had commented out (with the # sign, natch) significant chunks of the file, causing authentication to break when the Virtual Center service restarted as part of routine weekend maintenance."

It was the work of minutes to revert the change, and Jon's organisation was up and running.

Later, he put the word out to his team and a shamefaced colleague stepped forward to confess to the deed.

"He and I had a nice little review of our procedures in regard to modifying critical infrastructure systems."

Doubtless with the assistance of a baseball bat peppered with rusty nails.

And what of his evening? Jon has short shrift for those who gloat over accrued overtime: "Managers don't get overtime, we get one cold slice of leftover pizza."

We bet the shiny job title and embossed business cards make up for it, though.

[4]

Ever struggled to solve a problem, only to find a smug pair of fresh eyes achieved in a second what you failed to do for hours? Or swerved some pineapple-topped pizza by pretending the On Call phone was ringing? We've done at least one of those things. Have you? Share your tale with an email to [5]On Call . ®

Get our [6]Tech Resources



[1] https://www.theregister.com/Tag/on-call

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_datacentre/virt&sz=300x250%7C300x252%7C300x600&tile=3&c=33X-g7RZrlo3FFki61MKgKSgAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dtop%26test%3D0

[3] https://www.theregister.com/2021/01/01/on_call/

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_datacentre/virt&sz=300x100%7C300x250%7C300x251&tile=4&c=44X-g7RZrlo3FFki61MKgKSgAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] mailto:oncall@theregister.com

[6] https://whitepapers.theregister.com/

Proof reader

don't you hate it when you lose your account

Sometimes we need a second set of eyes to see what's there not what we assume is there. Ect instead of etc is my eternal stupid. My dyslicsic brain has done that one so many times I do a search these days just to be sure.

Re: Proof reader

GrumpenKraut

'form" <--> 'from" is my, erm, favorite. Spellchecker does not help, of course.

Re: Proof reader

Hubert Cumberdale

That's why I've written my own Word toolbar with routines to highlight all these potential problems.

Re: Proof reader

Caver_Dave

I use a Chinese guy for reviewing my certification documents. Having grown up with a different character set, he seems to find mistakes that those more familiar with the character set miss.

It apparently is all to do with the skill we learn as children, to read the whole word as one rather than individual letters, which makes us so bad at spotting transposition or even missing letters.

Re: Prelalal dnceodig

Flocke Kroes

If the fsrit and lsat lrtetes are cercrot the orehts can be in any oredr and you can siltl wrok out waht is mnaet qitue qilckuy.

Doubtless with the assistance of a baseball bat peppered with rusty nails.

chivo243

I thought this was on call? Not the BOFH! Although I have to believe that Simon would be using a cricket bat? No?

Great start to the weekend!

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

A K Stiles

Ah - the trusty clue-stick! In a previous existence we had a real clue stick mounted on a rack on the wall of the IT office next to a sign labelled "In case of emergency". It was taken down and handed to new starters in the office so they could appreciate its impressive heft. That was the only time it was taken down, and it never left the 4 walls of the office, I suspect just in case the temptation to actually use it was a little too strong...

(icon sort of clue stick shaped)

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

Michael

Ah, I remember visiting a former HQ building to work with the QA department to train them on some new software. On the wall was a frame with the message for repeat offenders and a small pistol. A new team leader happened to start working in the QA department that week. She picked the pistol off the wall and pointed it towards the room. Everyone but me hit the floor and the QA manager moved at high speed from his desk and pointed her hand upwards.

Apparently nobody could remember if it was still loaded with live rounds. They had been shooting with it recently. After that I decided to avoid visiting QA. Or putting my name on code commits....

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

Lord Elpuss

I'm not sure who should be fired/prosecuted/shot first - the person who decided to put a (potentially live) weapon in a workplace, or the person who decided to wave it around. Always treat firearms as loaded, never aim at anything you're not intending to shoot etc. Unless the TL actually was intending to shoot somebody; you never know.

Either way not a place I'd like to be working in.

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

David Robinson 1

I prefer an IBM Model M keyboard. You can run them through a dishwasher to remove blood, allegedly.

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

Anonymous Coward

Peppered? You don't pepper someone with one, you a-salt them.

I'll get my coat, it's the one with the pockets full of condom mints. =-)p

Re: Doubtless with the assistance of a baseball bat peppered with rusty nails.

W4YBO

Bad puns bring back memories of watching "Hee Haw" with my grandparents.

Pi r square. No, pi r round. Cornbread r square.

Nope, never, not me...

A K Stiles

Definitely never been responsible for amending a config file then forgetting to reload the config before a weekend / week off causing a system to crumple into a heap and refuse to start when, a few days later, it was subjected to the patching updates schedule. Definitely not me...

(more than once per job role)

Re: Nope, never, not me...

GrumpenKraut

Improvement on that one: Customer installs Linux on his machine in rack and sets up web server and so on. Several month later a power failure. Machine would not come back. Turns out everything was set up in the install system running in RAM and on RAM-disk, absolutely nothing on the disk proper.

Took a good while to figure out what happened.

Re: Nope, never, not me...

John Riddoch

We had a process where before any major changes/patching we should reboot the server just in case something had been left for the unwary. If the app didn't start after a simple reboot, we knew there was an issue NOT caused by our change. Without that step, any issues found after the change would be attributed to our change and Our Fault with all the attendant Blame being flung our way, especially if we had to back out and it still wasn't working.

Re: Nope, never, not me...

Jay 2

Once upon a time when I could actually escape the office to go on a course I was doing a RHEL refresher/certification and it was pretty much drummed into us that there are many ways to change things on Linux boxes, but don't forget to make the changes permanent so they they'd survive a reboot.

We've got some kit running an in-house mutation of Linux which rebuilds itself upon boot. So for example we can add routes on the fly, but the best way to make sure is to change the config, reboot (and therefore rebuild) and check.

I can relate to that

Anonymous Coward

Ah, the heady days of my youth on the electronics test benches of a maker of control and instrumentation products.

As part of the banter between benches, it was a common thing to walk past someone struggling to locate a particularly recalcitrant fault, pick up a PCB, eye it for a second or so, put it down and declare "oh, I see it" and walk it off.

The thing is, so often that was actually true (not always. That would have taken the fun out if it). To the proverbial fresh pair of MK1 eyeballs, the cracked track, whisker of solder or a bit of spurious copper left by a hair on the tracking photo mask, were all too obvious.

That was in the days before surface mount micro components, when all you needed was a decent soldering iron, a roll of tinned copper wire, desoldering braid, a solder sucker and asbestos finger tips.

Happy days

>Waves at anyone remembering Protech Instruments in general and Emu in particular<

Re: I can relate to that

Flocke Kroes

A beginner did an excellent job of etching, drilling and soldering a one-off two sided PCB but did not understand why it did nothing until I mentioned that the holes do not conduct electricity by themselves.

I wasted half a day debugging a 2.5V circuit until I noticed the 5V version of a chip had been installed. Some helpful person had fixed an "out of stock" problem by calling the supplier. Apparently many customers did not mind which chip was supplied because the 2.5V chip was designed to work in 5V circuits.

I busted a few prototype PCB's before I noticed the big inductor for the switched mode power supply was very obviously not big at all. Someone had helpfully dealt with a late part by authorising a different one with the same inductance.

~/.procmailrc

GrumpenKraut

Something like

:O:

* ^Received: .*\.sendgrid\.net

$SPAM

did not work. Fresh pair of eyes said the line :O: better be :0:

Since then I make sure to only use fonts that have a dot within the character zero.

Re: ~/.procmailrc

chuBb.

Reduced user login support calls by 90% and started a still ongoing turf war with marketing, by ignoring the style guide and editing with extreme prejudice the style sheet that choose a wanky modern font which displayed i L 1 and O 0 the same on all form fields and password emails for a clear monospaced one....

Re: ~/.procmailrc

A.P. Veening

Easily won by making sure such forms and emails have the marketing approved style and font when viewed by resp. sent to the marketing droids. Their problem when they mistype the new password.

Re: ~/.procmailrc

GrumpenKraut

To marketing person: "Your new password is 1lıl¡1Ii!!ıl|¡|I".

Re: ~/.procmailrc

Aladdin Sane

Great, now I have to choose a new password.

Re: ~/.procmailrc

Doctor Syntax

Where I started out the standard for submitting work to the card punch operators was to cross the letter O rather than the numeral 0. Maybe it encouraged students to be particularly careful when they moved into the wider world.

Re: ~/.procmailrc

Christoph

Done that with the very first program I ever wrote. The program itself was fine. But I was punching my own cards from the coding form rather than sending it in to the punch girls (yes, 'girls' - this was the 60s). I managed to try to send output to LPO instead of LP0.

GlenP

Around 35 years ago as an analyst/programmer I learnt the importance of "fresh eyes".

Working with DEC RDBMS (I think that's correct) I couldn't get one particular routine to run. After extensive debugging I referred it to DEC support who came back almost immediately, I'd used a reserved word as a variable name. The error messages could have been more helpful but I should have spotted it.

Same role my boss was struggling to debug a Cobol program and asked for help. Despite the fact my Cobol knowledge was minimal (at the time my experience was Basic, Pascal and Fortran) I spotted her mistake immediately. Brownie points for me, shame for her but easily done.

Tom 7

One of the first think I used to do was to create a file of all the reserved word in a language so that something could be done with them - editor highlighting or a shell script to grep them on save. That was before it took longer to learn how to program the editor than it did several languages. Thank god error reporting has improved faster than documentation.

chuBb.

There is a lot to be said for only using text editors configured to display comments in a different colour especially multiline day ruiners committed by whitespace phobic devs...

Tom 7

Using a version control system and Diff for anything going near production is handy too. When working from home your cat can get paid for those mods he's done while you make a cuppa.

At least he'd used the right comment character

Phil O'Sophical

It's amazing how much chaos you can cause by commenting out a line with a # when the daemon in question uses a semicolon for comments...

a shamefaced colleague stepped forward to confess to the deed.

MrMerrymaker

What a dunderhead this fellow was / may well still be!

I've never done this, nope sir, nope nope nope (other sounds of protesting too much)

Re: a shamefaced colleague stepped forward to confess to the deed.

Tom 7

Its always worth noting that even in the days of floppies etc there is almost always a way that intelligent forensics will reduce the possible perpetrators to a manageable level. I've even managed to convict myself of a Friday afternoon cock-up after I left someone I detested's leaving do early and returned to the office to wait for a lift home. I have no memory of the incident but I figured it can only have been me but no-one noticed I'd gone and it was a couple of weeks later when the system crashed and the date stamp on the file probably didnt lie. No-one else worked it out though.

The value of not working all hours

MJI

I have seen the effects of huge hours on productivity over sensible hours.

Little difference in the end.

On those 12 hours days you pace yourself (go slower), you make more errors, you brain struggle near end.

Normal day plus two hours on a Saturday remote gets a lot more done.

Re: The value of not working all hours

Tom 7

In my first proper job we worked on things like Vax780s and things of less power.That, in my job, left a lot of waiting time for jobs to finish and I used it to learn new skills. Because of the nature of the job you could do create some moderately accurate productivity data. Over about 6 years I worked our a continuous 37 1/2 hr week was the most productive. I could do a couple of weeks of 60hr weeks to get a job finished on time and then it would take a month of normal to get back up to normal. A month of 60hrs would lead to lower than normal productivity, largely due to recovering from errors but also from lack of enthusiasm.

The real answer to loss of productivity in longer hours I discovered working in the US where 12 hr days 6 days a week seemed to be the norm: just dont do anything that might tire you out like work.

AndyMTB

First thing I used to do when faced with a config issue was scan the file system for files modified in the last few days. Easy-peasy in unix, probably not feasible with Windaz registry, thankfully never was called to do anything serious on a click-click-drag o/s.

Eye of an eagle

Kevin Johnston

While working on flight simulator visual systems I was supporting a senior engineer with testing some upgrades and one area was not working as expected. We traced the fault to a single card but everything about it looked good according to all the circuit diagrams etc. As we walked back to the test area to have another go we passed another engineer who shall be known as 'The Guru' as from us passing 6 feet away at walking pace he looked at the card we were carrying and said "that's the wrong chip type". Apparently one of the 40 or so discrete ICs on the board was a 74Fxxx and should have been a 74HCxxx. We changed it and all worked perfectly

CVS

David Robinson 1

Policy is that all text config files are held in CVS or similar. You check out the latest version and make changes to that. Should anything untoward happen on deployment of changes, it makes it easier to roll back.

Christoph

I had a call that a web sales site had started giving weird errors. Got the client to send me their copy of the files, and found that:

1> They'd added a call to an exterior function that was somehow trashing the login cookie.

2> Someone had hashed out the line that checked that the cookie was still present (and kicked you back to the login screen if it wasn't) immediately before the call to the very complex SQL stored procedure that did all the charging.

At a guess they'd put in the exterior call, got fed up with going to the login screen, took out the line that did that, ...

Go directly to jail. Do not pass Go, do not collect $200.