News: 1666942032

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

No, I will not pay the bill. Why? Because we pay you to fix things, not break them

(2022/10/28)


On Call As the holiday season nears, prepare for the smaller celebration that is the end of the week with another edition of On-Call, The Register 's weekly walk through the worst of the world of work.

This week, meet "Brad", who sent in a story he says has haunted him for over 25 years. Such is the anguish he suffered after the email server he oversaw started to slow down, and a warning light suggested one of the drives in a RAID set was the reason for the freezin'.

Brad's workplace had engaged the then newly combined Compaq/Digital to provide hardware support. When the warning light was followed by a dead disk, he invoked the support contract and waited for a technician to appear.

[1]

Said worthy arrived and Brad was appointed his chaperone, so showed him to the stricken server in the expectation that replacing the deceased drive would see the RAID array rebalance itself.

[2]

[3]

The Compaq chap asked Brad to pull out the disk – an act Brad obligingly performed.

Note, dear reader, that this story is more than 25 years old. And in those far-off days, when plug and play was young, just yanking out a disk was not recommended procedure.

[4]

Brad did exactly what he had been asked to do. That he should have done something else first was something that should perhaps have been made more explicit.

Things went well … for about 20 seconds.

"The server then promptly fell over, as in shut itself down, taking all our email services with it," Brad recalled.

[5]

Brad asked what happened next, at which point the visiting tech absolved himself of all blame.

"It was you that pulled the drive out!" cried the person who had ordered the drive to be pulled out, who then failed to provide a fix.

Poor old Brad ended up pulling an all-nighter to restore email service and his colleagues' inboxes.

[6]To make this computer work, users had to press a button. Why didn't it work? Guess

[7]Senior engineer reported to management for failing to fix a stapler

[8]No, working in IT does not mean you can fix anything with a soldering iron

[9]Fixing an upside-down USB plug: A case of supporting the insupportable

Fast forward to a week into the future, when someone from Compaq/Digital called with a job reference number and a question: "To whom should we send the invoice for the work carried out on your email server?"

Brad replied with the strongly expressed sentiment that he should be billing Compaq, not the other way around. He then observed in clear and succinct terms that when one pays for tech support one expects support personnel to know, remember, and implement disk dismount procedures.

"We never heard about that invoice again and never saw that Compaq engineer again," Brad recalled. "And the room got really quiet when this happened. I think I raised by voice a bit."

Has a tech support incident ever made you bellow with rage? Or have you ever dodged a bill for terrible service? If so, revisit the moment with a bit of digital primal scream therapy by [10]clicking here to send an e-mail to On-Call and your take might appear here on a future Friday.

No tale is too small as we plot the return of On-Call snippet roundups for the silly season. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y1uoOYt9ElXAvcFX1v9CvAAAAQI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y1uoOYt9ElXAvcFX1v9CvAAAAQI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y1uoOYt9ElXAvcFX1v9CvAAAAQI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y1uoOYt9ElXAvcFX1v9CvAAAAQI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y1uoOYt9ElXAvcFX1v9CvAAAAQI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2022/10/21/on_call/

[7] https://www.theregister.com/2022/10/14/on_call/

[8] https://www.theregister.com/2022/10/07/on_call/

[9] https://www.theregister.com/2022/09/30/on_call/

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



Korev

As the holiday season nears

You mean Christmas?

Anonymous Coward

Only if you are a limey. Otherwise it's Halloween, Thankgiving etc etc

jake

Collectively known in these here parts as "the horrordays".

Me, I'm counting the days until solstice, when I start the first of the chilis.

chivo243

I've already made two batches this year... Why wait?

Icon, for you know, chili!

Lord Elpuss

Halloween's in muddyfunstering OCTOBER. So no. It's not Halloween. In the US they have Hannukah which kinda-sorta-maybe gives an excuse for calling it the Holidays, but in UK and Europe it should be Christmas. No ifs or buts.

phuzz

There's apparently some religious stuff going on in the winter, but then there seems to be that sort of thing going on all the time so it's not worth paying attention to.

However, the important part is that I get a couple of days off around the 25th of December, and another one off for the 1st of Jan.

That's two different sets of time off, hence, 'holidays' plural.

Phil O'Sophical

DEC the halls?

Admiral Grace Hopper

We wish you a Sperry Christmas and a Burroughs New Year.

Trollslayer

BTW, I have a biography of Grace Hopper, amazing person!

David Robinson 1

Why did the programmer put his Christmas decorations up for Halloween?

Because OCT 31 = DEC 25

(Mind, I know of people who already have put up Christmas decorations.)

It goes far beyond that ...

jake

Yes, any techie will confirm that OCT 31 and DEC 25 are the same.

But have you ever noticed that Santa and Satan are anagrams? Have you ever seen Saint Nick and Old Nick in the same room together? Who would YOU pick as the patron saint for a holiday best known for hedonism, libertinism, decadence and debauchery?

Re: It goes far beyond that ...

breakfast

So you're suggesting that Santa is some kind of Saturn-alias?

Re: It goes far beyond that ...

chivo243

good evil, god devil? Wait, what were we talking about?

localzuk

Bizarre thing to say really.

Last I checked, Hanukkah, Kwanzaa, and Ōmisoka all happen in December.

Plus, Thanksgiving, Halloween, the Winter Equinox, etc...

Or do you think only Christians read this site?

jake

I'll bet you a nickle that if we ever meet up with a real extra terrestrial civilization which evolved on a planet with a tipped axis, that culture will have similar celebrations, and for the exact same reasons.

Korev

"We never heard about that invoice again and never saw that Compaq engineer again,"

Did they make him Compaq his bags?

IBM Engineer

GlenP

I had similar from an IBM engineer on an AS/400. We were getting disk warnings on one of the drives which he attributed to a faulty cable. No real problem, I left it to the evening shift operator (the former DP Manager who'd semi-retired) to deal with the engineer after office hours. The AS/400 setup I'd inherited, and never had the budget to improve, was only RAID0.

I arrive the next morning to find the AS/400 is down and much wailing and gnashing of teeth.

Both the engineer and my operator should have known the correct procedure was to remove the "faulty" disk from the array first (we had sufficient spare space) then change the cable and add the disk back in. No, to save time they'd decided to just change the cable which turned out to not be the problem of course, and the drive failed to spin back up.

I then found the system wouldn't restore all the config settings off tape* (the exact details and reasons are lost in the mists of time). I managed to get the ERP at least running in a couple of hours but it took the best part of a week to get everything back fully working with all permissions, etc. in place.

I'd have liked to sack the operator and remove IBM from the support but neither was an option, unfortunately, so I had to live with it.

*1/2" Mag Tape on reels, none of those new-fangled cartridges! :)

Re: IBM Engineer

TeeCee

IBM midrange systems were inherently RAID 0 by default since time immemorial. Of course they never referred to it as "RAID 0", as they were doing it waaaay before the RAID concept was ever dreamed of. Besides, the bloody disks were anything but "Inexpensive". Still, good old IBM were trying to wring every drop of performance out of the hardware and forcing parallel disk channel use by deliberately fragmenting the data across all the available disks was a big win.

At least by the time the '400 came out, the backup systems had evolved to the state where you could feasibly backup the access paths (indices) as well as the data. Back in the days when tape bpi was measured in three digits, on the '38, even if you could restore everything, you were then looking at a week for the thing to rebuild its access paths before it would let anyone login.

Thus finding one day that some eejit had caught a sleeve on the handle of the isolator wallbox for the disk array, causing an immediate and ungraceful shutdown of all the disks and controllers, looked like being a bit of an issue. Good old System/38. I turned the disks back on and it picked up and carried on like nothing had happened. I've never seen another machine before or since that would do that.

Chief Operator: "You mean you can turn the disks off for a few minutes and then back on and it just carries on where it left off?"

Me: "Apparently. Let's not try it again though."

Anonymous Coward

Been there more than once.

A Sun engineer that didn't know how to shut down a system springs to mind - he got escorted offsite *very* quickly.

Having said that, back in their respective hey-day, I knew both ICL and HP engineers that were amazing, and really knew their shit. I learnt a lot from them. And there were more good than bad engineers back then.

Oh that brings back memories

TonyJ

Back in the day - mid 90's - when I was a hardware engineer myself the company I worked for was a bit fast and loose with the engineers it sent on site for various things*.

Basically if a field engineer was available and fairly close, they got the job.

Because as part of my role I repaired the PC's and and servers that came back to us, said field engineers would often call me for help.

I think I may have mentioned this one before, actually, but one of them called me to ask for help. Having hot swapped the drive in a Compaq Proliant (remember the big ol' towers that weighed as much as a small car and were the height of a desk? That sort). This was in the fairly early days of hot-swap disks to the point that the port colouring was still a new innovation to tell you a device was hot-swappable.

Anyway he had successfully swapped out the disk but then followed what he knew for desktops - booted to a floppy disk and typed format c:

Oops.

Mind you this was also the same guy who phoned me whispering to ask if opening the door on another type (I forget which model with age) would turn the server off. No hot swap on this one but a door interlock that yes, turned it off. He'd walked in an opened the door.

The thing is, he wasn't a bad engineer just not trained on the kit.

**Some things don't change, eh?

"not trained on the kit"

Mishak

I'm sure I've mentioned these event before, but I've seen some people treated very badly for doing what they had been told to do:

1) Night shift was tasked with manually populating diodes and tantalum capacitors on PCBs after the pick-and-place had done the ones it could handle. Every board failed when they got to the test cell because (on average) half of the components where in the wrong way round. Manager tried to sack the workers - until I pointed out they were unskilled and had not been told polarity was important.

2) Job experience guy was tasked with programming EPROMS - "Put it in here, press this button, take it out and put it in the pass or fail bin, as indicated by this light". He came in the second day and continued as before. Units hit the test cell on the third day, and all failed to start. Once again, management's response was to sack the guy - someone forgot to mention that the master needed to be loaded into the programmer first thing in the morning.

Yes, with a tape system

AlanSh

Remember the days when you could walk inside a tape backup system? Well, a customer of mine had one and was very proud of it. So, proud that he was showing off to his superiors and said "let's go inside". Managed to open to door - at which point, everything inside shut down and all the tapes dismounted. Well, it would, wouldn't it. Humans and tape mount systems don't work well together.

It took us 2 days to remount all the correct tapes and restart the backup processes. During that time, the customer was given an opportunity to work elsewhere...

Firmware

Anonymous Coward

We had something similar when a supplier was asked to do a firmware update on a modern disk array - they even ran it up to the manufacturer, to check it was good to go, as they had remote monitoring.

Cue chaos as wrong firmware was applied.

(Can't remember what we got out of it, but was worth it iirc)

bills and ducks

Anonymous Coward

After a similar incident at my employer, we invited IBM to a meeting to discuss field service's mistakes and the invoice they'd sent. The boss said "What does IBM have in common with a duck? They can both stick their bills up their arse."

Paul Cooper

Two things, both with the same employer in the late 70s!

1) Company moved to a new headquarters, and telephone access to our computer (a bureau service) was essential (this was pre-internet days, and British Telecom (it might still have been the General Post Office!) were the only supplier. We had arranged with them for the telephone lines to be connected on the day of the move, well in advance. Come the day - no engineer. Rang them; "Oh, the engineers are on strike!" One and only time I have come close to losing my cool; they were told in VERY strong terms that it was their problem, not ours, and we had their name on a supply contract! I must have hit a nerve - an engineer (probably one who had been promoted to management) turned up and we got our connection. Or it might have been a case of the squeaky wheel getting the grease...

2) The hardware included a disc drive - probably a few megabytes in a thing the size of a spin dryer. It needed fairly regular maintenance (I had the usual collection of crashed disc platters on display). Funny thing was that every single time the engineer came, there were screws left behind - they knew they'd be back, and why put screws back in that they knew they'd be taking out again?

jake

"Funny thing was that every single time the engineer came, there were screws left behind - they knew they'd be back, and why put screws back in that they knew they'd be taking out again?"

This was very common in the days before functional electric screwdrivers.

I bought a Makita "screwshooter" cordless drill back in '86ish so I could refasten the hull in my Owens (double-planked 5/8" Port Orford cedar ... they don't make 'em like that anymore). I quickly discovered how useful it was in my day-job, fixing computers and ancillary bits. I don't know how many 9.6V NiCad battery packs I've worn out with that tool, but it still works. They don't make 'em like that anymore, either.

Anonymous Coward

HP Printers - they use too many screws, always have one left over, must be spares!

UCAP

You buy an HP printer, you are screwed.

I'll get my coat ...

Potemkine!

Has a tech support incident ever made you bellow with rage?

From time to time. As the support is more and more outsourced, it happens more often.

Seen this week: a vpn tunnel managed by an external outsourcing company went down. It took several hours to come back. When I asked for a root cause analysis and actions to avoid this to happen again, the only answer I had (after several reminders) was "there was an error".

I don't know if the answer I made showed accurately my state of mind on this.

== Bring us Dabbsy back! ==

Binraider

Heh. A supplier, who shall go un-named had the temerity to ask why we didn't renew a support contract with them a few years ago; for a particular black box associated with instrumentation.

"Because, when we raise a support ticket, it is closed by [the supplier] because the black-box in question is out of support - please buy a new box".

Suffice to say they aren't a supplier any more. New black boxen that do the same job are available for considerably less through other suppliers, and have better standards of support that will attach onto the same instrumentation.

Martin Howe

Years ago in the late 90s my broadband internet kept failing. BT came out, engineer said he couldn't find a problem but if it persisted, maybe they would 'reset the line card at the exchange'. He did not find any fault in my computer or modem (this bit is important).

BT later sent me a bill claiming it was my fault as they couldn't find a fault of their own; note this was a typical corporate fudge; they did not prove my equipment was at fault, only that they couldn't find a fault with theirs.

Meanwhile the fault had persisted and somewhat later, I phoned them again and suggested they do what the engineer had said; they did it ... lo and behold, my broadband was back!

So I wrote to them and stated what had happened and demanded my money back (which I got), pointing out to them that if they'd done this by default, the whole issue would have gone away.

The only certainty is that nothing is certain.
-- Pliny the Elder