News: 1740385808

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Untrained techie botched a big hardware sale by breaking client's ERP

(2025/02/24)


Who, Me? Nobody starts the working week by planning to fail, but mistakes do happen and The Register likes to write about them in Who, Me? It's the reader-contributed column in which you tell us how you escaped from nasty scrapes of your own making.

This week, meet a reader we'll Regomize as "Kane" who told us about a job he said was "the most horrendous and grueling experience of my IT career."

"If I wasn't already taking blood pressure meds, I'm sure I would not have survived," he wrote.

[1]

Kane took on this terrifying job when he worked for a giant global hardware vendor and was asked to help an account executive sell a big batch of new servers to an existing customer.

[2]

[3]

The buyer wasn't keen, because the vendor had provided it with switches that were causing problems with virtual LANs.

Kane's sales colleagues put a question to the customer: "If we fix the switches, will you consider the new servers?"

[4]

The customer liked that idea, which is how Kane found himself at its offices one Saturday morning trying to fix a VLAN-related "partial outage" that had taken out much of its SAP implementation.

You'd think that when a giant global hardware vendor sends in a techie to fix that sort of thing, they'd be well-trained.

And Kane was very well trained – but on the new kit his employer hoped to sell and not on the glitching switch.

[5]

"I decided to show the customer how to make the changes with the web-based config pages," he told Who, Me? "I'd never been told not to use the GUI so I saw no problems with using it."

Unbeknown to Kane, the GUI was buggy.

[6]Techie pointed out meetings are pointless, and was punished for it

[7]Does this thing run on a 220 V power supply? Oh. That puff of smoke suggests not

[8]CompSci teacher sets lab task: Accidentally breaking the university

[9]Tired techie botched preventative maintenance he soon learned wasn't needed

"I ended up causing a Layer 2 broadcast storm that brought down most of the network," he confessed.

SAP was now unable to reach its external storage and looked, at first glance, to be utterly dead.

"I imagined that I had wrecked their ERP and there was no telling how long it would take to fix," Kane told Who, Me?

This was probably the point at which those blood pressure meds paid for themselves.

Fortunately, the infrastructure on which SAP ran was well-designed and properly resilient. It didn't take much work to get the network and the ERP working again.

But the client was not impressed, and a rival vendor scooped the sale.

Have you messed up a sales pitch? Pitch us your story by [10]clicking here to send email to Who, Me? We'd love to share your story on a future Monday. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z7xRVFpb01qdnHHrD3O8UwAAAcU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z7xRVFpb01qdnHHrD3O8UwAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z7xRVFpb01qdnHHrD3O8UwAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z7xRVFpb01qdnHHrD3O8UwAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z7xRVFpb01qdnHHrD3O8UwAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2025/02/17/who_me/

[7] https://www.theregister.com/2025/02/10/who_me/

[8] https://www.theregister.com/2025/02/03/who_me/

[9] https://www.theregister.com/2025/01/27/who_me/

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



IBM?

tip pc

sounds like those IBM/lenovo switches, bundled with IBM hardware as a supposed win win for the client.

i used to like those HP procurve switches and deployed hundreds back in the day.

cant beat a trusty cisco though. Not always best performance on paper but IOS consistency was the key to success.

its not uncommon to happen upon a cisco switch with uptime of a decade or more, of course no firmware updates in that time but that s a different story for a different day.

https://www.reddit.com/r/Cisco/comments/bgv7a3/uptime_record/

Re: IBM?

GlenP

HP procurve switches

I was a big fan of those until I had two fail to come up after a power off (due to fixed electrical testing on the premises which was too prolonged for the UPS to keep everything running). Of course this was late on a Saturday afternoon and, being a small office, there weren't spares readily available.

I had everyone up and running on the Monday morning but it did involve delving into my personal stash of come in handy gear.

Re: IBM?

Anonymous Coward

uptime is a record of how long it's been since your last successful boot.

Anit-Sales - or not?

Caver_Dave

I worked for a small computer dealer in the late 1980's.

My boss was high up in the local Chamber of Commerce and so we had a fairly captive audience (something critical to this story).

Consequently, most of the local businesses came to us when they wanted their first computers.

I used to go in and analyse their business needs and see where the use of computer would help. (This did not cover the PA typing the CEO's missives - that was always required!)

Around 8 out of 10 times, I would demonstrate to the company management that they needed a business process change rather than a computer at this point in time. But I would also state how this would enable them to use a computer in the future, what benefits would be gained and by which metric they should decide that they needed the computer.

My boss took the long term view and over the next few years we became the almost exclusive supplier to all the local businesses.

I considered myself to be the exact opposite of most salespersons.

Re: Anit-Sales - or not?

IanRS

I was once told, by a highly experienced and very capable salesman, that I should never go into sales. I was too honest about what the client really needed.

Re: Anit-Sales - or not?

GlenP

I once interviewed for a consultant role at an ERP provider that's no longer in business.

By the end of the interview I think we'd separately decided the job wasn't for me, my focus was on solving the customers problems (and I could have brought some talents to the company that they were clearly lacking in the UK), their focus was on "managing the project" or, in other words, selling the customer more consultancy time.

Re: Anit-Sales - or not?

lglethal

I can still remember the poster on the wall at a customers facility that we were visiting for an important Review.

Consulting - If you're not part of the solution, there's money to be made prolonging the problem.

Funnily enough, our Engineering team got on fabulously with them, the Sales team on the other hand...

Re: Anit-Sales - or not?

Brewster's Angle Grinder

I did that last month. Someone was offering to shower me with money to write a feature. I was prepared to do it, but I said, "It would be a waste of your money. There are dedicated tools [named a few] and this is the export feature you want to use them." They gave me a thank you "consultancy" payment.

Sam not the Viking

A major project for a utility was to be discussed with the End-User, Consultant, machinery-supplier (us), Electricity board, Control-panel supplier, Telemetry and the cabling-contractor. For some reason, the control panel supplier was leading the meeting and they had determined all the major items of equipment to be supplied. The intention was to use state-of-the-art starters to drive the machinery and that supplier had been identified and was joint-leader of the meeting……

After detailing their proposal, I innocently asked for some technical particulars regarding the starting procedure which I thought was unnecessary. A sort of waffle-response from the starter-people raised eyebrows. Seeking further clarification, the electricity supplier scratched his head and said that they would have to lay on a new HV cable through the town at a preliminary cost in the millions of pounds.

Flabbergasted, the End-User looked at his Consultant, who looked at the control-panel supplier who looked at the starter-supplier whose face revealed panic.

Consequently, the starting method was revised to the relief of the end-user. The control-panel had his order-value reduced to tatters. The starter-people went away with nothing.

A good outcome for us.

"Unbeknown to Kane, the GUI was buggy."

Bebu sa Ware

Young Kane must have led a sheltered existence if he was sufficiently naive to believe any GUI was free of serious bugs.

I run at the sight of a web interface and faster still if there is a whiff of java. Most platforms have a reasonable to decent command line interface (if you can get to it.)

A broadcast storm is pretty decent achievement. Curious how he managed that. Turned off spanning tree?

Did you really pull the big red switch?

ColinPa

Someone in our department was testing some software, and being a thorough tester turned off the power to the machines halfway through a test.

When the systems finally came back the data was in a mess. Some transactions had been run twice. Some transactions had not run at all, and some transactions were stuck in the middle.

The call to the development team went something like...

Her:"I turned the power off at the wall and when it restarted..."

Them: "You mean you shut the system down?"

Her:"No - I turned the power off at the wall"

Them:"But you never do that as it makes mess of the system - surely you just shut the system down in quick mode?"

Her:"No I turned the power off at the wall - I was simulating a power cut"

Them:" Wow - we've never seen anyone do that before..."

She worked for a very effective test department who were asked to write an article on "testing". It started something like

"If we were to make cars, the first thing we would do it drive it at top speed in first gear down a rough road with plenty of pot holes..."

Re: Did you really pull the big red switch?

Anonymous Coward

One of our users in a remote location decided to proactively solve a (completely unrelated) problem by rebooting the router. Except that they couldn't be bothered to reach into the back of the cupboard where it was piled, so they just unplugged the extension. The extension which all the computers were also plugged into.

And that's how we found out that modern NVMe drives have a write cache, that can be lost if they lose power in the fraction of a second between a file being written, and being flushed to the SSD. (We eventually decided not to do anything about this, and the window for data being lost was so small).

"If once a man indulges himself in murder, very soon he comes to think
little of robbing; and from robbing he next comes to drinking and
Sabbath-breaking, and from that to incivility and procrastination."
-- Thomas De Quincey (1785 - 1859)