Solving a big, yellow IT problem: If it's not wearing hi-vis, I don't trust it
- Reference: 1605255313
- News link: https://www.theregister.co.uk/2020/11/13/on_call/
- Source link:
Our tale today comes from "Ivor" and concerns the time a whizzy new minicomputer was being installed in the dealerships of a company known for heavy duty machinery that was big, yellow and driven by hard-hatted individuals garbed in high-vis jackets.
Unlike the earth movers, the computer itself was of the Big and Blue variety, and was an upgrade from the weekly centralised batch processing of old. Used to big iron of a different nature, those in charge regarded the new-fangled hardware with suspicion, but went along with the plan regardless.
"We were finishing up installing the software developed by corporate," recalled Ivor, "so data entry personnel could start updating all of the databases required to run the organization."
Crunches are not a new invention, and Ivor recalled the huge pressure to get things up and running. Data entry personnel worked the weekend in order that everything could be tested and validated ahead of a Wednesday start.
Ivor was placed on call at the site to support the activities for five long days of double shifts but, finally, it was done. "Inventory was up to date, customer accounts were complete and balanced. It was a go."
At last the big day, Wednesday, arrived. Processing began. Hardware purred along as planned… and then everything died.
"Everyone panicked," recalled Ivor. "Some department heads were calling for ours: 'Everyone knew computers are a bad idea. Too finicky…'"
And then it all came back up. Purring resumed. Processing carried on. The only indication of trouble was a log event noting a loss of power, but other than the blip of Big Blue's machine, everything seemed fine. The terminals, printers and other office equipment showed no sign of power problems.
Perhaps wishing to pre-empt the inevitable call from this most important of clients, Ivor and a colleague were stationed on-site until the dealership felt comfortable that the computer could be trusted.
The days ticked past, and we fervently hope the duo got the chance to play with some of the big, yellow digging machines while the computer steadfastly refused to go wrong again. Until Wednesday rolled around once more and the system abruptly shut down.
"This time," said Ivor, "the other member of the team was standing in the doorway to the huge maintenance shop lined with huge machines in for repair. It was on the other side of the wall of the computer room."
The shutdown had coincided with a tremendous noise from the shop. Grabbing the shop manager, the duo asked what the source had been.
"It turned out that Wednesday second shift was when they repaired damaged frames. They had a massive welding machine that was only turned on when it was needed because it drew so much power."
Big Blue's mini had been faced with an unsurmountable power slurping foe and, in the absence of a UPS, had collapsed into a heap.
While popping in a new transformer and changing the computer to the new circuits dealt with Welder Wednesdays, Ivor's headache did not end there and then.
"It still took months before some managers quit bad-mouthing the system," he sighed, "expecting it to fail at any moment."
Ever found yourself faced with smug technophobia only to found that your woes coincided with the dimming of the lights? Or spent your on-call time playing with the customer's big yellow toys? [2]On Call is waiting to hear from you. ®
Get our [3]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] mailto:oncall@theregister.com
[3] https://whitepapers.theregister.com/
Re: Data Entry
Certainly there is a story that occasionally does the rounds (can't verify it) that the QWEWRTY keyboard was designed to both improve efficiency in typing, but also limit speed as touch typists were more than capable of snarling up an old typewriter once they had practised for a while and were up to speed.
Re: Data Entry
I understoot it is so keys (well, the actual hammering thingys, too lazy to look up the name) don't interfer with another, jamming up the typewriter. You change back and forth between different "columns" (ok, you change sideway, not back and forth) and indeed sides of the typewriter keyboard so that keys adjacent to each other seldom get the chance to stick together.
Re: Data Entry
Dear mother is ex-typing pool and could type at 110WPM. She moved into the libraries IT for book data entry in the 80s and could type so fast that the terminal crashed. I remember her saying that the first computer that could keep up with her was the 8086 that my Dad was given to work from home on.
"bad-mouthing the system"
Hardly surprising though that not everyone was happy with the new technology. It must have been a nervous time for some people. Computers & robots were replacing a lot of workers on the factory floor / typing pool, etc.
Live that everyday
When is it going to blow up again... The system isn't robust...
Re: Live that everyday
Yeah... We've got that issue. Just waiting on the server to be replaced. It's only a Compaq unit... With 10s of millions of customer records in it. No biggie that the hardware is 10+ years out of warranty and risking caching batteries leaking at any moment.
No rush bean counters....
Oh disaster recovery? Yeah I asked about that too. Got stopped when the warranty ran out as it cost money to test...
Nervous much?
Re: Live that everyday
Make sure you have all your recommendations for replacement and distater recovery on paper (hard copy so it doesnt go down with the ship). Then when the inevitable happens you can show a long paper trail of requests for replacement and offsite backup, that were stymied and blocked by beancounters and management.
When the Axe falls, make sure there's someone else under the blade...
Re: Live that everyday
When I was but a baby programmer, a grey and grizzled developer imparted their career advice in terms of sure certainty: "You'll see a paperless toilet before you see and paperless office, and always keep a CYA file". This wisdom has followed me down the decades.
Re: Live that everyday
I was glad of that once when I had a load of recommendations and project plans from the previous 2 years to put 2fa/mfa in place for our email and the CEO fell for a phishing scam. Everyone asked 'why didn't we put some better security in place' and were rewarded with me pointing at the board level IT director as he was the one who refused the spend.
Re: Live that everyday
Yep -Thats exactly what the Risk Register is for. "The Business" can then choose to deal with the risks that have been flagged however it likes.
Re: Live that everyday
Which, in my sad experience, means ignoring it and hoping they won't get compromised. When they do, lie their arses off to limit the damage to reputation. As a bonus, try and shift the blame elsewhere (and yes, I am a great believer of the CYA method of filing - its saved me on a number of occasions)!
Re: Live that everyday
Reminds me of a time well over a decade ago now. I was resposible for the co-design and migration from Exchange 5.5 and x.400 on NT4 to Exchange 2003 and SMTP.
There was one single Compaq server in a rack that no one dared to turn off for a cold P2V because of the very high probability it wouldn't come back up. Apparently it was prone to this. It also happened to be the NT4 PDC (with Exchange on it!).
I was working for Microsoft at the time and this was in Belgium at a rather huge site just outside of Mons so the user base was enormous.
It took me lots of persuasion to allow me to use non-Microsoft tooling but even then, eventually, we had no choice but to cold clone it because with all the best will in the world we could not get it to hot convert.
To mitigate as much as we could we did manage to buy a small number of second hand spares from eBay but it was still a nervous operation.
It was a long time ago so I may have some bits slightly out but you get the jist. :)
Re: Live that everyday
That still goes on today. That businesses let it come to the point that EBay is the place they need to go to keep their mission critical systems running is just insane (and I am not talking about small businesses here - I am talking multi-million pound businesses).
Brownouts (fnarr fnarr)
I used to work somewhere that had one of the first HP/PA HPUX systems (I think, mid 1990s) outside the USA. They were very pleased with their automatic voltage sensing auto-switching PSU that just worked with 110V or 240V or whatever, supposedly. We were in Barrington Hall, in a village. At least the place had 3-phase supply.
We had brownouts.... we also had power cuts and the supply was delicate enough that procedure was to turn off everything we could when it went dark; sysadmins would then come round and turn it on one by one so that the startup load of all the discs didn't happen all at once and trip it out again.
But in some shapes of brownout, the HP/PA machine's clever PSU auto-switched to 110V. Great. But it was very slow at switching back to 240V when the mains came back up, and failed with smoke. Every time the project was delayed a week whilst they shipped another PSU; after some months they sent a better PSU without the stupid in it, and all was well.
Was it Diggerland?
I want it to be Diggerland
Re: Was it Diggerland?
I took the kids to Diggerland before they closed for the winter (early birthday for the youngest).
Despite having 'land' in the name it was surprisingly good fun for all.
Big motors
I once investigated an issue with Internet dropping at a custom-mix concrete place. The server was fine but periodically their Internet link would drop. Eventually we linked it to every time the 70 kW motor which mixed the concrete started up... moving the router over to the UPS (along with the server) solved that one.
Data Entry
We were involved with a similar role out and hit the problem of preparing the system with existing data. Decided to call upon the services of the, soon to be made redundant, typing pool and they were brilliant! Once they got going and realised that each data sheet was identical and they had to tab through each field including null values the stuff was going in at fast rate. One problem, they were entering the data that fast and being touch typist they very rarely needed to look at the screen the screen refresh rate was struggling to keep up and the buffer could only cope with so much. We had to ask them to pause every 5min to allow the buffer to clear and the screens update :-)