It's the day before the grand opening but we need a firmware update. It'll be fine
- Reference: 1640596273
- News link: https://www.theregister.co.uk/2021/12/27/who_me/
- Source link:
Our tale takes us back to the '80s, the decade of fun. Our brave reader – let's call him "Gareth" – was heading up a team building a state-of-the-art Brewery Process Block.
It was quite the thing to behold. "They literally came from all over Europe to see it," boasted Gareth. The clever modular design meant that bits could start being water tested even if other parts were not quite ready. It was also up and running – handy since a grand opening was planned.
[2]
And what an opening it would be. Heads of government and churches had been invited. Had the ceiling fallen in, a good few countries would be in need of leaders.
[3]
[4]
However, it wasn't the ceiling the boss was worried about. Looking at the plans and attendees, he blanched and parachuted in more people to be on hand. Just in case.
"He promised me that they would be invisible," said Gareth, "but said that, 'If we mess this up we'll never be able to sell another system.'
[5]
"Fair enough."
It was the eve of the opening ceremony, and things appeared good. Gareth was looking forward to a starring role in a lavish dinner that night, where garlands were due to be rained down upon him for his extreme cleverness. Also, there would be free booze and food.
However, the engineers were about to lob a spanner into the works. The system had never been shut down during the commissioning process. The hardware was new to the market, and none of the team had much experience with it. It was decided a firmware upgrade was required, which involved an hour replacing the ROMs and restarting.
[6]
A brief note about the system involved. There was a bus into which a bunch of Motorola 68020-based (a variant of this CPU was found in the Commodora Amiga 1200) cards were plugged. These cards could be sequence controllers (running custom software tailored to the plant), network interfaces or controllers that talked to the 300 or so interface cards that actually ran the plant.
Each card also sported a megabyte of RAM and firmware containing the operating system.
A firmware upgrade the day before all the bigwigs were due to admire the system in apparent operation at its opening? What could possibly go wrong?
The engineers did their thing, over all the boards since the powers-that-be decided that the big-bang approach made for easier maintenance. The system was restarted.
"The printers burst into life, logging the boot, and then... nothing.
"Repeat. Ditto."
[7]Ooh, an update. Let's install it. What could possibly go wro-
[8]How to destroy expensive test kit: What does that button do?
[9]When civilisation ends, a Xenix box will be running a long-forgotten job somewhere
[10]A tiny typo in an automated email to thousands of customers turns out to be a big problem for legal
Twelve hours of desperate investigation later, during which time Gareth missed his shot at stardom (and the wining and dining that would accompany it), the team solved the mystery.
"The system bus was simply overwhelmed so that the 10 boards – all with a 68020 CPU aboard – were waiting for a 'proceed' flag." Although there was no helpful error message.
"Nobody had ever come across this before so it took us a loooong time to track down the problem."
And the solution? Popping in a WAIT instruction at a strategic point in the startup code to allow the traffic to clear.
Hurrah! The systems came back to life!
"With 20/20 hindsight," admitted Gareth, "we should have taken an 'If it ain't broke...' approach.
"We could always have reverted to the old firmware I suppose and would probably have done so come midnight.
"But the opening went well. Phew!"
There's an unwritten rule in the IT world about Friday afternoon deployments. However, this is possibly the first time an organised, er, event in a brewery was almost knocked off course by an ill-timed update. Unless you know better? Let us know, with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Ycmc3GRDdFSzw7rSO0If@AAAAMQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ycmc3GRDdFSzw7rSO0If@AAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ycmc3GRDdFSzw7rSO0If@AAAAMQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ycmc3GRDdFSzw7rSO0If@AAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ycmc3GRDdFSzw7rSO0If@AAAAMQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2021/12/13/who_me/
[8] https://www.theregister.com/2021/12/06/who_me/
[9] https://www.theregister.com/2021/11/29/who_me/
[10] https://www.theregister.com/2021/11/22/who_me/
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Re: Friday rule
I've generally found that 'touch nothing no matter what' is a better mandate, unless in he club at 2am....
Re: Friday rule
He hops they won't do it again...
They had it running
Then they had to go for a firmware upgrade. With no apparent good reason.
Here's an idea : you fully test that the system can do what you need it do the next day, then you make up your mind on the firmware upgrade.
If the test had did badly, then yes, you upgrade, but if the test went well, you don't touch a damn thing until after the event.
Oh well, experience is what you get when things break unexpectedly - especially when it's your fault.
Windows upgrade in process
I remember going to a presentation of a great new technology and just after the demo started Windows update kicked in. The download activity was enough to impact the network connection, and instead of a slick demo it was a stuttering display. The audience offered suggestions on how to kill the update, but the marketing person did not want to go off script.
Re: Windows upgrade in process
The root cause for this is almost always one ting in my experience....
The presentation is usually carriend out on a machine who's ONLY job is in a meeting room.
A few years back, we would have a meeting every Thursday (you see where this is going) - the only meeting ever in that room @ 9am
.....and at 9:05am, it would dutifully start applying software patches pushed out to it by whatever patcher we employed.
It took a month before someone realised it was beneficial to move the meeting, but switch the kit on well in advance - or the day before to patch in advance
Re: Windows upgrade in process
Back in the Windows XP days, my company had a standard image with a tool that would checksum c:\program[sic] files, c:\windows etc to make sure that all patches had been applied and nothing had been changed.
This wasn't a bad idea in principle, but it was configured to run every four hours and in the days of single core Pentium 4s and hard discs this meant your laptop turned into a paperweight. This actually killed a number of high-profile presentations and generally made it hard to use your computer
why is it that people forget flow control?
Every component of interlinked systems that I have every worked on (Including the human-driven ones!) always involves flow control somewhere. In-band or OOB, it's always present.
IMHO, flow control make thing work smoothly, and 'bottleneck' restrictions are easier to identify, as long as you bother to have flow on/off assertions visible somewhere. You can then tweak the setup as required to improve things.
Also, if one is dealing with something synchronous you need to 'Think Synch'...
Firmware upgrades
Should really only be applied when.
1. It won’t run without them I.e. a serious bug
2 there is a massive security issue i.e. getting hacked
Otherwise test it properly before starting- but hindsight is wonderful…
Certainly not the night before a demonstration or high profile event.
If you need to do a fix then test it first, years ago I worked for a company building control machinery for factories, and one I remember if the factory got shut down as it made plastic sheeting it would have taken a week to get it back up and running, so that was approached very carefully. The problem was the heaters went off when the stop button was pressed so all the plastic solidified in the machine…
the heaters went off when the stop button was pressed so all the plastic solidified in the machine
A well-known problem with all sorts of stuff from chocolate via glass to steel. And our current 'strategy' for supplying electricity is going to generate more of these incidents.
Could have been worse....
memory issues with the dram.
internet connectivity with too many hops.
Maybe they needed to fix their cereal port...
Friday rule
Touch nothing unless you have no alternative.