Why yes, I'll take that commendation for fixing the thing I broke
- Reference: 1615796110
- News link: https://www.theregister.co.uk/2021/03/15/who_me/
- Source link:
Our story takes us back several years to when our hero, helpfully Regomised as "Henry", had taken a job working at a large, multinational bank.
[2]
Henry had been employed on the networking side of the institution and, keen to impress his superiors with both his skill and initiative, spent the morning of his first day there getting familiar with the network configuration.
There was a dedicated transatlantic link to the New York branch, which in turn acted as a hub for the rest of the US branches. All relatively straightforward stuff, and Henry continued his inspection until he spotted something odd. Compression had not been enabled on that transatlantic link.
[3]
It seemed a strange thing to not have enabled – doing so would surely make for optimal use of the bandwidth available. Money might be saved and transactions processed faster. He could well end up employee of the week, month or even year on his very first day!
Confidently, he enabled compression and took himself off to lunch, a job well done.
Just when you thought it was safe to enjoy a beer: Beware the downloaded patch applied in haste [4]READ MORE
The glow of satisfaction was short-lived. Upon his return the office was in uproar: "They'd lost the link to the US, no one knew why, and it had happened just before lunch..."
The manufacturer of the hardware is lost to the mists of time. The memory of Henry's bowel-loosening realisation that his first day might well end up being his last day has not.
A quick thinker, he offered to help in the "investigation" – what could possibly have happened to upset the network so? After allowing a respectable period of time to pass, while those around him flapped like headless chickens, he made his, er, discovery ("not too soon as to make it obvious"), pointed out the compression setting and carefully asked if he might possibly be given authorisation to disable it?
After all, what sort of lunatic would go thundering through a bank's carefully crafted networking settings without proper permission? Not Henry. Oh no. Not him.
He reversed his earlier not-as-helpful-as-thought tweak...
And...
The new guy had fixed the bank's longest and biggest outage, and had done it on his first day! He was a hero! A commendation was stamped on his record in recognition of the save.
We rather hope the bank has implemented stricter measures in the years since. And Henry? The needs of anonymity forbid us from discussing what became of him in later years.
Suffice to say we'd not want to take him on in a game of poker.
[5]
Ever had to test your poker-face while all around you panic? Or picked up a bonus when a visit to the unemployment office seemed more likely? Confess all with an email to [6]Who, Me? .
Get our [7]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YE8@TFO6VWBJRXGr4Pw@RQAAANA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YE8@TFO6VWBJRXGr4Pw@RQAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2021/03/08/who_me/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YE8@TFO6VWBJRXGr4Pw@RQAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] mailto:whome@theregister.com
[7] https://whitepapers.theregister.com/
Re: Experience is the best teacher
Yes, but it's probably the other way round, no?
" I'd wager that nearly every reader here has felt that same "bowel-loosening realisation" at least once in their career... "
which in turn lead to:
" We all know that proper Change Control is Super-Duper-Important "
TFTFY
Intelligence = learning from your own mistakes
Wisdom = learning from others' mitakes
Let's just say that I likely am intelligent. Not wise.
Re: Experience is the best teacher
In the early days, it was all too common ... Partly because many of the concepts were so new that there were no instructions as yet. We were making it up as we went along.
Re: Experience is the best teacher
Forgetting the WHERE clause when running an UPDATE statement on a production server is not a horrible mistake; it's a rite of passage.
Re: Experience is the best teacher
How many days would be saved by smss and its brother in warning you that you were executing a selection and that immediately after the section is a WHERE clause you may have missed accidentally
Re: Experience is the best teacher
MySQL has --safe-updates which stops you doing an update/delete statement without a where clause; this got added to all the client my.cnf on our servers after someone messed up a production DB. Amusingly, in the pre-Sun days, it used to be called --i-am-a-dummy
Re: Experience is the best teacher
Trouble is that "proper change control" invariably ends up implemented as "everything has to be signed off on by this bunch of rabid Nazis who find their phone a technical challenge".
Everything takes longer, deadlines get missed because the wrong font was used in the change submission, critical fixes cannot be deployed because CC won't provide an exemption for something they don't understand and never will.
Auditing?
Sounds like the bank needed to up their logging / auditing game.
Re: Auditing?
Back in the early days, it was fairly common for me to arrive on site to troubleshoot, pull up the logs, and be asked with shock (and sometimes more than a little awe) "You can DO that? WOW!" ... The same type of people were even more flabbergasted when we could do it remotely. It was all magic as far as they were concerned.
Re: Auditing?
Lots of places I've worked, I would be very surprised if it could be done. Most times logs were disabled in order to save disk space...Fortunately it was a long time ago... most of the cases....
Anyone who refuses to put their hands up to something similar has never worked in IT...... Of course that's in the old "wild-west" days, these days change controls etc make this sort of thing very rare.....
Documentation
Was there documentation of the set up? Documentation of why things were done this way? Even more important, why things were not done that way? Or did ne not bother to read it?
The good old days when management believed everything
When one’s section head of technology (they could not call him Section Head of IT for obvious reasons) has decided off his own bat to conduct a test of our new Cat 6 cabling by telling staff to pull each cable in the hub to check that it is firmly locked into position.
2 hours later, as his boss, I get to take all the credit for fixing the various outages by creating a very long memo to the effect that we followed up reports of ‘stretching’ in IT network cabling by testing each cable, and I even got someone working at the IT desk in a non existent department outside Cheltenham to confirm this.
Pressing the big red button
With the "DO NOT PRESS" sign, is a rite of passage.
Occasionally though your passage may experience problems.
A good laugh was had this side of the world.
Excellent.
Experience is the best teacher
We all know that proper Change Control is Super-Duper-Important, and yet I'd wager that nearly every reader here has felt that same "bowel-loosening realisation" at least once in their career...