Shock horror – and there goes the network neighborhood
- Reference: 1699255809
- News link: https://www.theregister.co.uk/2023/11/06/who_me/
- Source link:
This week, meet "Vincent" who once, many many moons ago, worked for some very important people at a very important American university. Very importantly, he worked in the Telecommunications division at said university, and was responsible for (among other things) maintaining the switch gear in a cabinet.
Now that might not sound very important, but the thing to understand is that the university at the time operated a network of T1 circuits that connected all of the various campuses of the university around the state. Every campus had a T1 switch that handled voice, data and video conferencing traffic, and they were all ultimately centralized in the cabinet in Vincent's office, which was labelled "Network Equipment Technologies."
[1]
So it was, you might agree, a very important cabinet.
[2]
[3]
Now, mostly what Vincent did was monitoring the switching gear in the cabinet via a Sun workstation and vendor-specific software. Lots of mouse-clicking, lots of typing – not much actual opening of the cabinet.
As it happens, though, the manufacturer of the power supplies for the bandwidth management system had provided easily-accessible test points on each unit. And each one was helpfully marked with the voltage that should be passing through at any given moment. It was not, strictly speaking, Vincent's business to monitor the power supplies – but the opportunity to do so was right there, and so much more interesting than clicking a mouse and typing.
[4]
Thus, one fine day, multimeter in hand, Vincent set about checking that each power supply was putting out the voltage it was supposed to be putting out – despite the fact he had absolutely no reason whatsoever to believe they were not operating as intended.
Now, you know exactly where this is going, don't you?
[5]After nine servers he worked on failed, techie imagined next career as beach vendor
[6]That script I wrote three years ago is now doing what? How many times?
[7]One door opens, another one closes, and this one kills a mainframe
[8]Scripted shortcut caused double-click disaster of sysadmin's own making
As he was unsocketing the multimeter probes from one of the test sockets, his grip on the tool slipped ever so slightly. Just enough for one of the probes to touch another socket, which created a small spark. The spark was just enough to alert the bandwidth monitoring system that something was wrong, and send the entire system offline in an instant.
Suddenly, the entire network was down. Every campus of the university, all across the state, had no internet access, no video conferencing, no VOIP system. Nothing.
Thankfully, it was a well-designed system and sufficiently robust that there was no lasting damage. Systems began restarting almost immediately, and the whole outage lasted perhaps ten minutes before it was back to normal.
[9]
Vincent knew he had a very short time to get to a position where he looked innocent. So he very quickly packed away the multimeter, closed the cabinet, and hurried back to his desk. He had just enough time to do that before his boss stuck his head around the door, wondering what in blazes had just happened.
"I heard a noise, looked up, and everything was restarting! Not a clue what happened, honest!" he sputtered. The boss, thankfully, believed him.
Every had a lucky escape like Vincent? Allowed your enthusiasm to exceed the scope of your job, or your expertise? Tell us all about it in [10]an email to Who, Me? and we'll share your story with others, to brighten someone's Monday morning. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZUjHUU@KrPqYqJ2uXm1AaAAAAJM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZUjHUU@KrPqYqJ2uXm1AaAAAAJM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZUjHUU@KrPqYqJ2uXm1AaAAAAJM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZUjHUU@KrPqYqJ2uXm1AaAAAAJM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/10/30/who_me/
[6] https://www.theregister.com/2023/10/23/who_me/
[7] https://www.theregister.com/2023/10/16/trainee_learned_an_important_lesson/
[8] https://www.theregister.com/2023/10/09/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZUjHUU@KrPqYqJ2uXm1AaAAAAJM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: The last time I heard a loud noise and things were restarting...
> I have this issue with most of my bosses: They cut corners to increase their bonus
Not a shock...
If you ever tell a lie, never ever add honest/honestly to your innocent gibberish. Honestly, it will definitely give you away.
The secret to successful lying to to make yourself look at complete idiot.
If you're late in to work you could try saying the bus was late/that car broke down etc. No one lends it any credence.
On the other hand, if you say "I crapped myself! I had to go home to clean myself up and get changed!" no one will ever question it.
to make yourself look at complete idiot.
Then you can be just telling the truth.
If you're late in to work you could try saying the bus was late/that car broke down etc.
If your car broke down, actually take photos, same with the bus. Get the receipts. Always.
Tell the boss you can't afford a better car and maybe if they could give you a raise your attendance could improve.
Or if you got late because of the bus, just say you couldn't get out of bed, because all you see that you are just a "bus w*nker" and then it came late.
and talk about that raise.
anyway if you keep coming late, just use different time zone.
Its happened at a University ...
... so blame the students.
I have learnt over many years to leave things well enough alone. Merely the act of checking on something can cause a system that had been working perfectly to break.
Blissful ignorance is my happy place
Even if it doesn't immediately break, it then becomes your responsibility if it ever needs anything. Because you're the one who knows what it was like when it was working properly.
That cat again...
《I have learnt over many years to leave things well enough alone. Merely the act of checking on something can cause a system that had been working perfectly to break.》
Not peeking inside the box lets Schroedinger's cat have a happy or at least longer half life ;) ie don't go about recklessly collapsing wave functions.
When checking voltages...
... it always helps if you make sure that the meter is set to measure voltage, not current.
Re: When checking voltages...
If no-one was watching, it didn't happen.
Re: When checking voltages...
What is someone wants to measure current voltage?
Re: When checking voltages...
Watt?
Re: When checking voltages...
Ohmy!
Re: When checking voltages...
Resist the temptation
Re: When checking voltages...
《it always helps if you make sure that the meter is set to measure voltage, not current.》
With very few exceptions I wouldn't let anyone "detained" within a University near a screwdriver let alone a multimeter.
There are those that don't know the difference between EMF(Voltage) and current, those that think they do and a very few that do. I am not sure that it ultimately makes much difference :)
There are individuals in those institutions whose very proximity to any piece of equipment more sophisticated than a slide projector can rapidly cause said device's mysterious failure.
Elsewhere never let a developer anywhere near critical equipment - lock up your gear if you cannot lock them up.
My long deceased but probably sectionable granny didn't often make much sense but "the devil finds work for idle hands" was one the few.
Working for a submarine telecoms cable manufacturer in university holidays when I was required to get a bit of "shop floor" experience. The company was pretty good about letting me have a week in several different departments which ended up in the "final testing" area of the shore-end equipment - think lots of full height 19" telecoms racks.
I was doing the final tests on the "spare set" for something like TAT-9 (which dates this whole story if you look that up) prior to delivery to BT. Along with various messing around with optical fibres and high speed data testing equipment, etc. the final tests were "how much current does each shelf in the rack draw?". The technique was to grab one of these wheeled stair/ladder things, go up to the top with your multi-meter to where the fuses all were, set it to current mode, put the probes into the terminals on each side of the fuse, hold them with one hand whilst you unscrew the fuse with the other. You can read the current, then screw the fuse back in and write the current down on the test sheet - all of which means you get the normal operating current without anything being turned on/off. You then rinse-and-repeat down the line of fuses at the top of the rack.
The final measurement was supply voltage to the rack. This came from a standard telecoms 48V DC supply (think big pile of lead-acid batteries).
Notably the instructions in the test did _not _ include the helpful advice to set the multi-meter to voltage mode first.
As a multi-meter in current mode is a basically just a thickish piece of wire the resultant spark and clunk from the circuit breaker on the battery pack were quite impressive. Followed by the extensive "rattle" as every relay in the various PSUs clicked off and the descending whines of the many, many fans in _all_ the racks that were currently under test ...
Apparently I went white-as-a-sheet which I suspect did something to dampen the laughter.
Thankfully the foreman scuttled over pretty quickly to put me out of my misery and said something like "Don't worry - it's just the circuit breaker, the racks will be fine. Everyone does this - once. You forgot to set the multi-meter to voltage didn't you?"
Why did you need to remove the fuses? You should be able to calculate the current if you know the fuse resistance right?
(Not an electrical engineer, just asking!)
So when I was younger and a lot more stupid, I was on my own racking an old A-Class server. It was an awkward height, and I was doing the work at the end of the day after racking half a dozen others (so I was getting a bit tired).
Queue me trying to get the damned rails in and my arms just gave out. Luckily the thing missed me, but it didn't miss the floor. Bent the front panel a little, but nothing I couldn't fix up. The system still booted up and ran fine.
My one and only secret screw up.....
The last time I heard a loud noise and things were restarting...
...the UPS battery had failed and sent a voltage spike to the stuff it was supposed to be protecting. Luckily most of it was salvageable and my boss invested in a line conditioner to sit between the UPS and the "important" hardware. I have this issue with most of my bosses: They cut corners to increase their bonus and send the employees that trigger the traps they set to the bread lines. "Vincent" is a lucky man.