So Steve identified a presumably undocumented single point of failure, caused an outage that presumably didn't kill anyone, and learned an important lesson meaning he'd almost certainly do things differently from now on, making him a more valuable employee......

And they fired him. Bravo.

Unless this was genuinely a pattern of reckless behaviour that had already been called out, I just don't get some companies' attitudes to throwing techies under the bus like this, especially where their own line management is complicit.

It's more likely to drive people into covering things up when they make mistakes, rather than being open and honest.

There are perhaps better ways that it could have been handled - such as giving notice that there may disruption, yanking kit out on the weekend instead, additional monitoring/alerting for critical services, etc. But eventually *someone* was going to come along and unplug that box, whether they've done any 'due diligence' or not.

But, as you say, the response that ensued was the telling part. Had Steve not ultimately lost a job over it, he'd have done well to find another job elsewhere and steer clear of such a toxic culture.

It is sadly all too common to throw people under the bus rather than address the challenges and issues head on, but there are people in senior leadership positions who will take the latter path. People who genuinely value you for more than just how you contribute to profits; people who wouldn't even countenance using you as a scapegoat and would instead throw themselves under the bus first; people who would offer themselves up to the chopping block ahead of you when it comes to redundancies. I've worked for a few people like that and it's genuinely transformative.

No job/workplace is perfect, but good leadership makes all the difference.

>There are perhaps better ways that it could have been handled - such as giving notice that there may disruption, yanking kit out on the weekend instead, additional monitoring/alerting for critical services, etc. But eventually *someone* was going to come along and unplug that box, whether they've done any 'due diligence' or not.

>But, as you say, the response that ensued was the telling part. Had Steve not ultimately lost a job over it, he'd have done well to find another job elsewhere and steer clear of such a toxic culture.

Steve (and his boss) were performing unapproved change work in a datacentre without informing anyone of what they were doing, when they were doing it and why. This is no longer the wild west of the 1990s - the story happened this decade - and you can absolutely guarantee that they broke a shedload of company rules designed to prevent people doing cowboy stuff like they did.

As he was contacted by his boss Steve would have every reason to believe that it was approved, especially with a solid reason of reducing costs by eliminating redundant rack space. If anyone were to get fired it should have been the boss.

OTOH it does seem a bit extreme to remove an unidentified bit of kit without documenting it. Maybe they were a bit stir-crazy after lockdown.

The word "presumably" is doing a lot of work there.

Another way of looking at it is that Steve and his boss found something they didn't know about, failed to make any checks and removed it without recording what they were doing, thereby causing the company huge losses.

Sure, Steve may have been a better employee afterwards, but the starting point sounds pretty low.

If taking out one seemingly minor box took the entire company down, there are bigger issues at stake.

As other people have said, why wasn't it clearly labelled if it was so important? Which moron had "architected" it such that there was a clear SPOF? Or which bean counter had nixed the resilient approach that an architect had come up with?

If anything there should be more chaos monkeys like Steve in organisations, because then things will be architected and built properly, instead of corners being cut all over the place.

Your average enterprise IT deployment is full of bits like a Brazilian favela that's been built with badly mixed concrete and is now held up with sticks and duct tape. Stuff that wouldn't pass building regulations in a million years. Yet even though western society has relatively stringent building regulations for physical buildings, we outsource design and build of critical IT systems to the cheapest, and don't apply any engineering rigour to any of it. Sacking people like Steve who are trying to make do with the sticks and duct tape doesn't fix the underlying problem.

> that presumably didn't kill anyone

Who knows? He did break lockdown rules.

Without more details you can't say.

He went on an unauthorized, and not known about visit to the data center that wasn't in any way time critical.

Why?

If everyone had done that during lockdown there'd have been chaos.

It wasn't as if he was the British prime minister.

There were different levels of lockdown at different times during 2020/1 - with different rules for different areas too.

Without knowing more details about when and where this took place, you can't claim anything about rules being broken.

Well, driving alone by car to a datacenter that is by definition well ventilated, working with only one other employee in a room otherwise devoid of humans (aka potential infection vectors) - this would qualify as fairly low risk, from an infection preventon point of view.

That's one way of looking at it, but you don't do shit like that without a plan, especially in a financial organisation where there can be *severe* penalties and difficult conversations with regulators.

They should have labelled the device with a clear sign saying "WHATEVER YOU DO, NEVER DISCONNECT THIS DEVICE!!!"

On the other hand, people being people, the paint would not have time to dry before someone unplugged it to see what would happen.

Aha, a PTerry fan!

It was already bound to happen, and unfortunately Steve was the chosen one.

Honestly, the person who set that shit up should've been fired. And they should've, at the very least, documented it or have put it inside a basement far away with signs saying "Beware of the leopard" .

> And they should've, at the very least, documented it or have put it inside a basement far away with signs saying "Beware of the leopard".

That'd make it hard to spot

We don't know that it was undocumented. All we know is that Steve and his boss didn't read any documentation.

The chances of up-to-date and accessible/discoverable documentation for a piece of kit that's *powered off* in a large organisation is relatively small. Documentation should be kept up to date. But that doesn't happen as diligently as it should.

It's generally a pretty safe assumption that if something that appears to be an active piece of kit doesn't have power then it's not "active".

I've never come across a piece of powered hardware yet that effectively acts as a patch panel passing network connectivity through even when it's disconnected from power. Power cord plugged in and device "off" is a different matter. But no power cord at all - the mind boggles at what this piece of hardware was doing and why it had been designed that way.

Power over Ethernet? Lots of critical network kit uses that, so that it can be powered from a central Ups-protected location. Steve should have checked.

From only a limited sample, but my PoE devices have lights, even when connected to the PoE router but not actively drawing power.

Ok, a managed PoE router may have been told to shut down power delivery to that port, to extinguish all the LEDs, but still keeping the port active for traffic - this is getting needlessly complicated![1] And if they have infrastructure that is complicated but no fallback for this bit of exotic kit...

Hmm, Steve may have got the boot but hopefully IT were also given a stern talking to.

[1] or it is one of the sneaky spy devices jake mentioned

There's nothing in TFA to say it had a power connector. Reading it I'd assumed power over ethernet although it could have been a passive filter or surge protection device.

Plenty of nics with a watchdog which will close a relay to bypass the card if it isn't regularly poked from software.

That would make the device a massive, and expensive network coupler, presumably coupling two cables from a patch panel which would be better off just directly connected...

We know it looked distinctly out of service (not powered up) and yet wasn't prominently labelled to warn others that this was very unusual kit (a sticker over the strangely unused power socket , with a note explaining why - which may, in a bad world, be a reference like "read doc#17, p.472").

Heck, I just wrote a sticky label to go next to an LED because I couldn't figure out or remember why it was on when the rest of that computer appeared to be off - and I only built that machine from parts last November!

Not leaving on-site labels for anything that is very out of the ordinary is shoddy. Yes, including kit that has been decommissioned but left in the rack: masking tape and a sharpie, at the minimum.

THAT is the sort of IMMEDIATE documentation that was missing!

> "Not leaving on-site labels for anything that is very out of the ordinary is shoddy. Yes, including kit that has been decommissioned but left in the rack: masking tape and a sharpie, at the minimum.

THAT is the sort of IMMEDIATE documentation that was missing!"

Amen.

Like code, I think the same should apply to hardware -- What [the hardware] does should be self-explanatory, and if it can't be, then document it!

And always document the why

Honestly, the person who set that shit up should've been fired. And they should've, at the very least, documented it or have put it inside a basement far away with signs saying "Beware of the leopard".

I see your HHGTTG and raise you a Pratchett...

If you put a big red button in a distant cave with a sign saying "End of the world switch, do not touch!" the paint wouldn't even have time to dry.

Looks like pulling the wrong network kit out cost him a packet

I had it drilled into me very many years ago: "If you don't understand it, ask someone who does."

Later in my own advice to juniors I added: "If you can't find anyone who understands it, leave it alone"

The problem is when you can't find *anyone* who understands it. Or the people who *do* understand it have all either left, retired, or "moved on" (metaphorically speaking).

Sometimes the only way to know what some unknown software service, hardware, etc, does is to "pull the plug" and see who starts shouting. You can try to do that in as controlled a way as possible, but you can't let perfect be the enemy of the good - because sometimes there isn't a "good" way to go about it.

> "pull the plug" and see who starts shouting

We did that once. Proper checks revealed nothing, so we pulled the plug to see who complained.......

Nothing happened. So we eventually went home.

Then at 6am the plant nightshift couldn't clock out, because some clowns had unplugged the box that controlled the timecard punch clock.

> > Leave it alone

> The problem is when you can't find *anyone* who understands it. Or the people who *do* understand it have all either left, retired, or "moved on" (metaphorically speaking).

> > Leave it alone

Sometimes your role is to be the one who comes to understand things. That isn't ignoring the things, and it may be that pulling the plug and seeing who comes hollering is part of gleaning understanding. But.. in due time, with due notice (as much as possible), and with understanding of the boss. :-)

"it may be that pulling the plug and seeing who comes hollering is part of gleaning understanding"

I've sometimes thought that might be the best way of dealing with manglements who think IT is just a cost.

I was once running data centre cables from network box to patch panel to structured cabling to patch panel to patch panel to structured cabling, etc, to get back to the core switch I needed. Having got there I found that the port I was allocated already had a cable in it. I contacted network management and was told to remove it and connect my cable. They would trace the erring connection and reconnect it properly. I found out later somebody had 'noticed the wiring was untidy, and bunched everything up to the first row of ports.' Presumably at least most connections had stayed within the correct VLAN, or it would have been noticed somewhat faster.

The worrying thing there is if you're the person who's supposed to understand it...

Some years ago, the opposite happened to me: A major broadcaster was replacing _all_ its internal and external analogue (and digital, of various flavours) circuits with shiny new digital circuits. We spent ages (years) planning this, because of the requirement never to go off air... All sites had main and standby systems, usually in different rooms, different power circuits, different entry points for the signalling etc and because of this we could set things up on the reserve circuits, switch to them, make the main circuits live, test, and switch back - no interruption.

Excellent. Over a whole country, no interruption visible to the viewers.

But... most rack mounted video equipment in those days used either analogue or digital inputs and outputs from the back of the rack, controls and indicators on the front. The cabling was loomed into great thick (but very tidy, when installed) bundles down the back of the racks and under the floor. Planning had revealed that there were whole racks which would become redundant, but whose space would be occupied by new kit. For very rare cases where equipment had to be moved within a rack, it was usually possible to move it up or down in the rack, or remove it, with a bit of jiggling, through the back - without interrupting its operation.

So, one apparatus room at a time, things were changed. Until we arrived at the one that spoiled our error free record: some bozo had wired a single unit in place with its output going round the wrong side of the rack frame. This unit carried the output from the station; it had to stay live. It also had to be temporarily removed from the rack so other stuff could go in its place; and there was no physical space to get it out the back because of the wiring routing... so this single wire ruined our two-year-without-interruption by seventeen seconds as it was disconnected, pulled out, and reconnected.

Given the time in the morning, I do wonder if anyone noticed, but still, it gripes!

"If you can't find anyone who understands it, leave it alone" ... unless it's actually on fire.

Lots of comments shocked that Steve got fired. Did you miss "Neither of us were allowed to visit the datacenter without approval from the very top, let alone ripping out servers without raising a change." ?

Steve's boss suggested they go. It's not wholly unreasonable for Steve to assume that it's a sanctioned trip. Combined with some COVID lockdown stir-crazy-jumping-at-the-chance-to-see-a-different-four-walls mania I can see how he might have not asked as many questions as he perhaps he should have done.

If anyone should be carrying the can for the escapade, it's Steve's boss, not Steve.

So that was reason to chuck the boss, who introduced Steve to join the wrecking crew. We don't hear that boss man's eventual fate.

I'd read that as being a new rule imposed in consequence but looking at it again I think you're right. However if his boss initiated Steve had every reason to suppose it was all kosher.

How can something that's powered down "provide a vital network connection"? Maybe when powered down it somehow acted as an RJ45 coupler? Because I don't care how "exotic" it is, it can't be routing or firewalling if it doesn't have power!

Or was it some device small enough to operate via PoE, and they wrongly assumed because it wasn't plugged in it wasn't running but didn't bother to check for any LEDs?

I have seen a couple boxen that were PoE when at idle, with no LEDs or other indicators that they were "live". When the proper wake signal was received over the LAN, they quietly did their jobs, still with no power indicators. The only time they had lit LEDs was when they were being configured with a (dumb) terminal plugged directly into 'em.

Yes, they were (and still are, in at least two cases I am aware of) clandestine security devices.

Someone should tell them that security by obscurity is no security at all ...

I've come across devices which have some form of relay (Might not be mechanical but you get the idea) where if the device has no power, the traffic by-passes the equipment. The equipment has to have successfully booted for it to flip the relay for the traffic to actually be processed.

Seen those as well, as a failsafe for when the device is unwell (the relay only flipping after the device has not only powered up but also POSTed).

But if you've gone to the trouble of removing the power cable then you could also replace the box with a plain old coupler. Seems odd to leave it the way it was - and anything odd should have a clear label attached!

This is from the "stupid tired operator tricks" files ...

Several billion years ago, as the Internet measures time, call it 1986ish, I was burning in some two dozen full racks of T-carrier gear.

After the end of the ten day burn-in period, one of the final tests was to physically pull the plugs on the redundant power supplies, and then re-insert the plug, before moving on to the next supply. Lather, rinse, repeat, first with the supplies plugged into "power A", and then the supplies plugged into "Power B". A Sun workstation logged the relevant voltages & currents, to be printed out as part of the complete verification package for each machine. I got to the end of the long line of plugs, absentmindedly noted that this plug was different from the rest & unplugged it ... only to take down the Sun box, and completely trash the drive holding the data. It's the only time I ever lost data on a CDC Wren SCSI drive ... and it invalidated the ten day burn-in results for about two million dollars worth of "must ship" gear at quarter end.

I shouldered the blame, as I had pointed out to my Boss that having the Sun plugged into the test power bus was probably not a very good idea. All I could do is claim exhaustion, working a couple weeks of fourteen hour days because Sales had over-sold production capability and for some reason TheBoard decided we had to make the projected sales figures. Fortunately, my Boss managed to cover my ass & I kept the job. Daft thing is that I wasn't even part of the QA group that managed the burn-in, I was only roped in to help because of a lack of hands ...