I was told that despite being an intern, I should have known that the server named ‘Orion' had the label ‘NS1'," Wayne told Who, Me?

Sounds like they were a bunch of Wayne Carrs

Doesn't everyone have an on prem. secondary DNS server ?

And more than one forwarder ?

Learn pretty early on to never trust labels. Most systems you can blink LEDs from the OS or eject the CD caddy.

Hostnames (/bin/hostname) can be suspect too as having a server and its replacement concurrently sharing the same name isn't unknown - check the IP and OS/software versions.

Just unplugging the network cables from the doomed host, walking around office and returning half an hour later to deliver the coup de grâce will usually pick up cases of mistaken identity.

The inconvenient truth is that many sites, and particularly those where the developers rule the roost, have little idea of what runs where or indeed why.

All well and good, but if you were explicitly told which hardware to turn off you don't actually care what services it's running.

Also, KVM switches are notoriously mis-labelled as well. The chances of a server physically replacing a different server but retaining the labels on both the power cord and the KVM switch... pretty high

Correct. You're not dealing with chances here, but certainties :).

At a place I worked, every tech "knew" that host "steel" was our primary DNS server, and that host "sellwood" was our secondary.

However, we never had hardware interns, only software interns. Hardware techies were never allowed into server rooms or telecom closets until they had proved to the tech lead that said techie had mastered our institutional tech-knowledge, and had consistantly displayed a good, non -cowboy-like attitude.

During the good era there, managers were denied access to server rooms, etc.

During the less-good era there ... one day I found myself in a server room with five managers (Why me? My manager's whim) for a server shutdown/photo-op. The server in question was a DEC Alpha, the only one we had. Checking the back, I noticed someone had already removed the network patch cable.

With the five managers' and my index fingers on the power button, it was "Say 'cheese', one, two, three!"-photo-flash-power-button-*click*-polite-applause.

Nothing bad happened, but I still disliked the situation.

The inconvenient truth is that many sites, and particularly those where the developers rule the roost, have little idea of what runs where or indeed why.

DevOps is great in theory. The problem is most developers don't bother with the "Ops" part as it's not as sexy. So they spin up a load of services which slowly bitrot over time.

That was one of the fun things with Linux servers, even early - they didn't just have a command to open de CD drive tray, but also one to close it again so it was quite simple to set up a loop. This was before the days of virtualisation where you just had a herd of boxes to do the job.

Never had the chance to let this loose on all servers at once, I would have loved to see the effect. Missed chance..

:)

Assuming a decent choice of OS, it's a rare thing to be forced into performing a 'Redmond Reboot' on a server - but clear and accurate labels front AND back reduce the stress...

On the eve of Y2K I was told to "Shut everything down" so I did.

Ten minutes later the 'phone rang and a highly agitated voice asked me why the routers were not on line. I told them that they had been shut down as instructed. This did not go over well and I was told to restart them immediately as it was causing chaos on the network. I did so and thought no more about it.

The following day I was summoned to a telephone conference to explain why I had shut the routers down when I should not have. I explained that I had followed the instructions in the e-mail I had received, and kept, which it said to shut everything down, no exceptions. A lot of muttering and distant shouts followed then I was told that the instructions had been wrong and that I was in the clear.

A potentially nasty business and I was glad that I had hung on to that e-mail.

In this day and age, you should NEVER delete any email (that's not spam/junk/etc.). Archive it of course, but hold on to everything. Nothing will save your ass like written instructions...

If you are certain it's something that will come back to bite you in the ass (e.g. you're being asked to do something you know will cause a massive fail, or is potentially illegal/immoral), then also keep hard copies of the entire correspondence chain, especially where you asked for confirmation, and put down your reasons why you think this would be a bad idea to do. Keep it, and keep it safe.

Nothing will be able to protect you like plunking down a few pages of A4, showing the full timeline leading up to the cockup, and your repeated attempts to prevent in the first place...

I learnt the hard way that good server naming requires intelligence and not just a pattern. I learnt this when the only difference between the 'nix server I was meant to be reinstalling on was only 1 character different to the live production one - and that character was "c" on one/"e" on the other. When your main clue is the server name at the command prompt, well the human brain can easilly see what it expects to see rather than what's actually there...

A also learnt that day the joys of colour coding your Putty sessions, so anything done on the production server was done in bright red text/orange on DR/green on Dev!

At my last place the server names were made up of a mnemonic for the location and a three digit code (latterly 4 digits..). And they shuffled them every 12 months or so changing the numbers. It was hard enough to remember what to log into on a Monday morning much less where the services ran. The services were named after sea creatures (because they swam in the stream of data...). Halibut is down ... err, what does halibut do again?

Long time ago in another place the servers were named after animals. Elephant failing to mount mouse caused great hilarity.

Names are important for comprehension, as I regularly commented during code review when faced with stunningly uninformative variable names (tmp, temp2, myBanana, newBanana).

perform a hard shutdown on a server named "Orion" that could be found in a certain cabinet.

If thats what he did , he's 100% not at fault

As an aside the worst labelling I've ever seen was on a collection of about 50 backup tapes to be rotated around a bunch of servers daily , consisting of hand scrawled unitelligible scribble often in pencil .

runner up is rapidly fading pencil scribble in patch cabs / wall sockets

"fading pencil scribble in patch cabs / wall sockets"

I use a small zip-tie flags and Sharpies. Note that Sharpies come in many colo(u)rs.

Sharpies, ballpoint pens etc fade. Real pencils use graphic and it doesn't fade.

I've mentioned before about our engineer who was sent to a site on 'Bradford Road, Manchester'. It was long before mobile phones and satellite navigation. Unable to find the premises, he eventually phoned home to find out that he should have been sent to 'Manchester Road, Bradford'.

At least they didn't get sent to Southamptom instead of Northhampton.

When I first moved to London, I had a flatmate who worked for a firm specialised in setting up exhibits at various conferences and expos.

One day, he picked up a load for a conference in Frankfurt, punched the address in the SatNav, and took off. 14 hours later he arrived in Frankfurt and could find any exhibition centre at the address he'd been given. Stopping at a Petrol Station he asked the attendant for directions to the Exhibition Centre. Cue hysterical laughter, and eventually the attendant pointed and said 600km that way.

My mate had been directed to Frankfurt (Oder) on the border to Poland instead of the to the slightly larger and more central Frankfurt am Main... 6 hours frantic driving later, and a very last minute stand being put together and my mate finally managed to get some rest...

Lesson learned: Always double check the address...

Or Fort William instead of Port William.

I know someone who did that.

There's a Kristiansand in Norway and Sweden. Ask the Italian army.

The instructions to shut down could have applied to " any, any, any Orion "...!

[Sadly, only the more elderly among us are likely to recognise the allusion.]

As long as any old Orion was running on bare metal, then it's rather apt.

And yes, sadly I probably do count as elderly these days, or at least "experienced"...

Any old iron? Any old iron?

Any, any, any old iron?

You look neat. Talk about a treat!

You look so dapper from your napper to your feet.

Dressed in style, brand-new tile,

And your father's old green tie on.

But I wouldn't give you tuppence for your old watch and chain,

Old iron, old iron."