For a while I would often be away for migrations to convert a local site to our standards (IP addressing. Email, naming etc). Mostly 2 or 3 of us for a weekend blitz. One of two on networking and myself on servers, desktops and mail migration

On one trip, we finished early-ish so my co worker decided to tidy up their cabling by pulling everything out and repatchong

I coulf see why, but he had the early flight home and I was there for any issues for a couple of days. My days were spent trying to find various ports which were buried under floors (non raised) or behind large cabinets

Local it knew what it was. The local users were just told it was teething problems and I was left cursing

Good days though

… goes unpunished.

I guess Spencer left his Marks on the datacentre

"third line support/infrastructure guy." That status may sound lowly

Does it? I thought that was premier league in the actual running a network part of "I T" , second only to 'architect' or 'director' etc.

Terminology varies I guess , in fact at my present job 3rd line rather unconventionally means the poor bastard who has to walk to the user if {whatever} cant be done remotely - so its unjam the printer or show them the on/off button .

Yup, 1st line = customer facing. 2nd line = escalations and implementations, 3rd line = projects and planning

Back in the day, 1st line support was the person who answered the help-line phone and read through choices on a menu. If they got to the end of the choices without resolving the problem they transferred you to a 2nd line support person - They had some knowledge of the system, and usually had access to the manuals and the "known problems" database. If they couldn't fix it, you were told that they were escalating your problem, and that somebody would phone you back. The "somebody" was the 3rd line support person - This could well be one of the people who had actually designed or written the system... In our small business I was often all of them. My calls ranged from "My computer thingy's disappeared" to obscure interaction problems (usually caused by Microsoft).

I've never had that kind of pressure but :

Every time I do a favour/job/diy for friends or family I find myself facing a severe lack of resources compared to my house and having to improvise, ranging from anything like wood saw , socket or voltmeter to I.T. bits like spare hard drive or monitor or vga cable.

More than once I've come across people who inexplicably dont have 100 spare "IEC Connector C13 6A" leads (aka kettle) lying around in a box.

Even real basics like "Have you got a bit of rope to secure this to that ? No ? an old extension cord or any elctrical cable will do .. NO?? are you wearing shoelaces then?"

Some peoples houses are just too damn uncluttered.

My problem is a tiny bit different.

When doing something for family, I try to prepare by taking things I might need. But then again, you could equip a small workshop with what I carry around in the back of my car!

But the problem is often "I know I have one of those, and I think I've seen it in the last few weeks, but I can't remember where it is!"

Whatever it is, I normally find it in the box of 'useful' bits that I put together for the last job, which I never found the time to put away again afterwards, and which may still be sitting in the back of the car anyway!

"Have you ever regretted taking on extra work?"

Yes. Every time I've completed the job in hand and then said to myself "Let me just....."

Or my pet-peeve of a line whenever you're in an office to fix a problem that has been logged:

"Oh, by the way - whilst you're here..."

And of course you're boned either way. Say "yes" and fix the extra issue and people just hoard problems for when you visit and it looks like you take an hour to fix a 15-minute problem. Say "No, please log a ticket first" and you get the inevitable feedback about poor IT support. You just can't win...

You have no idea. I have a health condition that keeps me away from the office unless I'm really needed, and I don't or can't trust someone else to take instruction to do it (IMHO, it's bad planning to allow one of your core infrastructure teams to drop to just one member, especially if they're ill).

So I often turn up for one of these essential pieces of work, which is often sized to fit in the available time (plus a bit of contingency), and find people are lining up to tell me of other woes they think I can fix "while I am there"!

The last trip, I ended up having to give our pre-prod environment an unexpected health check and fix, which almost compromised the main work I was there to sort out!

In our small business, it happened so frequently that we called it an ”OBTW" happening.

I'd just started a new job and was in a machine room looking at a server. There was another guy who was messing around with some fibre cables in another cabinet in an adjacent row. He was trying to trace one in particular and decided to give the cable a tug. All I heard was a load thump and a shout of pain as a small router got pulled off the top of the rack and hit the guy squarely on the head. Not all equipment in a rack is actually 'rack-mounted". He was very lucky that day!

Whenever I go to do a job in the comms room or similar, I usually end up carrying a box of various cables so that I can replace anything that looks dodgy or is about to be stretched to the point it can play music on it. Sometimes I don’t use anything but on rare occasions have had to go back to the supply cupboard. Fine when the site has supplies but when working remotely you almost take the entire stock room with you.

Especially tricky are when the old kit uses c13 but the new uses c15 I.e. high temp power leads… c15 power leads are some of the elusive kit that exists, mostly because someone uses them where a c13 should be so you can’t find any spares….

Whilst doing proving trials on a piece of kit at a power station, we decided to 'clean the area up' a bit. With a roaming vacuum cleaner, my aide-de-camp unknowingly knocked the air-supply to one of the machinery instruments.

Unexpectedly, rotating machinery started shutting down, other bits of the system stopped.... A long chain, all the way to the power-station big bits underwent a staged shut-down procedure. As our part was the first to react, a quick scan of the instrumentation revealed a rather low pressure-indication. The offending air-valve was spotted and normal service on our machinery resumed with positive results on the rest of the system.... blame, finger-pointing and most importantly, embarrassment avoided or at least minimised.

The inquest via the data-acquisition system implied an 'instrument fault'. The sensor was changed (unnecessarily) and I made sure the offending valve was locked in the on-position. That part of the process was then duplicated, with a further standby as it was clearly a single-point-of-failure; not good in continuous processes.

Good result ---->

I've learnt to walk around places with my hands in my pockets. Stops me touching things that I'd then be responsible for fixing. It also helps for when installations are CCTV'd- you with your hands in your pockets is less suspicious that the person who doesn't have them in!

Oh I don't know, someone else caught on CCTV with their hands in your pockets would I think look rather more suspicious...

As a Glaswegian, I appreciate the suggestion that one leaves Glasgow to go back to civilisation.

Unless that is the sole purpose of the maintenance you are undertaking, and it includes an outage window (which will be necessary when you need to unplug/replug things to de-knotify tangled cabling) and no don't trust redundancies like dual LAN / N+1 power supplies etc. to handle that even if they are properly tested at regular intervals because Murphy's Law and all.

There's no good that can come of it. Even if you do it successfully without causing any problems, there are exactly two possible outcomes. One, no one notices it so you get zero credit and anyone else working in that area will not appreciate it or try to maintain the organized state you created. Two, it is noticed and that means the finger of blame will be pointed squarely at you if anything you possibly might have touched goes bad - it'll be up to you to prove your innocence by finding a ticket someone else worked that meant they touched it after your tidying.

If the location was truly secure, as in machine guns and German Shepards, then its just across the loch from me.

And I'll overlook the jibe about the lack of civilisation in the Greater Glasgow area, aw'right pal?

Am I the only one that read the story as "the takeaway is that you must always have redundant data links for critical connections"? I'm sure that they wouldn't have had such a panic if the problem was that the bandwidth was cut in half (LACP or 802.3ad) or that the traffic simply moved to the redundant link (any other channel bonding mode) after a single fiber failed.

Redundant data links *that have been tested regularly and monitored as if they were live".

I have seen someone attempt a demonstration of how redundant their links were - up until the live link was taken down and the redundant link refused to work (it was up, but just wouldn't work). That took the entire production environment down. If I close my eyes, I can still see the look of terror on the poor guy's face (and he was a friend - I felt so sorry for him).

Or the other case I saw, where the main link went down, and we found the redundant link was also down - and had been for months. No-one noticed because the redundant link state was not being monitored.

1 - A five minute job never takes five minutes. More like five hours.

2 - Never start a job on a Friday afternoon. The job will be far worse than you could ever imagine and there will be no way to back out.

I know the feeling! A while back we had a fixed electrical installation test on a Saturday. It was mid-afternoon by the time they got round to the server room supply and I duly shut everything down and unplugged it from the wall (I've been caught before).After the testing was complete I fired everything up only to find two network switches were completely dead. I'd usually have a spare or two around but I think they'd gone to another site to fix a problem, it was too late to get replacements that day and Sunday wasn't looking hopeful either.

I could get the servers back online with the kit I had but it was looking likely the local staff would be without their PCs and phones until some time on the Monday morning. Cue some creative thinking, I was able to swap a small switch in a separate cabinet for an even smaller one and retrieve the 24-port switch I happened to have at home (it fell off the back of a previous employer's office, honest, Guv) so by lunchtime Sunday everything was connected up again. New switches were sourced on the Monday and another older one was replaced at the same time to provide a spare just in case.

I was working on implementing an imaging filtering algorithm by a competing research group to compare to our own new shiny algorithm. The authors of the competing algorithm weren't allowed to share their implementation due to IP issues, so in the interest of fairness, we decided to do our level best to squeeze every last drop of performance out of our implementation of the competitor's algorithm (as much as we did to our own algorithm) before comparing them. We had successfully reduced computing time by a factor of three on both methods, when I thought of another optimization for our competitor. It boiled down to reusing earlier results rather than recomputing them, at no extra memory cost. Easily done. It knocked another 7% off the wall-clock time. Not much, but nice to have, and it took just ten minutes to implement.

It worked like a charm on all images in our test set. Except for one, where it seemed to enter an infinite loop, never terminating until we killed it. I spent hours and hours trying to work out what was wrong, but ultimately had to give up. We left out this last optimization in the final test. The 7% didn't make much difference anyway, as our algorithm was far faster, and had a better worst-case computational complexity: O(N log N) vs O(N 2 log N).