Because the comments on this one should be more than worth the trouble to pop enough for everyone.

*Begins handing out shopping bags full of fresh, hot popcorn*

Grab a drink, take a bag, & enjoy! =-D

I last used it in the 90s...

Ho boy.

I can well believe it. About 5 years ago, I worked for a company who used an extinct, 80-char-limit terminal based programming language / database system as their main business logic. A system that was probably relevant about the time I was born, and obsolete before I was out of school./

Yeah but it worked, didn't it ?

Yes and no

It had become a behemoth to maintain. The business logic wasn't one coherent program, it was whole piles of scripts daisy-chained together, triggered by cron tasks. In the first two weeks that I was there, it fell over three times, provoking P1 incidents and forcing the company to fail over to paper-based processes (at 1/4 the speed). Changing any one script or program could have unintended consequences down the line, though they did at least have a test environment to iron out most of these kinds of issues before any changes hit production.

Getting it to integrate with any other system was an exercise in time and patience. When hiring, they couldn't find anyone who knew how this particular language worked, so they had to train every new hire from scratch.

When I left, they were starting to look at migrating to more mainstream and current systems.

The more I work with modern technology and the seeming clusterf*ck mishmash of disparate stuff mushed together to form a somewhat functional whole that sometimes stops working because apparently Mars is in retrograde and Phobos is transiting or something the more I think the old stuff might not be that bad. Sure it's sometimes a bit obscure but if it breaks you can probably actually trace exactly WHY it brakes because it's simple enough to do so. Instead of layer upon layer upon layer of crud, turds, brainfarts, incompetence and "good enoughs".

It's possible, where I work, we were just finishing transitioning from Netware to Windows about 20 years ago. Admittedly, by that time, it was a few legacy systems that were still running Netware.

I wouldn't really blame this on Ralph, it's not the kind of thing that can be anticipated or even tested for. Sometimes shit like that just happens, it doesn't make it your fault when it does.

Totally agree, however the lusers will still blame it on Ralph no matter how unfair it is. So goes the lot of IT manager.

“ So goes the lot of IT manager.”

Depends. In a sufficiently large organization, the IT “manager” will be one that leaves mails unread but will be quick to shift the blame to the humble sysadmin. I wear both hats, but still report to someone with a bigger hat. Who has limited skills as a sysadmin, but credit where it’s due : when all-nighters have to be pulled he does assist in whatever way he can (i.e. supplying food, and allowing the foot soldiers to expense a “job well done” dinner afterwards).

"I was," he said modestly, "god of my domain."

Those whom the (IT) gods wish to destroy, they first put them in charge of Netware.

So that's why they become insane.

Couple of years ago (more like 5) a vulnerability was found in the ASA code that meant it was possible to crack ikev1 vpns open.

I happened to be on call that weekend.

The security manager found a “fix” that someone had published on the internet would I install it.

I said I would provided the department director sent an email confirming the risks were understood - which he did.

The code change was applied to the firewall at which point my vpn to the office dropped, after 10 minutes it still wasn’t back so I made the trip to the office (fortunately only 3 miles away). Walked into the comms room and stuck a console lead into the back of the firewall to find if stuck in a boot loop the system would restart get to read the fix code and restart about 2 minutes later.

A few hours later and still waiting for tac to answer (it turned out the fix had been found by a load of people and it did the same to them), I started on extracting the config from the firewall.

This had to be done in blocks as although there was a backup it didn’t have the 60 or so vpn keys needed. And the thing wouldn’t stay up long enough to get all the keys out of the config before it crashed

A few hours later I had the config so I could trash the bad config file and then reload the 5000 line config through the serial port.

After 24 hours over 2 days we had a working firewall again and the biggest claim for call out payments in the departments history (24x £25) for one weekend.

Never trust fixes from random people on the internet

Indeed.

If it was a Cisco problem, the only valid patch I would install would have to come from Cisco.

Anything else and you're just asking for trouble.

If pressed, before applying the patch I would have searched for any problems with the patch (aka wait a day or two). In this case, the problem would have been largely reported and I would have gone, printout proof in hand, to explain why I wouldn't install said patch.

But hey, armchair general is easy, isn't it ?

This shows how old I am..

As a newbie support bod, I was enthusiastic. I had a degree and could do anything. Or so I thought.

Being the keen geek I was (and still am), I installed NT4 SP2 the second I could on my own work machine. I've always had two machines at work. One for serious day to day work, and a test machine I could afford to break if something went wrong. I installed SP2 on both, and used it quite heavily. After several days, I'd had no trouble, so when my boss asked if I thought it was reliable, without hesitating, I said yes.

What a naive fool I was.

We rolled out SP2 (which was actually quite an important upgrade IIRC) en masse. Within a day, we had users reporting their machines were blue screening. I did a quick survey of all the users (thankfully, we only had a couple of hundred PCs), and found just over half were blue screening.

We did resolve the problem, but IIRC, the resolution invloved me going round re-installing a *lot* of machines.

Now, I am still involved in testing, but everything is testing on multiple machines, virtual and real, and tested by multiple users before rolling it out to the estate. It's only rolled out when ALL testers are happy to sign off that it is good.

Oooh, even numbered service packs... I was in a Windows training course in the late 90s and the instructor pretty much said that even numbered service packs were of low quality. SP1 fixed the problems in initial release, SP2 was broken, SP3 fixed those problems before SP4 would break again. IIRC the same applied to Windows 2000.

I still remember that NT4 SP2 bug as I was just starting my life as an IT professional circa late 1996. Luckily my senior IT bod and I only installed SP2 on our devices initially and I think it bluescreened when you tried to access the floppy drive. (which was quite important back then)

Fortunately Microsoft quickly released a hotfix (which was the first one I can recall) but a good lesson on ensuring you always deploy to test devices first.

I do remember with NT just looking/changing the TCP/IP settings meant you had to reinstall the SP

Borkzilla has done its best to educate us to wait a while before installing a patch or a service pack.

I have well integrated the lesson.

"This shows how old I am.."

Remembering CP/M gets closer to showing how old you are. Remembering GEORGE shows how old you are.

One that always springs to mind for me was a patch for NT4 Terminal Server SP6.

It was a fix for one of the more rampant viruses back then - might have been I Love You but I don't really recall.

Anyway it came out in a hurry but had a list of known issues so I could at least forewarn our customers.

Almost all agreed that it was worth the possible downsides to be protected.

The initial patch went on without a reboot. I tested it. I rolled it out to a couple of clients. All good.

A newer patch came out that mitigated some of the risks/issues of the original. Without thinking I downloaded it and rolled it to a customer with the infamous words "and it doesn't even require a reboot".

At which point their servers rebooted... wait, what?

Turns out the updated patch DID reboot. A sign of things to come, eh?

And a lesson learned. Thankfully it was one of my kinder customers.

Software has way more "undocumented interactions" and "side-effects" than any mere pharmaceutical ever did!

My favorite from the land of the bizarre lives in several versions of Windows, from NT 3.5 onwards...

Put in a CD. Hit play.

Debug a C++ program with MSVC or equivalent. Stop at a breakpoint, and wait for the CD player to switch tracks...

BSOD!!!

Every. Freaking. Time.

Easy developer solution: buy portable music players for work. Hmm. Come to think of it, I could have expensed it... I was contracting. :)