in observability. The side cars are running full steam, the metrics are flowing. They all break at the same time in slightly different ways. Which is chicken and which is egg? SNOWbody knows...

I like that term... I'm always recording my MTTI - well, in some cases my personal MTTI is in the months to years, but that's mostly because of several stubborn outlying cases.

Like spending 18 months proving that an outside MSP were a shower - literally defrauding us with impossible product claims, breaking things that we hadn't touched in years and then blaming us for it, interfering with deep level networking that they didn't understand, even actively kicking one of my guys off a remote desktop session on a server and then *restoring an ancient VM checkpoint* without permission in the few minutes it took us to work out why we couldn't get back into that server and why things were going off. The server jumped back months in time because they didn't know the difference between "Delete Checkpoint" and "Apply Checkpoint". Again, I had to prove every little bit of that from logs to prove our innocence.

Despite full documentation and contact methods, they stomped over the IP of a critical iSCSI storage device on a protected VLAN in the middle of the working day... to install a test VM to try to prove that a product they'd sold us would work on Linux VMs (and we hadn't bought that software yet, and they were spinning up a bare Ubuntu machine to test it, so hardly critical functionality!). Again, no notification or change management, just our storage collapsing and nobody admitting fault until I provided logs showing exactly what happened. Oh, and they never got that product working on Linux VMs (P.S. Avoid "Datto" backups like the damn plague... such a bodge-job of a backup platform!). They did exactly the same a few weeks later for another device and knocked out the boss's main printer as it fought to keep hold of its static IP that was being stomped over, and we initially got the blame for that too.

It wasn't even like we were at opposition - we were the long-established team, we were NOT going to be made redundant or anything, the MSP were brought in because we were short-staffed but couldn't ever provide any on-site personnel and the job was very hands-on. We had full documentation and change management, and some outsiders were given remote access AND insisted on having all our docs (and even then rewriting them into *their* standard format for consistency with the rest of their customers, which took weeks!) and then completely ignored them all. They sold us products that even the manufacturers had warned them would not work the way they wanted, multiple times, (I know, because I bypassed the MSP and spoke to the company directly and they basically said "Oh... you're the end-customer, yes we specifically warned your MSP about this and they ignored us, please don't blame us! I'll give you that in writing if you need it."). and one of the devices they sold us literally sat unconfigured for a year because they were supposed to get it working. It was installed in our rack. Not cabled. Never turned on. (And we know because WE controlled access to the racks). Because it simply wasn't compatible.

They removed a perfectly functional and secure IPSec VPN and replaced it with an almost identical "zero config" VPN device at great expense. Except that they couldn't then configure it correctly, so access control and CCTV at our secondary site literally never worked properly while it was in place - because it was reliant on routing rules that they were too dumb to be able to implement. Additionally, the irony was that the devices they put in still had to sit behind the very routers that were ALREADY providing the original IPSec VPN. So, in fact, all we did was turn off a working VPN and then put two expensive boxes in addition to that existing equipment, configured in a way that couldn't work. That all got ripped out when they left and put back how it was.

Hell, they charged us extra to "pre-configure" a server, which turned out to mean "they put the default Windows ISO on it". They never actually LICENCED it or activated it, though. So when it fell over - and we were having nothing to do with that device - because of the activation timebomb, that server just went down. Again, in the middle of the working day. And they'd been the ones to move several critical services over to it without asking. Their response to that was pathetic, too, and still they managed to cling on. The next week, the on-board network cards on it failed and they were entirely useless and couldn't diagnose it, and refused to send anyone to site for at least 24 hours. Again, middle of the working day, critical services down. Again, someone hand-waved it away while screaming at US when it was nothing to do with us.

It took 18 months to get a huge body of evidence against them, far beyond what should have been necessary, dozens of incidents of downtime, enormous arguments and confrontations, even accusations of "obstruction" aimed at us (but pre-guarded against because I'm no fool, so that particular project was actually entirely hands-off on our end, with simple end-goals, which were verified by them to be their responsibility and a "simple job", and they were provided with everything necessary and given many days of supervised assistance where I had my team on-hand to help but recording everything they were asked to do in case of reprisals later). They couldn't get it working, went crying to senior management, and tried to blame us for failing to live up to their own promises.

It eventually took half a dozen IT experts, some independent but also including at least three who were personal friends of the big bosses, who all immediately agreed with us and not the MSP, before we ever got any motion on it. And then it took the CEO of the MSP literally YELLING at me down a Teams call... which the bosses were all able to view... to actually get rid of them.

I had warned my employer that MSPs and in-house teams DO NOT MIX, no matter what contractual boundaries you're supposed to have put in place, but especially if nobody is managing those boundaries.

And eventually that workplace were then led to discover that all the things *they'd* been told in secret meetings with that MSP were lies, and finally admitted so to us. To the point of us being instructed: "Well, if we got rid of them, could you get it working again with what you have?". We could. They did. So we did. In a fraction of the time and manpower that the MSP had already tried to do so. And the response from senior management when we presented the working system: "Okay, so they absolutely were lying to us all along then. They said it wasn't possible.". They had been contracted to basically do anything necessary if we were unavailable, they were supposed to be our path of escalation, substitute and "mentor" us on how to properly configure servers and services (haha!), so it was all entirely within their remit and they couldn't do the simplest of things.

I stuck around purely to prove my innocence (I do often treat that part as a game, I have to say, because I *know* I'm not screwing anyone over and that all my reasons that I initially give are exactly what we'll end up circling back to and coming to the same conclusion, it's just a case of how long that "MTTI" actually takes in most cases). Then once we were done and dusted and I'd proven my case, I took the first job offer that came my way. Incredibly, that workplace swore off any MSP use and hired only on-site staff again to replace me, which I wasn't expecting but at least was the right thing to do!

My MTTI took a bashing, but not because of me. There are still no incidents of "non-innocence" on my part to impact it, though, even after 20+ years in the job. So it'll return to average over time. Generally a few weeks, in fact.

Was surprised this article was neither about Trump who always seems to be claiming his innocence and is decidedly mean (or is that cheap in en_US?) nor even the meaness of "paedoguy" sledging Musk whose innocent required a dubious judicial decision.

Liz Holmes is likely looking at a MTTI of 11 years unless Trump pardons that dazed bunny during his next administration.

The whole ServiceNow pitch triggered my bullshit detector - I suspect it is a right load of codswallop design to relieve gullible organizations of even more of their hard earned. Who are you going to call when trying to shrink your MTTI when dealing with ServiceNow?

As far as I can see the whole problem is fundamentally structural. Once an organization devolves critical infrastructure to mercenaries it can only really expect to lie back and think of England.

When your dashboard reports a problem, how do you know whether the problem is with the monitored system, or with the dashboard system?

I've seen/had to use quite a few flakey dashboard and/or device-administration programs. Otherwise-working devices not showing up on the dashboard, or not responding to commands from the device-administration program were common failure modes. Yet I could print to those printers, or access the C$ admin share of those PCs.