I just love it when you can shut the trap of a stupid manager mouthing off like that. Good on Doug, and kudos for being prepared.

The rule I use is very simple: "If its not written down, it does not exist". Strict application has kept my tail-feathers out of the soup many times.

An ex-boss used to complain that I should use email less and 'just go and talk to people' as it is 'quicker'. Until someone tries to stitch you up!

ALWAYS have requests and decisions in writing :)

Hear, hear!

Even if I do speak to someone in person or over the phone for anything like this, I will always request a written follow-up. On the off-chance anyone complains, the standard answer is "It's procedure that we must have written approval for audit purposes".

If only some of my auditees demonstrated the same understanding...

Back in my 9-5 career, I very much enjoyed thwacking manglement over the head (individually or collectively) with large piles of wet-ink paper-trail.

Until I discovered that I could make a lot more money doing the exact same thing as a consultant, with absolutely no danger of getting fired.

It's amazing how heavy even a signed post-it can feel when wielded appropriately.

I've found myself wishing for a new boss a couple of times in my life because their pattern recognition is oddly acute when it suits them. They notice my habit of, when asked for an opinion that may save my bacon down the road, I offer to email it for ease of review. If this works out in my favour (but not theirs) a few too many times, some have sneakily offered to 'save me the trouble' of typing it out.

Tip: if someone is offering to 'save you the trouble' of putting things in writing by deflecting to a quick chat, a quick written summary ("in case you forgot to mention" something) can neuter their gambit.

quick chat - teams, skype,slack, etc................ and print

I've mostly been fortunate in having bosses who didn't argue on technical matters. Generally they've been pleased to have someone making recommendations/suggestions on technical improvements. The 'costs' always raised eyebrows but if it didn't I'd be worried. However, the 'summary' email after a meeting has always been used to record the agreed conclusions, whether they were agreed or not....

On the other hand, the 'difficult' boss, who had all the words but poor understanding, took to shouting, berating and ignoring experienced technical input. So he issued stupid stupid instructions to 'save' costs which always turned out to be more expensive when they had to be rectified as 'someone else got it wrong'.

I'm well out of that scenario (----> ). Not long later, and after a lot of other resignations, he also moved on as his cock-ups caught up with him and he had lost anyone else to blame. But he is one of those dangers that you need to know where it is located.

I had a... cow-orker.. who was proud of his ignorance in terms of data analysis.

Yup, this fell back on him.

We had told him so: two statistical physics PhDs, an MSc (statistics), an MSc (biostatistics) and an MSc (meteorology).

Boy, felt good... :D

(one of us told him by email: this is exactly what we discussed last year - and yes, all five of us do remember, and this is the fix we told you about, "in case you forgot")

(I admit he has great organisational skills, but other... shortcomings, thankfully he is now more in an admin role in a different part of the company)

Oh absolutely, and the contrast between them screaming and ranting, and you quietly and calmly producing the evidence doesn't go unnoticed.

I had a similar experience where I was blamed for seriously delaying a project. Fortunately I had had a bad feeling about the whole thing and was careful to document and save emails which I was able to produce to show I'd been chasing the third part for months and had fulfilled my side of things. Sad that you have to do this but it can save you a lot of grief.

Just thinking aloud .....

Is 10x home broadband the normal rate for "properly managed leased line" ?

and how much quicker would that be fixed ?

If it was the UK would we still be waiting for Openreach (if physical line fault) ?

Could this company have paid for 9 more phone line installations (as a one off ) and then had 10 separate consumer grade connections (from different providers) and 10x the speed and resiliance?

#devilsadvocaat

With consumer broadband, they'd likely then have just had ten lines out of action and ten routers needlessly consuming power in a cupboard somewhere.

In other parts of the world they don't have the infrastructure/service provider segmentation we have here. If ProviderA wants to provide service somewhere, they need to install the wires.

Mines tend to be somewhat remote, so I imagine the install fees would be quite high.

Openreach have different SLAs available to ISPs. Business grade connections are prioritised higher than residential. Leased lines are even higher. Yes, you'd still be "waiting for openreach" but that wait would be a lot shorter.

If it were 10 separate ISPs on the openreach infrastructure, there would be one cable fault taking out the whole bundle. That is not resilience.

Also, openreach aren't the only game in town; they're just the most well known and most widely deployed. If you're prepared to pay, there are lots of providers who have their own infrastructure that they can roll out to you. The fee may well be in the £millions though.

From our own experience you get a leased gigabit fibre line from £600-750 + VAT per month depending on where in Scotland. So about 10-40 times the cost of domestic connections.

The difference is it is 1Gbit/sec, not "up to" and there are not "fair use" provisions. Basically if you get "gigabit" domestic fibre it is "up to 900Mbit" but typically shared with 31 others (maybe more) on a passively split/multiplexed GPON network, so your peak-time speeds could be as low as 28 Mbit. Unlikely to be that low, but that is worst-case.

https://en.wikipedia.org/wiki/GPON

If it was the UK would we still be waiting for Openreach (if physical line fault) ?

Could this company have paid for 9 more phone line installations (as a one off ) and then had 10 separate consumer grade connections (from different providers) and 10x the speed and resiliance?

I guess in this case they'd all go through the same exchange and/or roadside cabinet so would be vulnerable to the lot going down at once

OpenReach will fix a leased line fiber fault within 5 hours. Or at least that's the SLA, and they take it seriously.

And if there is an SLA, there are penalties attached. And yes, you can drag them into court...

Every friday I'm slightly astonished at reading On Call, with the bosses berating an employee in public out of the blue.

Nevermind that they're wrong, because this is On Call, but even when they are right - when has that attitude ever made a situation better?

Do they stop for even a second to ask themselves what the problem is with the employee and how to address it effectively, or do they just do what the lizard brain says?

Unfair to lizards which are fully functional living creatures adapted to their environment.

More like a pithed toad in my opinion. From memory the heart still beats but that is about it.

People say things

1) To be helpful

2) To be sociable

3) To let it out, and ease their frustration - also known as passing the monkey (https://www.pocketbook.co.uk/blog/2018/03/27/monkey-management/)

This prat of a director is undeservedly fortunate in not having employed the BOFH as his mindless rant would surely have resulted a richly deserved introduction into the not so gentle art of defenestration.

Never fails to amaze just how insanely mean (cheap) that otherwise extremely profitable enterprises can be when provisioning mission critical services.

Even junior miners are normally in the $billion league.

"Never fails to amaze just how insanely mean (cheap) that otherwise extremely profitable enterprises can be when provisioning mission critical services"

How else do you think they become "extremely profitable"?

Reminds me of a story of my Dad's...

He was setting up a call center so ordered two lines in with one being the backup in case anything happened to the first.

All went well until some digging work was happening in the car park. One scoop later, all comms went out in the building... both lines were cut in one go.

IIRC the result after a lot of shouting was fees refunded and a new line put in by the provider to the other side of the building at no extra cost.

I heard of a company in France had two offices one in the east, and one in the west.

They ordered two networks - one to the north, one to the south. They only touched at the offices.

The French telecoms company "optimised" its networks. It routed both networks onto the new central backbone.

Of course the central backbone failed - it cost the telecoms company a fortune.

Every minor scrap of a written record may turn out to be detrimental

There's a wonderful Austrian saying for you fitting the story.

However, in this case the protagonist was saved by the written record.

Schrift'l is the diminuitive of Schrift, meaning a diminuitive of a written record.

Gift'l is the diminuitive of Gift, which translates to poison.

I'm not proficient enough in English to make the rhyme work in English.

This saying is often used, in political commentary when some records turn up to haunt politicians over some scandal/issue.

On the other hand, if you have reason to believe that a written record might come in handy in the future, this saying also applies.

In the context of this story, the written record turned out to be a bane for the penny-pinching managers, since they weren't able to deflect blame.

I'd imagine [1]Ross_Ulbricht really regrets not switching his chat logs off that recorded everythng he ever did in his business and made for some rather damning evidence in court.

[1] https://en.wikipedia.org/wiki/Ross_Ulbricht

Cover Your Arse.

I was introduced to the CYA file at the start of my career and it has been of use as recently as yesterday. Today isn't over yet.

"Has a paper trail saved your bacon? If so click here to send On Call an email"

Man, I could write a twelve-volume compendium that would rival the thickness of The Art of Computer Programming.

You're gonna need a bigger mailbox.

How about the senior manager who refused to put down his acceptance of project risks for several months, until backed into a corner?

He now has the issue that the risks are 'accepted' with the caveat that they are solved before we go live.

Nothing has been done about these risks and he's hoping it'll be someone else's problem.

Anon as it's still unfolding, well exploding really

Many, many years ago I took a job in mortgage sales processing. It was a foot in the door of a decent company (the main reason to take it at all), but it came with some significant downside of having to answer the phones to mortgage brokers and the BS they spout while earning ludicrous commission.

The most common problem was applicants lying about their income to ridiculous levels. The paper trail accompanying would more often than fail to back up such claims. To say nothing of the many applicants that had suffered bankruptcies that were very much still visible on file.

Didn't matter, dickhead mortgage brokers would still call in and give the grunts on min wage grief over our refusals. Not my problem that you didn't do your due diligence on your client.

Of course, after telling such folks where to go; inevitably those same brokers would then resort to using one of the (should-have-been-criminal) self certification outfits and lie through their teeth to get their dodgy-as clients sale.

There are some people that still aren't clear why the sub-prime crisis became a thing, and why both the housing and debt markets are so hopelessly broken. All in the name of propping up a few rich bastards. Very few were ever taken to court of course.

This literally happened this week:

Me: Hey, you know that domain that you bought and host entirely with an outside company and that IT have nothing to do with? Well, just to let you know that I noticed that the SSL certificate is expiring soon.

Them: Oh, they [the web hosting company they hired] handle all that.

Me: Okay, just letting you know. The website will flag as insecure if it expires. It's best practice to renew way in advance, so it should have renewed already [and it's a LetsEncrypt, so there's no excuse as the software should start to renew after 60 days and the certs last 90 day, so they have 30 days to notice that the software hasn't done its thing]

Them: Okay, I'll let them know.

[2 days later]

Me [meeting them in person on an unrelated matter]: Cool, hey, don't forget about that certificate thing, it doesn't look like it's renewed yet.

Them: I'm sure they know what they're doing and they're on top of it.

Me: Okay, just don't want us to look bad because our clients see that.

Them: I'll let them know. I'm sure it'll be fine

Me: Okay, but I work in IT, used to build websites for a living, manage all our other websites, and normally it would have expected it to have happened by now.

Them: I'll let them deal with it.

[1 day later]

Me: Hey, just to let you know that that certificate expires mid-day tomorrow.

Them: [No response]

[1 day later]

Me: Hey, that certificate expires soon, still doesn't look like it's renewed.

[half-a-day later]

Me: As I feared, the website is now saying it's not secure in all browsers. Looks bad to our clients.

[a few hours later]

[Ticket comes in from another user involved with that site, saying it has errors. What are we going to do about it?]

[Redirected to the person above, with a note that there is NOTHING that *we* can do about it at all]

[next day]

Me: [mentions it casually to our boss].

Boss: Tell them again, but copy me in this time, we can't have that, it looks awful to our clients.

Me: [sends a polite nudge email, with CC:]

[a few hours later]

Nothing

[a few hours later]

Nothing

[two days later]

Still waiting.

[Insert Christmas holidays here]

I wouldn't mind, but I literally only noticed as I was splatting our sites through SSL Labs and thought I'd be nice and do the externally-hosted ones too, just to check. And all our internal-hosted sites are LetsEncrypt too, with A+ ratings.

So I know that LetsEncrypt should have renewed 30 days ago, so their cron job is likely dead or non-existent, that any hosting outfit should have turned on the "failure notification" emails for that, etc.

It's literally the only domain where even the nameservers are pointing at their DNS instead of our usual hosts, so I have absolutely no control over where it points, the content, the server, or any ability to modify or override it even temporarily.

Basically if your belt and braces appraoch fails, you at least have the shirt tails to cover your arse.

When Contracted to install Windows Server and an Exchange server at a V Remote site so the only comms was sattelit. They had an Inmarsat setup that provided pay per use, so pay per call, pay per KB etc. I strongly advised getting the other Inmarsat conection which was a higher monthly cost, plus capital cost of new equipment etc

It was declined.

I ran cost calculations stating that due the "chatiness" of Exchnage server to server comms, plus people sending attachments in emails etc, that a monthly bill would be approx 3 times what the install costs, plus annual running costs of the other system.

This was rejected by the companys Accounts department, and IT departments as unnecesary costs.

It was borught up in the project kick off meeting with MD and others present so was officially minuted again rejected.

So installed and quite running away as required.

First mothly sat bill comes in, I proceed to get ordered to customers office to explain the rediculously high sat bil, I arrives and sits in front of MD, and FD and hands over the miues, my proposal my costings and the rejections emails.

Within 2 days was offered another contract to move remote site over to the new monthly fixed costs sat system they were getting installed.

Having a paper trail has saved my backside more times than I care to count.

I'd say it's a necessity of modern life dealing with senior managers and, most definitely, C-Suite (those buggers get all-to-easily distracted with the latest 'money saving' fad).