Two clichés, one headline: 'No good deed goes unpunished' and 'It's always DNS'
- Reference: 1610957707
- News link: https://www.theregister.co.uk/2021/01/18/who_me/
- Source link:
Our tale takes us back to 2016, when "Sam" was working for an IT service provider. Sam's services had been rented out to a company that had just been gobbled up by a much larger entity. "There was a rebranding exercise under way," he told us, "and one of the changes was to switch a few thousand users to the parent company's email domain."
[2]
All straightforward enough, although the specific project Sam was working on was not directly related to email when the inevitable cock-up occurred.
Unsurprisingly, all the users at the newly acquired subsidiary lost email access. The problem was odd – it cleared up overnight, but things fell over again when the users turned up. The new parent company insisted the problem could not have been at their end and no other subsidiaries were affected so it must be Sam's site.
[3]
Pop quiz: You've got a roomful of electrical equipment. How do you put out a fire? [4]READ MORE
It was also very much not Sam's problem.
However, as the outage ground into its third day, curiosity got the better of him and Sam did a little investigating. It transpired that all the users at the subsidiary were connecting to the parent's Exchange servers via the internet. Microsoft's venerable Forefront Threat Management Gateway (TMG) took care of firewall duties.
Sam had experience of the now-discontinued security product. It featured Flood Mitigation, and there was a very good chance that the sudden wave of traffic from the subsidiary could easily be mistaken as a denial-of-service attack.
Going beyond the call of duty, he agreed to explain his theory to the company's outsourced team. "The call," he told us, "was laboured, there had obviously been a lot of tension prior to this, but it was over in 10 minutes."
The Exchange team took Sam's suggestion about as well as a bucket of cold vomit. The setting, they insisted, was not even turned on. Although confident it was, Sam backed away from demanding proof. It wasn't his problem anyway.
And yet still the issue niggled away, doubtless magnified by the wailing of users around him.
Another idea came to Sam. He had changed an application to use the parent company's email server, but received an internal IP for it that used a recently set-up network tunnel. If he tweaked the hosts, Outlook went there for mail, perhaps it might work?
It did indeed. Outlook was roused from its sulk and performed perfectly. Internal system spoke unto internal system and all was splendid.
Sam presented the solution to the beleaguered service management team. A local DNS mod would make all the problems (at least those related to email) go away. It was, after all, how internal corporate email was usually set up anyhow.
Due process was followed, the parent company was informed, and Sam logged what he planned to do and his contact details in the requisite change management form. The change was to take place at 3pm.
Doubtless weary from batterings administered by angry users, the manager of the local team approached Sam shortly before 3 and asked him to action the change: "He said they had taken enough heat on this, there was really no need to wait until exactly 3pm.
"So I went ahead a few minutes early."
Email began to flow once more. For a few precious moments Sam basked in the glow of being the hero of the hour, day, perhaps year.
Then all hell broke loose.
First Sam got a call from the change manager at the parent company. Email had started working so the change wasn't needed.
"Er, yes, that's right," he replied, "because I fixed it, albeit a few minutes early..."
And that was when the shouting began.
"The next three days," he told us, "were the most stressful I've ever had."
First he was berated for making the change early. Then the bigwigs got involved and the aggression was ramped up a few more notches. Five years on, and "I still can't comprehend the anger," he told us. Was it mere embarrassment? A regional subsidiary taking things into their own hands where the head office had failed?
Whatever it was, the parent company wanted a head presented on a plate for the email fiasco and escalated things to Sam's employer (who were more concerned with keeping their client happy than the ins and outs of the situation).
It got to the point where accusations were flying that Sam had broken the email in the first place or that he had violated the integrity of DNS. Heck, he probably had no idea what he was even doing!
The parent company and the newly acquired subsidiary were also at loggerheads as the spat intensified. Shrieking from head office demanded that the change be reversed, while those at the other end of the chain feared what might be unleashed should email suddenly then stop.
As a mere contracted resource, Sam had a miserable time and began to dread the start of the working day. "Eventually," he told us, "I went to the guys at the service managers to tell them how much I regretted ever helping them and about the flak I was continuing to get."
Sam had looked for help when things first kicked off but had been to told ignore the fuss. The service manager clearly had not grasped how bad things were.
This time they did. It took 10 minutes for the local boss to turn up at Sam's desk and say: "You will not hear another word about this."
We assume bottoms were kicked rather than the email server reborked, but true to his word, all the nasty emails and angry meeting invites suddenly stopped.
"That was the end of it."
While the level of ineptitude on show might, admitted Sam, be hard to believe, "those who have worked in the cesspit of outsourced IT, I'm sure they'll get it!"
There is a saying: no good deed goes unpunished. It's almost as common as the one about DNS. But what would you have done in Sam's situation, or that of the team at the parent company?
[5]
Ever thrown a drowning user a life jacket, only to have it angrily thrown back? Tell your story in an email to [6]Who, Me? ®
Get our [7]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_datacentre/networks&sz=300x250&tile=2&c=2YAVqR1i@YkccBZB8wtV@MgAAAMc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_datacentre/networks&sz=300x250%7C300x252%7C300x600&tile=3&c=33YAVqR1i@YkccBZB8wtV@MgAAAMc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2021/01/11/who_me/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_datacentre/networks&sz=300x100%7C300x250%7C300x251&tile=4&c=44YAVqR1i@YkccBZB8wtV@MgAAAMc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] mailto:whome@theregister.com
[7] https://whitepapers.theregister.com/
Been there, done that
Mailbox Permissions Issue
Team mailbox configured with very eclectic permissions, e.g. Some folders open to all, some to 1 or 2 people etc., some allowed to edit, some not. Stupid but ours not to reason why.
We got a request in to update a couple of permissions for new staff, colleague used a tool we had to make the changes but made wrong change at top of mailbox and removed everyone's write permissions.
Grovel from colleague to PA to get an idea of permissions again and all quickly sorted. Screenshots taken for future reference and supplied to PA.
Next day permissions had gone screwy again but at various sub-folder levels. Wasn't any admin staff this time.
PA - You obviously didn't do it right, I'm fed up of you IT monkeys making my life so difficult
Us - Well we sent you screenshots of how it was all configured, here are new ones showing the changes somebody has made
PA - I know it was you lot, couldn't have been anyone on my team we're all perfect, just f* fix it before I report you all to CEO, etc. etc.
Us - Sure, here you go, with new screenshots to prove all good (+ mailbox auditing turned on in background without telling her)
PA - It's broken again, you lot must have done it, I've already lodged a complaint
Us - Really? Here's the audit trail for that mailbox showing that 'Joe Bloggs' who she had insisted had to be an owner of every folder despite asking us to make permissions changes, her manager if I recall correctly, changed XYZ at time ABC
PA - Not a word :-)
Life Lesson
"Never volunteer for anything" - the most valuable lesson we learn on the journey from PFY to greybeard.
Re: Life Lesson
I just said that to a trainee! Not 5 minutes ago!
Re: Life Lesson
Did you volunteer to do so? Or were you instructed?
Re: Life Lesson
Indeed never even take an interest, show curiosity or ask a question.
If you do then everyone else will metaphorically take a step back, and suddenly it becomes your problem and your responsibility. At least until it's fixed, at which point everyone else will of course take credit (or try to).
This time they did. It took 10 minutes for the local boss to turn up at Sam's desk and say: "You will not hear another word about this."
The local PHB sounds like a good person to have around, shame it took so long for them to get involved.
My Manager!
He blocks all that kind of crap from the team. He'll field the complaints, and discuss it with us, and take his findings back to the complainer, it really is nice! We need to fix the issue, not take the heat. Good Manager!
Re: My Manager!
Had a previous manager who was the exact opposite of that.
Wouldn't take any heat for team, would be on phone or at our desks hanging around and demanding instant solutions to anything that affected anybody senior even if it hadn't been logged as a priority.
Less patience than my kids and less happy to take responsibility than them.
Re: My Manager!
These good managers exist.
We had a problem where there was an urgent fix needed and we kept having our chain yanked by management, and by people wandering round offering to "help". About half of our time was being wasted answering calls and saying no thank you. Our boss had the knack of wandering by whenever the volume of noise/frustration from our office increased. He said he would handle it, and told management that all communications had to come through him. It was a good feeling telling a senior manager to go and talk to Pete, and hanging up.
Pete came round every hour and took us for a tea/coffee/cake/chocolate to a) get status, b) get us away from our desks to talk to each other.
We fixed the problem. Afterwards Pete said there was a huge customer demo needing the fix. Management didn't want to tell us about it in case it scared us.
Later we were given a "thank you lunch" to which the senior manager came, and paid for it.
Re: My Manager!
A now retired (thankfully) manager
Me: I need to concentrate on some project work
Boss: No problem
10 mins later: PING!! Email for a BAU job dropped into my queue... FFS!!
Re: My Manager!
Time to take a coffee break. And just make sure the project misses the dead line (with all interruptions documented).
Re: My Manager!
I had one like that, he wasn't particularly good at any form of IT/Design/etc, but deliberately only employed people who were. He said his job was to protect us from all the corporate crap and in return, he'd bask in the reflected glory of our (hoped for) successes. He was as good as his word.
Re: My Manager!
When I was bogged down with a particularly tough problem that I just couldn't suss my excellent boss stuck a note on my now closed door stating that I had the plague so zero visitors pls - we were a fun relaxed site with plenty of time for social chat & coffee etc so a note saying Keep Out was a massive blessing. He backed it up with more formal email to all involved requesting No Disturbances for the time being.
On top of this he also took all the flak that came from other sites until deciding who would deal with it. He retired a few years ago and I'm two jobs down the line now, but he was a top boss.
Change management matters
If the change was scheduled for 3pm then why did he make it early? Change management is there for a reason...
Re: Change management matters
He was ordered to, by his manager
Re: Change management matters
Because it was requested by somebody above him (and he didn't know any better at that time, I would at the very least have asked for written instructions as I have learned the lessons about CYA).
Re: Change management matters
"Change management is there for a reason"
In this particular case it looks like the reason was sheer officiousness.
I’d share my stories
But the therapy cost me too much money to relapse to those dark, dark places ...
Its all over bar the shouting
Shrieking from head office demanded that the change be reversed
Its a pity that didnt happen , then Sam could clarify his fix
"yes the email is not working again because you demanded that I remove my fix"
.
"those who have worked in the cesspit of outsourced IT, I'm sure they'll get it!"
I have , and those the levels of shouting , buck passing and ass covering occured often.
Is there anyone technical over there that you can talk to?
Way back when, I was asked to integrate with new customer. They had a SOAP API and we were .NET so pretty easy to get going. I had a test endpoint with fake data, all seemed good. Then I started experimenting and discovered their credential service wasn't over HTTPS. I got told that the mistake was mine, so I wrote a 5 line curl script that hit the HTTP endpoint to prove my point. Someone in the email chain forwarded that onto someone in senior management their side who understood enough tech to know it was bad.
Customer went quiet for 2 weeks, my tech contact their side wasn't replying to my emails. Got an invite to a call, me, my manager, a junior who was shadowing me and about 4 from the customer. The project lead their side tried to talk around the subject, putting back dates etc before someone senior to them butted in saying the architect their side had been fired and the team (a handful of tech) had walked. I remember my manager mouthing the F word and the junior went white. We were asked if there was anything we could do our side - you know - to secure /their/ service. My manager said something along the lines of "Is there anyone technical over there that you can talk to?"
The project got canned and they continued with an ungainly passing of Excel reports.
I am surprised Sam's contract wasn't terminated.