One good deed leads to a storm in an Exchange Server
- Reference: 1625470205
- News link: https://www.theregister.co.uk/2021/07/05/who_me/
- Source link:
Our reader-supplied story concerns the reader’s colleague: a fellow Regomised as “Lucas”. Lucas and our reader worked for a mid-sized company in Belgium. It was large enough to have a few sysadmins, but not large enough for them to all be of a senior level. Lucas was a junior sysadmin: “always eager to help, but not experienced enough,” recalled our contributor.
“One day, email (Exchange) just stopped working.”
[3]
This was 15 years ago, before the days when everything was sent cloudwards. Exchange 2000 or 2003 was likely ruling the Belgian on-premises roost back then.
[4]
[5]
Naturally, one would suspect the junior as having had a hand in the failure, but Lucas had taken the day off. It therefore fell to the main system administrator to look into the problem.
Emails were being queued. Thousands and thousands of emails were bunging up Exchange’s orifices. It was, in fact, working after a fashion, but oh so slowly.
[6]
The senior sysadmin dug deeper and found, upon further investigation, the problem had been started by a single email, which had been flagged as suspicious.
Poor Lucas. Our reader explained: “Wanting to do more for the company and even be available on his day off, [Lucas] had set the spam/virus filter to send suspicious emails to his personal email address.”
He had created his very own mail storm. A suspicious email had arrived. The server had obediently forwarded it on to Lucas’s personal email address. This triggered another warning about a suspicious outgoing email. Which resulted in an email being sent to Lucas’s personal email address. Which triggered yet another warning.
[7]
And so it went on. And on.
The rules of the time were not smart enough to spot the loop the server was in, so the queue rapidly filled with electronic shrieks of alarm. Goodness knows what state Lucas’s inbox ended up in.
It took a short 30 minutes to clear the queue and Lucas was given a stern talking-to regarding the use of his personal email account.
“He was replaced about a year later.”
The question, dear reader, is who was really at fault here? Certainly, as certain UK government ministers have learned, personal email addresses do not really belong in the workplace, but is there an argument to be made that Lucas had actually helped highlight a flaw in the rules engine?
Rather than being replaced, should he have been thanked for his diligence and the stern talking-to given to whoever had devised the infinite loop scenario?
Or was he, in the great tradition of Who, Me? incidents, just a bit silly?
Use the comments below to spill your brains and send your own story of infinite loops and all too finite skills via email to [8]Who, Me?
Get our [9]Tech Resources
[1] https://www.bbc.com/news/uk-politics-57642791
[2] https://www.theregister.com/Tag/who-me
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YOLYP1zHEOu-ErunMzxPygAAAEc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YOLYP1zHEOu-ErunMzxPygAAAEc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YOLYP1zHEOu-ErunMzxPygAAAEc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YOLYP1zHEOu-ErunMzxPygAAAEc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YOLYP1zHEOu-ErunMzxPygAAAEc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] mailto:whome@theregister.com
[9] https://whitepapers.theregister.com/
Re: Probably a bit silly but....
Yeah, that'd improve his Outlook...
Re: Probably a bit silly but....
.. should have remembered ...
And that he was the fuse in the sysadmin circuit.
O.
who was really at fault here?
Lucas, for multiple reasons. A "potentially suspicious" email could still have contained business-related confidential or personal data, no emails should be automatically forwarded out of the company mail system. Also, if he's on a day off then he should have either arranged for his colleagues to cover him, of let them know he wasn't really off, anything else could lead to duplicated work (as it did).
In the early days, just about at the time Exchange was first being launched (and allegedly there were only a handful of people in the UK who knew how to configure it) we'd moved from having modems on individual PCs to an ISDN based proprietary box which one day suffered a similar snow storm. Tracking tools were limited but by a process of elimination I tracked it down to the engineering manager. He'd sent a perfectly valid email to a supplier rep which elicited an out of office message that required a read receipt !
As mentioned in the article things weren't sophisticated back then so the read receipt prompted another out of office message which also requested a read receipt...
I suspect the rep at the other end got a dressing down over it from his employer but it took a while to manage to break the chain at our end.
Re: I suspect the rep at the other end got a dressing down
Well, I wouldn't expect that as it shouldn't be the rep's business to know their mail server workings. Their sysadmins, OTOE....
Sort of, guilty as charged
Around the same time I managed to bring down two mail servers, I set an out of office response, sending server replied with a "this email address is not monitored" email, cue email loop. How come no-one spotted this possibility when creating automated OoO messages!? Surely the second thing you'd think of is "what if two people have OoO set"? Mailing lists were a thing back then
Re: Sort of, guilty as charged
Around the same time (or possibly a bit earlier), I built an out-of-office system based on procmail.
Loops were indeed known about and the advice was to include an X-loop header that can be filtered on.
Later supplanted by logging replied messages and only sending one reply per address per day.
where's the unsend button?
I worked for a senior manager. A project was in trouble, so the senior manager sent out an email to the managers clearly laying out the situation, and the action plan. 30 seconds later he realised that should have said "do not forward this email", so he rushed into my office and asked if I go to to IT support and kill it.
I rushed down two flights of stairs and into IT support.
"Too late they said" they could see it had already be forwarded to over 100 non managers.
I went back to the senior manager and said "sorry - too late". He said don't worry, that is good news. He had received some emails from the grunts ( sorry, the professionals) with words like "Thank you for a clear statement, it looks like someone is finally doing something about the problems". So overall he came out well.
memories
That story triggered a memory of a similar thing happening to me just over 10 years back.The customer used office and Exchange 2007. We were suddenly faced with a mail storm of our own, the logdisk of one database just kept flooding with streams and streams of logfiles, gigs at a time. Those disks weren't all that big, so it filled up to capacity in a matter of hours. We were frantically trying to figure out what was happening, even involved Microsoft to try and find the root cause (all the while manually deleting logfiles, which had been sanctioned as a one time workaround). After several hours of searching we finally got the root cause. A user was transfering to a new department and thought it was a good idea to zip up all his work and email it to himself as sort of an archive. His mailbox was about 1.2 GB when he performed the action, the file was 800 MB. In itself this would not be a problem, unfortunately we used Exchange caching and that's where everything fell apart. in Outlook 2007 an OST could not exceed 2 GB, it simply stopped working after that. The user with the 800 Mb attachment (they had no limit, customer enforced) sent his mailbox to 2.8 GB, filling up the OST to the 2 GB mark, crapping out and restarting the process, with the massive log flood as a result.
Good times!
easy
It's easy to screw up with an email system. Reply to all, emails storm, etc ...
Re: easy
I archived a couple fun ones for personal use, always funny to read back some of those old shitstorms ;)
We've grown with email
And we've learned lots of things through lots of errors and surprises, but I think the one bit of email functionality that has to have caused the most trouble is the out of office auto-reply.
It is rather useful, generally speaking, but it can still cause havoc even now. Thankfully, email servers have been taught to not auto-reply to auto-replies.
Lucas
Sorry my good man, private email box even back then was a no-no regardless of the intentions.
However getting another year also suggests this incident wasn't the reason for the departure either.
Fwd: Fwd: Fwd: Fwd: Fwd: Fwd: Fwd: Fwd: Fwd: Fwd: etc.
A colleague once triggered a mail flood between his mailbox and mine.
When me or the other senior tech on my team went on vacation we'd set up a regular OoO and, for a select number of addresses, a supplemental forwarding rule to the other senior to make sure requests from higher ups were properly followed up
Well, I went on holidays and set the rules, but my colleague had to take a day off and did the usual. Unfortunately the forwarding rules weren't set to just forward mail from those contacts, but also when they were CC'd... so when the first email form someone on the list was sent to his address (or mine, really don't remember who first triggered the glitch), mails started being forwarded from one account to the other until it froze the Exchange server)
I think he got a bit of an earful the next day, but nothing of consequence (except learning not to do it again)
Probably a bit silly but....
He should have remembered that no (intended) good deed goes unpunished!