Just when you thought it was safe to enjoy a beer: Beware the downloaded patch applied in haste
- Reference: 1615191312
- News link: https://www.theregister.co.uk/2021/03/08/who_me/
- Source link:
Our tale, from a reader the Regomiser has elected to dub "Simon", takes us back to the early part of this century and to an anonymous antipodean institution of learning.
[2]
Simon was working at the local Student Union (or "guild" as the locals called it), which was having problems with uppity education staff censoring the emissions of students. Simon was therefore commissioned to set up a fully independent newsletter.
"We had scored access to the Oracle user database," he said, "but only via the awful Filemaker Mac database. So I built a bridge to export it out to MySQL.
[3]
"So far so good."
His next task was to tweak Mailman (we're pretty sure he was talking about the [4]GNU Mailman mailing list manager, which was at that point in the throes of youth) to point at MySQL.
A hunt online turned up a patch, which he duly applied. A view was created to massage the data into something that Mailman would like, and Simon configured things for approved mail only (this was, after all, to be a newsletter rather than discussion list) and declared the independent newsletter ready. Or so he thought.
Simon told the guild secretary that the new toy was good to go: "all she needed to do was send a mail to that server, then go to the admin and approve the list."
"And then, being a late Friday afternoon, I went to the pub (fortunately just over the road from the guild)."
He was just about to embark on his second pint when a co-worker burst into the watering hole, blurting: "The server's gone crazy – everyone is receiving hundreds of out of office replies. EVERYONE!"
"I knew in an instant what was wrong and it was bad," understated Simon. Pint carefully put down, he sprinted to the office and, in his desperation to turn off the possessed machine, managed to knock the server off the desk in his frantic yank of the power cord.
"At least it was turned off."
It transpired that Simon's helpfully downloaded patch had done a bit more than just point Mailman at MySQL. "Turned out," he said, "that patch had completely obliterated both the permissions system AND the bounce detection code."
The result was that the newsletter had gone out as planned. However, it had hit 50 staff-member mailboxes with Out of Office replies set, which were sent back to the reply-to address. The hopelessly borked mailserver dutifully forwarded these on to every address in the list, thus triggering another round of Out of Office replies, and so it went on. Over and over. Right up until Simon physically pulled the power and accidentally sent the server crashing to the floor.
"I never drank so hard as I did that night," he said, "it was the worst day of my career."
Really the worst? We know of one person that accidentally spammed a company's entire internal and external mailing list with a message enquiring where the toilet paper was kept. He followed it up with a cheery "Sorry, found it now!" much to the delight of clients concerned for his digestive wellbeing.
[5]
You too can share your memories of email shenanigans or mailing list mayhem, with a confession to [6]Who, Me? - right here . ®
Get our [7]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YEYDzL7t5DoOw2nXTKs0TgAAAEE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YEYDzL7t5DoOw2nXTKs0TgAAAEE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] http://www.list.org/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YEYDzL7t5DoOw2nXTKs0TgAAAEE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] mailto:whome@theregister.com
[7] https://whitepapers.theregister.com/
ILoveYou...
why not just pull the Ethernet cable?
Funny how people like to reach for the power when pulling the comms would have been just as effective and arguably less destructive to the hardware.
Re: why not just pull the Ethernet cable?
When several servers share a common chassis the mapping from NIC to port is in software so you can't identify the cable to pull without checking the somewhat complex configuration or relying on your non-existent or out of date documentation.
Much easier to just hit the tit. That's if you know which physical server is the one to kill, and that's a whole new series of Who, Me?
Re: why not just pull the Ethernet cable?
Also for an Ethernet cable (at least one where the clip hasn't broken off), it's usually fiddly to reach round the back and into the rats nest of cabling, find the clip and press it then pull the cable. Also presuming that someone hasn't been diligent in their cable routing and tie-wrapped the thing to its brethren, making it doubly tricky
The power switch (either at the wall or on the case if applicable) is normally more accessible, albeit potentially less fast-acting (for the case switch) or more damaging (for the mains one) depending on what else the machine happened to be doing at the time.
Also can get compounded of course if there's redundant/secondary Ethernet connection as well.
Re: Funny how people like to reach for the power ...
... but if he'd tried to pull out the ethernet, would the server still have crashed to the floor? :-)
Not as bad as...
So I'm checking in to workplace in USA from some burg in France and wonder why the server's a bit 'odd'. Finally track it down to some idjit has sent a page message to a group email hookup a few dozen times. Only it's a high-level email group and multiplies out to 3000 page messages in total to president/vice-presidents/lesser-gods/etc.
And I'm checking in in the morning. That means it's 0x:xx o'clock in workplace timezone where pages are being spewed without end. Ho-ho-ho!
I smash the page queue and logout. Login later that night, and strangely no threatening emails. No mentions in passing either. Ever.
Either I was prescient and caught it *just* as the madness struck, or the external page gateway was borked long enough for discovery, or *somebody* really likes me.
I haven't seen a good game of Reply-to-All Tennis in years
Step 1 - Some poor fool incorrectly sends an email to a massive recipient list
Step 2 - Indignant recipient complains via reply-to-all.
Step 3 - More and more and more people Reply-To-All complaining about all the previous Reply-To-Alls.
Step 4 - At least one wag will send a spoof "complaint", taking the piss out of the complainers
Step 5 - Goto Step 3
Happy days. Did we all learn some manners, or what?
Re: I haven't seen a good game of Reply-to-All Tennis in years
Come visit our place (you know, virtually, not physically, dear
It's still a regular occurrence - sometimes accidental mail to all, "Please let me know if this particular set of circumstances applies to you currently" followed by 270ty-billion replies of "I'm not", "Not me", "Why have I received this?", "I don't need that"...
Re: I haven't seen a good game of Reply-to-All Tennis in years
I was working at a giant global company when that happened (2013 I think). The number of idiots doing a reply-all (to all 150,000 employees) saying "Please remove me from this list" or "Please don't reply all" was amazing.
It started at around 5pm UK time and took till 8.30pm before IT pulled the plug on the mail list.
Remember the NHS borkage five years ago when an email went out to 840,000 colleagues followed by inevitable reply-alls?
https://www.bbc.co.uk/news/technology-37979456
Career-limiting email
I remember someone who received an email entitled "xxx promoted to engineering manager" and forwarded the subject to a friend with a comment along the lines of "the peter principle in action".
Unfortunately in his haste to share the news he mistakenly pasted the subject line into the CC field, where the mailer found the word 'engineering' and obediently sent the insulting email to the entire engineering organization, some 5,000+ people (including the recent promotee).
He compounded his error by then sending an apology to the whole alias, drawing even more attention to it. A few months later he left to find other opportunities, presumably ones without email...