'One rule for me, another for them' is all well and good until it sinks the entire company's ability to receive emails
- Reference: 1591946109
- News link: https://www.theregister.co.uk/2020/06/12/on_call/
- Source link:
Today's tale comes from an individual we're going to call "Al" and concerns that triumvirate of evil: a cluttered Windows desktop, Lotus Notes, and the Devil's Redirect.
Al was toiling away in desktop support back in the early part of this century. While he was a little reticent about his employer, he did tell us "this field was all about sales. Specifically, salespeople who had connections."
"We routinely paid salespeople we poached to sit on the sidelines for a year to run out the non-competes," he added, "because customers typically hired us last minute based on who took them to the most recent sports event or let them win the last golf outing."
Ah yes, non-competes – Google and Amazon [2]know all about those .
One particular member of the sales team, who we will call "Frank", was a regular pain in the posterior of the desktop support team, triggering groans and mutterings of "your turn" whenever the inevitable ticket turned up.
"For context," explained Al, "soon after Windows 95 came out, he called up because his laptop was slow.
"Upon arrival, I discovered that his system tray was TWO FULL ROWS, FULL SCREEN WIDTH of icons. All apps running in the background on what at the time was probably a machine sporting about 2GB of RAM."
Clearly a psychopath. We can only imagine what his desktop must have looked like.
"When asked which apps he really needed, the answer was, of course, 'All of them.'"
Our tale begins with Lotus Notes taking one of its occasional nosedives. "No one could send or receive email, which put the whole company dead in the water," recalled Al.
Al's phone trilled with news that the problem had been traced to his favourite user: Frank.
Obediently, he trudged over to Frank's desk to work out which of the multitude of "essential" apps was causing the borkage.
"Frank proudly informed me that he needed access to his email off hours," Al recalled, "and didn't want to be locked into the company email system (see: portability of contacts in case you get poached)."
Possessed of that most dangerous of things – a little bit of IT skill – Frank had hit upon a neat solution. A rule that would forward all his corporate emails to an account he'd set up with his ISP. Sorted!
Except Frank had forgotten about the limit on his ISP mailbox which, predictably, had been hit long ago.
For every message that was forwarded to the ISP, a "mailbox full" response would dutifully be sent. Which Frank's forwarding rule would then send it back to the ISP. Which would respond with a "mailbox full".
"You can see where this is going," sighed Al.
Fixing the problem was relatively simple. Frank's Rule Of Insanity was ditched and messages purged from the queue. Lotus Notes recovered, and the company was back to normal shortly after.
Al was the hero of the hour, or at least as much of a hero as one can be in a company that appeared to prize its precious salespeople above all else.
And Frank (the user, with the rule, in Lotus Notes)? "Frank was shortly poached by a competitor, whom I'm sure had no problems with him ever again...
"Or not."
What are the most apps you've seen crammed into the system tray after a user whined that things seemed a bit slow? Have you seen an ill-considered forwarding rule bring down an ill-prepared email server? Tell us all about that call with an email to [3]On Call . ®
Sponsored: [4]The Forrester Wave™: Streaming Analytics, Q3 2019
[1] https://www.theregister.com/Tag/on-call
[2] https://www.theregister.com/2020/06/10/aws_sues_marketing_honcho/
[3] https://www.theregister.co.uk/cdn-cgi/l/email-protection#d3bcbdb0b2bfbf93a7bbb6a1b6b4baa0a7b6a1fdb0bcfda6b8
[4] https://go.theregister.com/tl/1956/-8460/the-forrester-wave-streaming-analytics-q3-2019?td=wptl1956
Re: Out of Office...
At one company we used Novell Groupwise. This allowed you to delay sending emails until a particular time. Very handy for sending press releases when needed say after 12am. It was also great for slagging people off after you've left the company, making it look like you're working late etc. So anyway two staff members at this particular company were having a friendly bit of banter via email. Unbeknownst to each other they were both going on holiday at the same time. They had both had the clever idea of sending the other a mildly insulting message after they'd left for sunnier skies. There wasn't a send once set on the out of office of either of them. So every time an automatic response was sent it triggered another in the other mailbox
The mail server filled up fairly quicly as one of them had included a picture of a beach in their out of office. Alarms started going off on the Network manager's desktop as the server started to reach capacity. Then we started to lose the ability to send and receive emails before the mail accounts were identified and finally dealt with.
The culprits weren't named but it was interesting what he said in the autopsy of what had happened. In the five years since Groupwise was installed this was the first instance of that happening. When pressed further on that he said most people didn't know you could delay sending email. That lead a lot of staff to believe it was people on his team - I couldn't possibly comment.........although I know exactly who it was.
A Windows 95 PC with 2GB of RAM? Sorry, that's incorrect.
When Windows 95 came along, a decent PC would have in the region of 16MB to 32MB of RAM, and it wasn't uncommon to find lower-end systems sporting 8MB. In fact if I recall, Win95 had problems booting if you had more than 480MB of RAM in your system.
Came here to say just that!
Bytes? BYTES?
We had bits, and we felt lucky.
Our father woke us up at one o'clock, two hours before we went to sleep, murdered us in cold blood, made us install Computer Associates middleware before eating a breakfast of cold gravel, then sent us t' mine via Lotus Notes. In the hot inky blackness of the netherworld we'd wind individual bytes of memory with razorwire, whilst the Foreman read aloud from a selection of early PHP code. Then at ten o'clock they'd kill us by flooding the mine with SPX packets as the salespeople played illegal copies of DOOM. Our corpses would float up, where they'd revive us and make us fix a field full of HP Laserjets with "PC Load Letter" errors. At the end of week, we'd be expected to pay a shilling for privilege.
And you try telling that to the kids of today. They won't believe you!
Lotus Notes, eh? There was a website [1]dedicated to how bad it was (the archive.org copy sadly does not have the row of stock photos of business people facepalming across the top).
[1] https://web.archive.org/web/20120222220227/http://lotusnotessucks.4t.com/
Lotus Notes gave me a career for 15 years, so I'm not usually one to bad mouth it.
That website was out of date even when it was published, as it mostly deals with Lotus Notes 4 - 5 shipped in 1999.
The biggest problem Notes had was just that IBM underinvested in it massively. The backend was superb, and didn't need much more investment. The client needed some improvement, but only got it in fits and starts. Classic IBM management failure, really.
Its day is past now. You don't have to use the same app for mail, calendar, to-do and so forth. Notes as a platform was ultimately killed by what's also slowly killing off Outlook - the web browser. Outlook took some Notes mail seats temporarily, that's all. ;-)
(And we still don't have any great mail clients. My feature list for a mythical "perfect mail client" has features from about five different email clients, and would be hard to build. Especially as I'd like it to be cross platform on desktop, web, Android and iOS. Ultimately, we muddle through with what we've got.)
Yup have to agree, even most didn't apply to 5 by the time they got a UI that was usable it was bloated and slow, the Notes webmail client though was brilliant.
As out of offices were sent via an agent in the nsf (Every 15 minutes if I remember correctly) the impact would have taken longer to be felt as long as the Domino Server transaction logs didn't run out of space.
Even the scheduled agent OOO went away - I think in Release 7. There had always been a trigger for agents to run "on new mail", but the concern was that an email to everyone would overwhelm the agent manager queuing system, and not all of them would get processed. IBM did a little work so that the Router could run simple agents itself, rather than sending a trigger to the agent manager process.
That meant that OOO became effectively instant without large load on the server.
And the Notes OOO had, since version 4 at least, kept a list of who it had already replied to. By default, it was set to only send one reply during the entire duration. That really helped keep email storms at bay. Of course, there's always someone who'll turn that off, but Notes made it easy enough to spot and react to because you had decent logging and a server console you could look at...
Anyway, enough rambling. I'm showing my age. ;-)
I'm don't doubt that Domino and replication were the bee's nuts, but the client was all most people saw. It didn't need some improvement, it needed a lot of improvement. Server-side Lotus also died because the client was terrible to use for years.
Re: Perfect Mail Client
One of the best features in Lotus Notes, which has STILL not been ported into Outlook by Microshaft, is the ability to detach attachments and save them to separate storage, thereby maintaining the audit trail but saving space on the Domino/Exchange server. We have enough trouble with full mailboxes where I work as it is, current circumstances have just exacerbated the problem!!
Re: Perfect Mail Client
DAOS really is your friend there, only storing one (encrypted, unidentifiable from the OS) copy of identical attachments per server, so a, say 55GB mail file only takes about 9GB on disk, plus the space for attachments it shares with other mailboxes. Vastly superior to the old SCOS in use up to v7, which was a nightmare to maintain.
Lotus Notes, eh?
many companies still use Lotus Notes.
I was using it at IBM the other year, many councils still have it lurking somewhere.
The retailer i now work at still has lurking hidden in the background.
The best thing about lotus notes is that its built on a DB and effectively permits you to build apps on it, email, forms, processes, messaging etc all within notes. Its massive and bewildering but served/serves a function that nothing else seem to.
I'm quite glad i don't have to use or support it but appreciate what it did.
Luxury
We had to revive ourselves. Plus discover Quantum mechanics to re-introduce the universe from the vacuum left by clippy.
I spoke to my nephew who was attempting to do home videos during the lockdown. The aspiring Spielberg had called for advice on something technical. The phone was struggling to edit the footage doing any special effects. Turns out his phone is a very old one his mother had upgraded from. It had just 1GB of RAM and not much onboard storage although he did have a micro SD card to expand on this. I explained patiently that the computers when I started work didn't have 1GB as total storage let alone just RAM.
He didn't believe me at first and after I insisted just said "You must be really old then" instantly killing off his next birthday present. He redeemed himself and his future present when I asked how old he thought I was. Twenty five, when I'm nearly twice that made me smile.
And there I was, a couple of days ago: running into an 19yo who did not have any clue what to do with a word (.docx) file she received via e-mail. Apparently, with today's youth it's all "cloud documents".
I thought the approved document transfer protocol was to snap a picture of your computer screen with the phone, and then send that photo as a mail/message attachment...
I see! That's how it's done today. 'bout 15 years ago, some "boomers" attended an MS Excel training. They did get home work and exchanged it between students: printed the spreadsheet on paper and then faxed it each other.
The natural evolution of secretaries photocopying floppy disks...
I was in complete agreement with you until I went and checked [1]Wikipedia and found out that yes, Microsoft indicated a theoretical 2GB RAM limit.
Given the price of RAM at the time, I still highly doubt there ever was a machine with 2GB running Windows 95.
[1] https://en.wikipedia.org/wiki/Windows_95#System_requirements
Motherboard chipsets themselves often had a limit as well as to what they could support.
I still highly doubt there ever was a machine with 2GB running Windows 95.
i'm not sure 2GB of RAM would have fitted in a laptop, even with the size of the laptops back then.
2GB would have been a lot of RAM Modules, plus consumed a huge amount of power.
At Tip PC, re: 2Gb RAM.
I can confirm such a beast existed, I was running it.
I'd found a "memory tree" card that allowed you to add many more sticks of RAM than the motherboard had sockets, supposedly so the hobbiest could use a bunch of 1, 2, & 4 Mb sticks to max out their system. I'd filled mine with the max capacity available at the time (employee discounts FTW!), and used a utility to turn most of it into a RAM disk. Windows had more memory than it knew what to do with, the RD allowed me to save downloads faster than the HDD could maintain write speeds, & as long as I remembered to save from RD>>HDD before shutting down, I could zip along faster than my piddly little 'puter had any right to be.
Even better was using the RD to load programs into in which to run them. I quickly found out which ones could be safely run that way, which ones ran too fast when purely in memory, & which refused to run at all. Another DOS based RD utility allowed me to create one from a pure DOS prompt, thus allowing me to run my DOS games the same way. It was heaven...
Right up until the damned thing caught fire & turned itself into slag.
*Weeps rainbow arcs of sparkly tears at the memories*
I had 64MB of RAM in my Windows 95 PC -- at the time that was considered a lot of memory.
I remember that I had to take half of it out whenever I wanted to play GTA, because DOS4GW didn't like having as much physical RAM as virtual RAM. When I got in touch with someone at Rational they were all "oh; we never considered that someone might actually have 64MB of memory".
@JimboSmith - my kids (teenagers) initially didn't believe that when I was their age, the internet basically didn't exist (at least for Joe Public). Neither did mobile phones.
And then I pointed out that today I have more storage space on my person (phone and thumb drives on my keyring) than the entire department had when I was doing my first degree, and that their phones (not exactly cutting edge models) have more processing power than the Space Shuttles did, and their calculator probably has more than put man on the moon.
To put it in context, my PhD thesis (written in LaTeX) was entirely stored (including diagrams) on two floppy disks (and backed up on several more plus the tiny HDD of the machine), and written on a 486 PC with less memory than my current watch.
After a successful first year of business, in the late 1980's, we installed a HDD into one of our two PCs (compatibles). At 10 Mb and costing £1,000, we thought we would never ever fill it.
"my PhD thesis (written in LaTeX) was entirely stored (including diagrams) on two floppy disks"
@Custard [OT]: as a matter of interest, have you ever tried recompiling your thesis on a recent LaTeX version? I sometimes wonder about the longevity of digital formats. I'm guessing markup based systems are probably the safest bet
Coat: mine hosted, in the distant past, a Selectric maths golfball and a set of Rotring pens...
That was my first thought as well. 8MB - 32MB, yes, believable, 2GB? No way.
Memories
I bought my first PC in 1995: a Gateway P75 with 8MB of RAM and a 630 MB hard drive. Soon upgraded to 16MB and Windows 95 was much happier.
Re: Memories
A memory upgrade was usually the next purchase after spending £59 (1995 pounds, at that) on the upgrade CD-ROM. 4MB to 8MB led to usability, going to 16MB was expensive but felt smoother and racier than silk knickers.
On the other hand, there may have been some editorial confusion. An earlier paragraph said, 'Al was toiling away in desktop support back in the early part of this century,' at which time 2 GiB would have been rather more reasonable.
Good riddance
We had a developer who spent all of his time talking and doing not much work. Customer's loved him because he could talk the talk. The rest of use muttered in our beards and did the work he should have done. He was "poached" to work for one of our customers as the architect. It was a classic win-win situation.
A year later at a conference I was talking to someone from that customer and asking how the new architect was. The reply was "He was bloody useless, didnt know anything, and was moved to a sales position within 3 months".
Re: Good riddance
Know the skills of your staff and put them to good use. If someone is "useless" they probably are in the wrong position.
Re: Good riddance
Remembering that in some irredeemable cases, the right position is at worst not in your organisation, and at best with a competitor.
It reminds me of a comment I got when I was leading an install team once and was a man down. The leader of the larger team working at a nearby customer offered to lend me one of his engineers (who wasn't the most useful tool in the toolbox), and one of the other straight-talking Scottish engineers in his team came back with "don't do that, then he'll be two men down!"
Suffice it to say, we made do with who we had until my engineer returned.
Re: Good riddance
I once worked on an outsourced helpdesk, one of the guys there was always asking for assistance with the simplest of support calls, or giving incorrect advice. I won't give his full name for fear of a lawsuit, but he gained the nickname behind his back of 'Useless Eustace'.
A few years later I was contracting at another outsourcer, I happened to look at the user profiles on the PC I was using and spotted a familiar name. I turned to one of my colleagues and said 'Oh, I see you've had XXXXX working here', his response was 'You mean Useless Eustace...?'
Re: Good riddance
Had something similar, a User whose first name was not Wayne although that's what we called him when he wasn't around. He would take it on himself to move his PC half-way across the office (despite his boss telling him not to. Every. Bloody. Week.) then log a call complaining his mouse wasn't plugged in - of course not, he had deliberately left it on his proper desk. He would often move a networked printer (HP LaserJet 3) because he could not be bothered to get up and walk two desks there and back. He had lots of other tricks but if I give too many away, he might recognise himself. He might anyway.
His surname was Kerr...
The worst?
Two extended 1080p monitors full of desktop icons, including uncompressed video and images, running on a laptop with <512gb SSD, also had Creative Cloud and OneDrive in default user profile locations.
And you think your laptop is running a little slowly, sir? Ah.
Al
Was Chevy Chase involved?
Re: Al
I doubt he returned the calls...
Apps?
They were called "applications" in them days. Or "programs".
Been There...
We had almost exactly the same scenario years ago of two bouncing out-of-office messages, neither with "send once" set. It took a while to manage to stop it happening as the server was so overloaded. I can't recall now whether that was with Notes or with the predecessor system which was a proprietary email box connecting to the world via an ISDN line, I suspect it was the latter.
Seen something like this before
One of our consultants had been working at a major customer's site for several years, and had an e-mail account in the customer's system. Because he was rarely in the office, he set up a similar rule in Exchange to forward all of his internal corporate e-mails to his alternate e-mail address with the customer with a receive acknowledge enabled. Everything worked fine until the contract he was working on was completed (successfully I should add); the customer deleted his e-mail account at COB on the Friday.
Urgent call for me on Sunday afternoon from one of my company's directors - they need to get a significant proposal out to another customer PDQ, but the e-mail system had ground to a halt. A not-so-quick remote log on to the e-mail system later (for some reason, soon to be discovered, the corporate Internet connection was very slow as well) I had a look at the external e-mail server's logs (no, not exchange - the relay server was Postfix, something that I was really glad I had set up) and discovered that it had over 30 thousand e-mails queued for transmission - the two e-mail servers spent the entire weekend playing ping-pong with the messages until things had reach pretty catastrophic levels. More to the point the process was continuing even as I watched.
Emergency action was very quickly taken. The relay server was quickly shutdown (the metaphorical silence was deafening) and its outgoing queue was purged. Incoming messages for the guilty party where temporarily redirected to a bit-bucket (to allow the customer's e-mail server to calm down and purge its queue) and a message was sent to everyone in the company saying that any outgoing e-mails may have been lost and may therefore need to be resent. A more thorough report was sent to the director, who had a little "chat" with the guilty party Monday morning (while I was resetting things back to normal).
Out of Office...
There are several cases where auto-answer is not a good idea, but things improved a lot once all mail servers started to use a database back-end to store only ONE copy of an email.