Overload: A one-way ticket to a madman's situation
- Reference: 1592205307
- News link: https://www.theregister.co.uk/2020/06/15/who_me/
- Source link:
This week's confession comes from "Jon" and takes us back to around 1992, just as the Windows juggernaut of Microsoft was readying itself to crush IBM's OS/2 under screens of purest blue.
"Email was an interesting new 'thing' on the internet," Jon recalled. To stitch together the network that we all take for granted, many email hubs used good old-fashioned dial-up technology to call each other up and perform an email transfer.
Jon's Canada-based firm had purchased some software from an outfit in Boston, Massachusetts, to do the transfer. The service would be used to regularly collect and send internet email.
"Testing went remarkably well," he said modestly. An Intel 386/40-based machine running OS/2 was used for the production installation and again, everything went swimmingly.
For those of a certain age, the arrival of the Intel 80386 microprocessor series was somewhat of a watershed moment at the end of the last century. Mostly compatible with the 16-bit era, the 32-bit i386 architecture continues to linger, despite the best efforts of modern software manufacturers to kill it off. The DX variant was the height of luxury back in the day, particularly when compared to the poverty of the cut-down SX incarnation.
Delighted to see hardware, software and operating system working in perfect harmony, Jon fired up a test script. The correct diagnostic messages duly popped up and, flushed with success, he deleted them.
"The last step," he said, "was to program our new email hub to dial the master in Boston, at 5 in the morning on cheap telephone rates, in order to pick up the day's email."
Ooo, a mystery bit of script! Seems legit. Let's see what happens when we run it [2]READ MORE
It was shortly after 5am that Jon received a panicked phone call from the Boston-based supplier.
"Our mail hub had connected correctly," he told us, "with several hundred thousand diagnostic messages in the queue..."
That was more than enough to crash their hub, and the team at Boston would be ever so pleased if Jon could possibly fix the problem. Right now.
Realising his error, he dashed to the office and shut down the diagnostic script which he'd forgotten all about and was still cheerfully generating test messages, Sorcerer's Apprentice -style. Oops. The program was hurriedly shut down and the remainder of the queue wiped.
Steam ceased to be emitted by the Boston hub.
Being an honest fellow, Jon confessed his sin.
"To my surprise the response from both the business partners and our supplier was amazement – that OS/2 had managed several hundred thousand email messages without error."
Thanks to Jon's inadvertent load testing, it was confirmed that – yes – the company was ready for business in the email department.
Oh, and please – no further testing, OK?
Ever inadvertently set the network on fire (physically or digitally) but managed to pass it off as testing? You have? Now is the time to confess all and share what really happened with an email to [3]Who, Me? ®
Sponsored: [4]Running Your Modern .NET Application on Kubernetes
[1] https://www.theregister.com/Tag/who-me
[2] https://www.theregister.com/2020/06/08/who_me/
[3] https://www.theregister.co.uk/cdn-cgi/l/email-protection#92e5fafdfff7d2e6faf7e0f7f5fbe1e6f7e0bcf1fdbce7f9
[4] https://go.theregister.com/tl/1956/-8477/running-your-modern-net-application-on-kubernetes?td=wptl1956
I meant to do that!
Would have been a great response!
Re: I meant to do that!
Without notifying anybody, that would have been a career terminating move.
Re: I meant to do that!
You never made it into manglement did you?
Re: I meant to do that!
No further than getting some manglers fired, much to their surprise (but that was not so much manglement as -skillfully- applied politics and backstabbing).
Been There, Had That
In my previous job as an email administrator we had a request to add a server to our email relay for a new project that was under development that would generate and send email notifications to senior staff within our customers company. As part of the request process it was asked how many specific messages were likely to be generated per day by the new system, and the answer given to us was 'around 200'. The request was approved and on the agreed Friday I night added the server ip address to the message relay on our Ironport cluster, ready for the project manager and a couple of developers to do some testing on the Saturday.
As the on-call person that weekend I get a call around 9.00pm on Saturday from my manager saying a large number of our customers senior staff had received multiple copies of the same email throughout the day on their smartphones and can I 'find out what the hell is going on?'
I duly login and check the Ironport logs and find that the new server added to the relay the night before had sent out around 100,000 messages during the day so I immediately removed it from the relay, updated my incident ticket with the details and added that until the project manager could fully explain why this had happened we would not be relaying messages for the project any time soon.
On the Monday the project manager sheepishly admitted that they had done some testing on the Saturday that had completed correctly but after he had gone home the second developer stayed on run some more test scripts unauthorised and had left them running but hadn't checked them thoroughly first, and due to a bug they were churning out blocks of messages at regular intervals.
The developer got a bollocking for using unauthorised and untested scripts, the server was added back into the mail relay and shortly thereafter the project went live without a hitch.
Re: Been There, Had That
The developer got a bollocking for using unauthorised and untested scripts, the server was added back into the mail relay and shortly thereafter the project went live without a hitch.
I'd say those scripts were tested (and failed).
Re: Been There, Had That
Looking forward to reading that developer's story on Who Me ? shortly.
Please kill our machine, it can't be done... Challenge accepted
I'm known in my office for being a "bit" of a cowboy but I also tend to be the "panic manager" that gets called in when things go to hell. But thats another story.
Many years ago we had just gotten new development and production machines that had "GASP" 2!!! CPU's so there was no way the machine could be killed by a runaway process... I was asked to see if I could do it (on the development machine obviously). Easy peasy. Code a 10 million entry string array and write a bubble sort algorithm and put all that in an infinite loop. Within seconds it had 100% utilization of the one CPU and was chowing RAM and swap space like the cookie monster on speed. The DBA managed to kill the process and I was asked NEVER to throw that process in twice as it would totally disable the machine. Oddly enough I've never been asked to stress test anything again. Can't imagine why....
Re: and was chowing RAM and swap space like the cookie monster on speed.
I used to run simulations like that during my PhD. I had to diagonalize a 43x43x43x43 density matrix on a machine with not enough RAM. Mind you, if there had been more RAM, I'd just have upped the sizing :-)
I couldn't do this all the time, however - it tended to annoy the other users. Gave the hard drive a good workout, though. CPU usage percentages were down below 10% (I think I recall even 3% at some point).
Re: and was chowing RAM and swap space like the cookie monster on speed.
Eigen would be proud...
Testing
I blame any cockup I make as the system having failed under a test. It's even vaguely true ;-)
Not me...
And related here before. A fellow programmer was the proud recipient of the first VT1000 in the company. This was an X-Terminal (i.e. X-Windows terminal) at a time when the rest of us were on VT100 or VT220 terminals or DEC Rainbow PCs. It had a "huge" 17" display and, most importantly, it ran X over thin coax Ethernet.
Our programmer did some demonstrations for those of us not so lucky to have such an object of desire. He showed off xEyes, a pair of eyeballs in a window that followed the mouse pointer around the screen. Gales of laughter. Then I asked if it could instance more than one copy... He dutifully filled up the monitor with over 50 copies of xEyes, very carefully placing each one. The VT1000 was stuttering a bit by then. Then he quickly moved the mouse around the screen in random directions. The first couple of eyes kept up for a brief few frames...
Then the VT1000 stuttered to a halt.
Then the VAX on wich the xEyes were running stuttered to a halt.
Then the network collapsed.
Over a hundred eyes and very quick mouse movements were too much for the VAX and for coax Ethernet running at 10mbps.
Re: Not me...
Ah the fun of xeyes! My favourite was xroach with the --squish option!
Brings back memories
of my first PC. Quite a beast it was in its day, sporting an 80386DX with Cyrix 387 floating point coprocessor, a whole 4 MB of RAM (later upgraded to 8), a graphics card with a whole further MB of RAM) and an Adaptek SCSI Controller with 88 MB disk! Cost quite a fortune at the time. Sped up my development work no end, and Windows 3.1 and MS-Office worked quite happily in 4 MB. I don't think that would do for the latest incarnations, would it now.
Re: Brings back memories
Wow, 4MB on a 386 was quite a bit back then. And the name "Cyrix" takes one back to those olden days...
I did manage to upgrade my 486 to 20MB at one point, helped a lot with gaming. Quake (Team Fortress), Rage, ... (yeah, and the dynamic object oriented machine earlier on as well, DOOM for short...). Memory fades a bit as to when I upgraded to which machine later on.
Testament to the solidness of OS/2...
Heard that some folks preferred to use OS/2 when they have to collect massive amounts of data over serial links as Windows could not keep up with the demand.
...still trying to find the article where OS/2 on a single CPU outperformed NT on a quadprocessor setup. Ah, those were the days.
I remember having to support a video editing system that ran on WfW 3.11. The manufacturer recommended running OS2 with Wfw runnning in an emualtion window/mode as the performance was better!
That was a serious and unscheduled test gone horribly right.