Yeah, we'll just take that first network handshake. What could possibly go wrong?
- Reference: 1659949393
- News link: https://www.theregister.co.uk/2022/08/08/who_me/
- Source link:
Our story goes back to the mid 2000s, when "Bill" (not his name) was working in the information security department of a large retailer.
"One of my many responsibilities was to implement a vulnerability management system," he said, "Everything was going fine: all testing of Windows and Unix systems had been successful, and we had placed it into production. The last step was to roll the system out to our AS/400s, which ran the guts of the business, in lieu of mainframes."
[2]
Bill explained the system to the AS/400 manager. He boasted of its success on the other platforms. He trumpeted its safety features. Of course the green light was given ("this was before we had implemented much of a change management system," he added).
[3]
[4]
The only proviso was that the job be run outside of business hours. Not a problem; Bill did the necessary scheduling and left for home.
Overnight all hell broke loose.
[5]
Upon his arrival the following morning, Bill was pounced upon by a gang of managers. "What did you do?" they demanded.
He'd simply scheduled a run of the vulnerability management system. What of it?
"Apparently most of the AS/400s had hung during the evening's production run," said Bill, "Orders had not been processed; a full-scale outage was declared; and the AS/400 team along with most of the IT managers and their bosses were on a crisis call for hours."
[6]
Bill saw his job and perhaps his IT career pass before his eyes. He protested: "But [the AS/400 manager] agreed to the run. And why didn't you call me? I could have stopped everything in five minutes."
True. However, the manager hadn't expected Bill to hit all the production systems at once ("good point," he admitted, "in a bit of hubris I hadn't considered that").
The connection between Bill's system and the outage wasn't made until the run was completed and systems restarted.
Preparing his resignation, Bill mumbled something about the software being actually written to minimize system load and none of the Windows and Unix systems had been touched. His excuse struck a chord with the AS/400 manager, who did some more digging.
It transpired that a vendor had also deployed some new software on the AS/400s. The outage, it turned out, had been a hang triggered by a combination of that new code and Bill's system. Nothing should be that flaky, and so a call was scheduled with the developer to work out what had gone wrong.
"The call was instructive," understated Bill. The supposedly skilled developer had implemented their system to take the first network handshake as the amount of virtual memory to allocate.
Not… ideal. And a HELO message sent by the vulnerability management system on the appropriate port, when converted to a number, might demand terabytes. The AS/400 would then frantically try and allocate enough memory, stopping any useful work from happening in the process.
We imagine there was somewhat of an intake of breath before Bill asked how an inappropriate packet would be dealt with. "They (rather smugly, I thought) replied that we should not allow that."
Suppose there was an accident? They couldn't be bothered to deal with that either.
[7]Lapping the computer room in record time until the inevitable happens
[8]Pop quiz: The network team didn't make your change. The server is in a locked room. What do you do?
[9]An international incident or just some finger trouble at the console?
[10]When civilisation ends, a Xenix box will be running a long-forgotten job somewhere
We'd argue that Bill's vulnerability management system had inadvertently exposed a gaping hole in the system. His management, not impressed with either the skills or attitude of the vendor, agreed. A call to the senior partner of the vendor was placed and a new version of the software was swiftly hustled up.
"And a great deal of cautious testing later, we were scanning again (though we permanently avoided touching the IP port that the software ran on)."
And Bill kept his job.
The [11]Somebody Else's Problem Field was strong with this vendor. Ever accidentally stumbled over a flaw and assumed it must be your fault? Or did you leave a whoopsie in the code and thought "Nobody will ever come across this"? The kindly Register vultures [12]await your confession . ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/Who,%20Me?/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YvDewxDWbHgW5Czd9wDvlgAAAAQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YvDewxDWbHgW5Czd9wDvlgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YvDewxDWbHgW5Czd9wDvlgAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YvDewxDWbHgW5Czd9wDvlgAAAAQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YvDewxDWbHgW5Czd9wDvlgAAAAQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/08/01/who_me/
[8] https://www.theregister.com/2022/01/24/who_me/
[9] https://www.theregister.com/2022/05/09/who_me/
[10] https://www.theregister.com/2021/11/29/who_me/
[11] https://hitchhikers.fandom.com/wiki/Somebody_Else%27s_Problem_Field
[12] mailto:whome@theregister.com
[13] https://whitepapers.theregister.com/
Re: The guiding principle
> Send the customer the bill!
In this case they sent Bill and the AS/400s fell over
Re: The guiding principle
I got that lesson in sanitising my inputs very early on in my career.
Was working with Google's Search Console, getting a list of the top 10 search keywords used to find a client's website. Google provided it as a comma-delimited field in a JSON object. My code, confident that Google knew what it was doing, split the string on the comma and cycled through the array up to 10 times.
Except that one time when Google bloody sent through a keyword with a comma at the end of it , thus chucking an extra, empty element into the array that my software proceeded to choke on. You'd have thought that a company the size of Google would have thought to strip formatting characters out of the data being formatted, but no. Likewise, I was a dumb for not considering that I could get either crap or no data back and writing tolerant code for it.
Re: You'd have thought that a company the size of Google would have thought...
There's your mistake.
You'd have thought that a company the size of Microsoft would have thought...
You'd have thought that a company the size of IBM would have thought...
You'd have thought that a company the size of Oracle would have thought...
Nope. They're all just as human as everyone else, the idea that their code and system design must be of some mystical higher order than yours will eventually be disproved by something that makes your brain hurt with its awfulness.
...before we had implemented much of a change management system...
There, that's the problem. Although in all fairness, sometimes it's a bit tricky to reproduce a full production load in a test environment and some problems may remain concealed during test...
Why has this software got zero defects?
30 years ago I remember hearing about some software had to be installed - mandated by the country's government. It had close to 0 defects and very few fixes, and was held up as a good example of how software can be.
Until the government started doing audits to check it was installed, and insisted that the software be >activated< rather that just installed.
Activating the software killed performance, and the software had so many problems, the auditors agreed it could be turned off!
When the auditors checked that their own government's software was activated. they got told the same thing (no way are we running with this).
The guiding principle
> how an inappropriate packet would be dealt with. "They (rather smugly, I thought) replied that we should not allow that."
Which does seem to be how pretty much all code gets written. Whether in 1980 or 2020, very little has changed in the rush to get product out the door ASAP. Following the four stages of development:
1.) does it compile
2.) does it run (not core-dump)
3.) does it produce the expected results from the expected inputs
4.) does it consume less resources than what a medium-sized country might have available
After that, it's golden. Send the customer the bill!