Sysadmin infected bank with 'alien virus' that sucked CPUs dry
- Reference: 1673852470
- News link: https://www.theregister.co.uk/2023/01/16/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Alf" who once worked on a team that tended Windows NT on thousands of high-powered workstations at a large investment bank.
Alf's time at the bank coincided with the launch of SETI@home – a desktop program that purported to analyze radio signals collected from outer space in case they contained sign of alien life. ET-hunters collect colossal quantities of data that might conceivably include signals from another species, and in the pre-cloud age boffins came up with the cunning idea of creating a program anyone could download so the analytics could be conducted by PCs around the world with their spare CPU cycles, instead of requiring boffins to buy their own large and expensive hardware.
[1]
Alf liked the idea of helping out, and it didn't hurt that SETI@Home ran leaderboards that recorded the world's top alien-radio analyzers.
[2]
[3]
In addition to the workstations he tended, Alf had access to a development lab packed full of powerful computers. He counted a dozen multi-core servers, a pair of top end multi-CPU workstations for each of the lab's engineers, plus "a room full of other miscellaneous hardware that needed to be supported or was undergoing evaluation for the bank."
"All this compute sat largely idle at night, so being the conscientious chap I am decided to donate all this unused compute to science and downloaded the SETI client and set it to work," Alf confessed.
[4]After 1.5 million days of computer time, SETI@home heads home to probe potential signs of alien civilizations
[5]After years of listening, we've heard not a single peep out of any aliens, say boffins. You think you can do better? OK, here's 1PB of signals
[6]Server broke because it was invisibly designed to break
[7]Don't lock the datacenter door, said the boss. The builders need access and what could possibly go wrong?
[8]Cleaner ignored 'do not use tap' sign, destroyed phone systems ... and the entire building
This went well and Alf saw his handle climbing the league table of SETI@Home contributors.
"Not being content with this, and using my skills as a Windows engineer, I created a version of the SETI client that could be executed over the network on multiple workstations simultaneously," he admitted. Before long he had pressed around twenty of the lab workstations and half a dozen servers into service searching for little green personages.
[9]
In the first week of that effort, Alf rocketed into the top 100 contributors globally.
It couldn't last, of course.
"One Monday morning about two weeks later I received a phone call from my boss asking if I could come in early as there were reports we had a massive virus attack underway and it was all hands to the deck for engineering," Alf recalled.
[10]
Alf arrived in time to hear his boss report that the bank's lab had been infected by a virus that was consuming 100 percent of CPU capacity and had spread to every workstation and server in the departmental lab. If it spread to production systems the bank would be in very, very, deep trouble.
Alf "put on my best innocent-but-concerned face" to – hopefully – mask the fact he felt fairly sure he was the cause of the problem.
Specifically, he wondered why the script he'd created to shut down SETI@Home hadn’t done its job, just as it had every day for the few weeks during which he'd run the software.
"As war rooms were being readied and conference bridges set up I had to act fast to avert a mass panic, so quickly executed the shutdown code on all the lab kit to return everything to normal and disabled it from running again, carefully covering my tracks by removing it from the network."
It worked, Within an hour the war room had stood down and the "virus" was dismissed as an anomaly.
"Given it was limited to our lab and didn't reoccur, nobody followed up with a more in-depth forensic analysis or examined proxy server logs – so I avoided discovery and potential embarrassment or worse, getting fired," Alf explained.
But he couldn't resist an investigation of his own, because he wanted to believe his script was sound. The truth must be out there.
"It turned some cabling work was done over the weekend, which had disconnected all of the lab equipment from the network, including the network share that hosted my script."
The code that shut down SETI@Home therefore never had the chance to execute.
Alf was left lamenting that he'd cracked the top 20 global contributors list, but found no trace of ET … and learned it was perhaps best not to use a program called SETI@Home somewhere other than, you know, home.
But he's still keen on distributed data-crunching – now focusing on efforts that focus on curing cancer and other diseases.
Have you been called out to fix a mess of your own making? If so [11]click here to send email to Who, Me? and we may feature your story in this slot on a future Monday. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y8UuVZoZdBHvCy1utPtkEgAAAEI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8UuVZoZdBHvCy1utPtkEgAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8UuVZoZdBHvCy1utPtkEgAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2020/03/04/seti_at_home_halt/
[5] https://www.theregister.com/2019/06/19/aliens_where_are_they/
[6] https://www.theregister.com/2022/12/16/on_call/
[7] https://www.theregister.com/2022/12/23/on_call/
[8] https://www.theregister.com/2023/01/06/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y8UuVZoZdBHvCy1utPtkEgAAAEI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y8UuVZoZdBHvCy1utPtkEgAAAEI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:whome@theregister
[12] https://whitepapers.theregister.com/
Re: NIce one!
Ha! I know of a friend who did something like that in our university's computer lab. Learned quite a bit about obfuscating processes, ressource management, etc. We also learned not to mess with the sysadmin.
Re: NIce one!
We also learned not to mess with the sysadmin
.. proven by the fact you're still around to talk about it. Never mess with a sysadmin.
:)
"perhaps best not to use a program called SETI@Home somewhere other than, you know, home"
I disagree, somewhat.
The mistake was in implementing the plan on his own. What he should have done would be to present the idea to his manager and get approval.
Of course, that meant his idea could also have been shot down, but hey, them's the breaks.
Disclaimer : I was a long-time contributor to SETI@Home myself, and ran it on every computer I had at home, plus my work laptop if I could.
Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"
I've always been of the belief that anyone who follows the creed "It's better to ask for forgiveness than to ask for permission", is not someone I want to be working with.
Something that might seem like an easy, no harm install, could easily cause massive problems that the installer knows nothing about - from a security point of view, from compliance, from legal or regulatory points of view. How often have we all had meetings where someone says, "Well what if we just install this to solve the problem..." followed by someone else shouting "Noooo...." and then explaining how that would cause much MUCH bigger problems.
Asking permission, might see your idea shot down, but asking for forgiveness, might see your career shot down...
Re: "perhaps best not to use a program called SETI@Home somewhere other than, you know, home"
Ha, I almost got sniped there.
Yes, all those points. I sometimes toy with the "better ask for forgiveness" idea, but I know where (and to whom) I can do this.
SETI go home
In a former job, one of my colleagues wanted to play that game of getting on the leaderboard so set up SETI@home on all of the office computers.
I left a hardware simulator running overnight. It was only supposed to take about 5 hours, so I could pick up the results and start work first thing the next day.
It eventually took about 17 hours, and so messed up a day's work plan. Not such an invisible cost.
I requested they be removed from every machine except at most, his.
Obviously this was in the days of a more liberal security & management policy.
See title.
No good deed goes unpunished.
I suspect the regomiser had a hand in picking ALF as a name. Somehow.
We've all done that......
... if we are a certain age.
I did exactly the same, we had a ton of rack servers that outside of biz hours did bugger all, but otherwise were SQL clusters, or webservers behind F5
These were obviously all set up and worked swimmingly until the clocks changed, and something went t*tsup and comms went down
Cue immediate need to investigate the issue.
Now I had a gut feeling what it was, it was, and removed trace of it.
Excuse given?
There was a known issue between Broadcom network adapters and HP Procurve switches when auto-negotiate was relied on, where comms would fail and infra refused to set the TX speed manually despite this.
I recommended they be set properly and the issue went away... Infra successfully blamed, my bullet dodged.
Still awaiting comms from E.T.
sitting idle means they are using less power
At least modern PCs / Servers when the CPU is idle they are using less power. Give them a heavy load then the power consumption goes up. So it's not actually "free" to give them something like seti@home to do. Also generates more heat therefore the cooling system has to work harder also drawing more power.
So in effect it's stealing to use the "idle" computers in this way.
We've had incidents where I work of people mining bitcoins using the organisation's electricity.
Re: sitting idle means they are using less power
The trick is to do it in winter.
Since heating electrically is 100% efficient (or near enough, I guess some might get wasted as noise or blinkenlight), it would be 'free'- in the sense that you spend as much electricity using a 1000W server rack as a heater as you would from a 1000W heater, and get the same thermal output.
I've heard of offices, back in the glory days of on-prem servers, heating swimming pools with 'waste' heat from the CPUs. It's only a waste if you don't find a use for it.
Re: sitting idle means they are using less power
But by that logic the heat exchangers used in HVAC systems are ~250% efficient. Therefore using computing to generate heat is less efficient and still costs the company money (albeit less than in summer).
Re: sitting idle means they are using less power
Good point, I'd forgotten about heatpumps somehow! They are indeed even more 'efficient' than using heating elements - able to move more heat for a given wattage than you could produce. That does rather put a dent in the scheme.
(for anyone worried about breaking the laws of thermodynamics - the trick is that you have to consider the whole system. Ultimately, the heat you get out of HVAC is produced by the sun outside your building, and then you move it indoors by trickery, rather than trying to make it indoors directly by heating a wire)
Re: sitting idle means they are using less power
I agree with finding a use for waste heat that you're going to generate anyway (swimming pool, etc.) but the problem with saying that during Winter a 1000W server rack costs the same as a 1000W heater is when you wouldn't use a 1000W electrical heater but the equivalent output from a gas central heating system.
But these days it would probably be cheaper to buy nice warm jumpers and not turn the heating up.
----------> Starting to miss the cold more and more!
EDIT: Beaten to it by the original and one and only AC!!!
So when the AC above goes AC do we get a recursion error?
NIce one!
Alf looking for aliens! If this hadn't been set in a bank, and a previous colleague was still alive, this could have been a network I worked on... Whaddya mean there was work done over the weekend?