Competitive techies almost bring distributed disaster upon themselves – and they didn't even find any aliens
- Reference: 1599462911
- News link: https://www.theregister.co.uk/2020/09/07/who_me/
- Source link:
Our story comes from a reader Regomised as "Felix" and takes us back to the early part of this century, when [2]Nokia's 3310 was quite the thing and SETI@Home was the tool of choice for those wanting to help the search for extraterrestrial intelligence.
[3]Now no longer handing out tasks , the SETI@Home project was a large-scale example of distributed computing, analysing radio signals in search of the markers of intelligent life. It worked by breaking the work into chunks, which could then be processed by computers running a small client application to chew through the data.
During this time, Felix was working in a technical support role for a large telecoms vendor and was on a 12-hour shift rota where two "day" shifts were followed by two "night" shifts. While the site was occupied by thousands of workers during the day, at night Felix and his team of five or six had the place pretty much to themselves.
Being a major player in the business, the vendor had extensive labs, lavished with kit in order to model its customers' networks and handy for recreating problems. It was also handy as a playground for Felix and his 24-hour "Emergency Support" access card.
Around this time SETI teams were becoming a thing, and Felix told us: "We installed SETI on all lab machines possible, pointing to a SETI server back in our office that acted as a proxy and queuing system for the downloaded data chunks known as 'work-units'."
"Our SETI server," he confided, "was a completely illegal machine actually built from odd bits of dead machines and parts from the spares bin and it was lovingly known as 'RoboPC' – a splendid Siemens Nixdorf P200MMX with two LAN cards!"
The gang also popped the client on a variety of Unix machines. Alas, even with the content of the labs churning away, it still only amounted to a few dozen machines and some way from the greatest SETI teams. The quiet time of the long night shifts made for the devil's work and Felix and his pals set about expanding their SETI empire.
Thanks to a bit of nefarious password snafflery, they acquired some administrator credentials and worked through all the other office PCs (used by the day workers), managing to add another 50 computers, all Windows NT 4.0 devices, to the fleet.
With the aid of tools such as [4]Seti Hide concealing their activities from the user and running SETI as an innocuous-looking service, Felix's competitive instincts were at least partially sated.
"Our SETI stats had never looked so good," he boasted.
"Shame we hadn't noticed a small typo in the scheduling of our SETI drones, meaning they were actually running permanently, not just out-of-hours as we'd planned."
"Oops."
Fortunately for Felix, his shift patterns changed just as the wheeze was discovered. He pottered into the office on a normal day-shift morning to find the place in uproar. "Something like the Monty Python 'Spanish Inquisition' sketch," he recalled.
One of the more enthusiastic members of staff had noted that a suspiciously large number of PCs in the office were crawling along, running hot and showing 100 per cent CPU usage.
"Really helpfully," he growled, "they'd reported to IT."
"The PFYs were stumped, the problem came back a few minutes after the usual 'just turn it off and on again' reboot and they just couldn't work out why a 'system service' was gobbling CPU."
Sadly for Felix, the telco had recently acquired a networking outfit and deployed its analysis products onto the facility's infrastructure. Worse, the software was quite a bit better than that which had gone before and a network engineer soon spotted an awful lot of connections going to the gang's cut-and-shut RoboPC.
Making his excuses, Felix moved fast: "I got there just in time to power off the SETIserver and stop any further investigation of its possible physical location... phew!"
With the server off, the NT machines were unable to get any more work units "so the problem magically went away".
RoboPC was never found, and the next few night shifts were spent extracting SETI from all those office PCs.
Wary after their brush with a potential career-limiting SETI incident, Felix and co instead amused themselves with the likes of reverse SSH tunnels. "We also had a Dell Optiplex GX110 desktop hidden under a raised floor acting as a Wi-Fi AP, turned on at night, turned off during the day to avoid drawing attention to itself...
"Some nights we pointed a Pringles can antenna out of the window... hooked up to a company laptop that had been forced to boot RedHat Linux 7.3 with a 802.11 PCMCIA card with external antenna lead.
"Heady days indeed."
SETI@Home may be on hiatus, but there is always [5]Folding@home [6]to chow down on all those CPU cycles . Perhaps you've also repurposed the odd PC or two when you thought nobody was looking? Share your tale with an email to [7]Who, Me? ®
Get our [8]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://www.theregister.com/2020/09/01/nokia_3310_is_20/
[3] https://www.theregister.com/2020/03/04/seti_at_home_halt/
[4] http://www.setihide.de/
[5] https://stats.foldingathome.org/team/250966
[6] https://stats.foldingathome.org/team/69263
[7] mailto:whome@theregister.com
[8] https://whitepapers.theregister.com/
Once had a junior BOFH mess up his off-hours crontabs and flooded /var/tmp and /tmp with SETI log and results files.
Given that these servers were running the companies global DB2/SAP instance, it got immediate attention.........
I Suppose...
Running SETI out of hours isn't as bad as the overnight operator at one former employer who was running an entire computing bureau (back in the days when companies without their own systems would submit jobs for processing).
Re: I Suppose...
We had a bored night-shift op who loved MUD. He was playing over the modem and using PSS(?).
One night he came up with the "brilliant" idea of running a team through the dungeon. So he had a 4 character team running, each with a dedicated modem and using the PSS account to the Essex uni's machine...
And that for nearly all of his 12 hour shift!
The resulting phone bill and PSS invoice was "substantial", to say the least. Luckily his best mate was responsible for the modem pool and the PSS. He got an earfull from his mate, who then managed to distribute the high invoice evenly over the valid projects, so they were all a little larger that month.
Lesson learned, he didn't run a big MUD raid again, at least not as long as the company was using a modem pool for access.
Re: I Suppose...
using PSS
I suspect he was using PSS* to connect to JANET and then on to Essex Uni.
I did similar in the distant past before email became common (although it was on Prestel). Dial in to the VAX at work over a modem link, PSS to Janet, connect to the Northumbrian Universities network then log in to the MicroVAX sitting in the corner of a friend's lab so I could send him an email. Carrier pigeon may have been more efficient! :)
*Packet Switch Stream, we used it for connecting to various interesting places such as Harwell where we had time on their Cray.
Re: I Suppose...
Yes, I think you are correct.
Ah, reverse SSH tunnels.
Last "proper" job I had, they went to all sorts of lengths to lock the network down, filtering Facebook – even Gmail – and blocking USB sticks and the like (while idiotically still allowing Dropbox). They didn't seem to realise that if they still allowed outgoing SSH, all bets were off. It was part of my arriving-at-work routine to fire up the tunnel so I could actually check my email. A bit of tinkering meant I could even map my home network shares. Made life a lot easier when I wanted to take work home...
Re: Ah, reverse SSH tunnels.
That's just a normal forward tunnel.
At a previous job we had enjoyed a fast ssh connection in to the company in order to work from home, but after a while a pointy haired boss decided we should all use a new and incredibly slow VPN as it was "more secure". Suffice to after a week or so getting nothing done, word got around to have a careful read of the ssh man page.
So then I set up my work machine to automatically ssh in to my Raspberry Pi at home, with a reverse tunnel set up. Then when I was at home I could ssh in to the Raspberry Pi from the company laptop and connect back to my desktop at work.
RC5 cracking on the sly
Before SETI there was the distributed.net RC5 cracking challenge. I ran it on a few early Pentium III industrial PCsI had access to at work, but as there was only one PC with internet access in the office, I had to run the client and manually copy the data to and from the client for each machine. Later I managed managed to find a few more of the boxes which were going unused, and set them up at the back of a cupboard where hopefully no one would notice the noise from the fans and or wonder why it was a bit warm. Manually setting them up was a pain, so I created a shared drive and automatically ran the client on the internet PC out of work hours. The system worked very well for over a year, so well I forgot about it when I moved on, and it carried on submitting results for around 6 months after I left the company!
Not mine thankfully, but a friend of mine had a (legitimate) folding@home setup running on their test rigs overnight, when no overnight tests were scheduled.
This setup happily ran for a few years without issues and with approval from management. Then they decided to go cloudy for these test rigs and despite warnings from the test teams not to, IT copied the existing setups straight onto cloudy machines. And then set the limits for cloudy instances up appropriately generously so that the multi threaded memory heavy tests would run well.
And then they got hit by a stupidly high bill in the first month because of course the folding@home jobs had gotten copied across and were configured to use every bit of CPU and memory they could. This cost rather a lot more in the cloud than it had previously in power bills.
Pringles can?
Pah! We used a metal toilet brush holder. It worked really well.
(Don't worry, we bought it new).