Backup failed, but the boss didn't slam IT – because his son was to blame
- Reference: 1734075066
- News link: https://www.theregister.co.uk/2024/12/13/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Ray" who told us about his time working as the network admin for a small US-based pharmaceutical concern, at which he and a sysadmin had implemented an automated nightly backup over a high-speed fiber optic link to a remote location.
Ray is a sensible chap, so he also did two clever things. One was implementing a slower cable internet connection from a different ISP to handle the rest of the employer's traffic. The other was configuring the two links to fail over – so that if one died, the other would pick up the load.
[1]
Ray knew the cable connection lacked the bandwidth to handle the backups, but was confident data would be safe.
[2]
[3]
Sadly, his confidence was misplaced.
"All of this had been stable and operating well until first thing one morning when the sysadmin informed me that the backup had failed due to a connectivity problem and asked me to look into it," Ray told On Call.
[4]
After firing up his brain with a coffee, he found the router port used for the high-speed optical link had flickered on and off all night and remained unusable.
Ray called his ISP, which didn't answer. He escalated to his sales rep, and received "a terse response that they were aware of the issue and working on it."
[5]Tech support chap showed boss how to use a browser for a year – he still didn't get it
[6]Techie left 'For support, contact me' sign on a server. Twenty years later, someone did
[7]Tech support world record? 8.5 seconds from seeing to fixing
[8]Hide the keyboard – it's the only way to keep this software running
Everything was sorted out later in the day, and the sales rep was courteous enough to call Ray with an explanation.
It turned out that a teenage gamer in the same town as the pharma outfit was beefing with some rivals, who responded with a DDoS attack pointed at his IP address.
That DDoS knocked the ISP out for a few hours, taking the high-speed backup link with it.
[9]
When the boss came to ask why the backup had failed, Ray was able to explain that his two-connection rig at least meant the general user population had remained online, and then explained the cause of the outage.
At which point the boss's facial expression changed to one of fear and worry.
"What's wrong?" Ray asked?
"That was my son," the boss replied. "He is 14 and doesn't know when to keep his mouth shut."
And then the boss spun on his heels and walked away.
Has your office tech been taken out by teenage terrors? If so, [10]click here to send On Call an email and we may decide to share your story in the New Year. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z1wT1NJudNbAEDmQc2xvegAAAA8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1wT1NJudNbAEDmQc2xvegAAAA8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1wT1NJudNbAEDmQc2xvegAAAA8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z1wT1NJudNbAEDmQc2xvegAAAA8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/11/29/on_call/
[6] https://www.theregister.com/2024/11/22/on_call/
[7] https://www.theregister.com/2024/11/08/on_call/
[8] https://www.theregister.com/2024/11/01/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z1wT1NJudNbAEDmQc2xvegAAAA8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
> My veterinarian's child was responsible for taking down the local Mom and POP3 ISP
Fixed that for you...
That sounded like a decent internet setup for the time; the star of the story deserves a Rayse
Oh yes, the famous "additional router from the boss's son"
Happens quite often in smaller companies: Another internet connection is ordered, son sets up the router, connected to company LAN. Of course a home-router, which assumes it is DHCP ipv4/ipv6 master, has no rival anywhere, and any rival must be by design ignored and bombarded with "I am the only one here!" logic.
Result should be known to Reg On-Call / Who-Me readers. Including who has to sort out what is going on.
Re: Oh yes, the famous "additional router from the boss's son"
is that what happend ? its not quite how I interpreted the story but thats by the by.
I've certainly come across what you describe, I've found a router someone ahs brought in from home presumably to use an adapter to turn on cat5 socket into many. It was busy trying to allocate IP addresses
Backups...
My worst backup incident occurred when I was on holiday, and was thanks to the boss! I may have recounted this before on here.
Back in the days before cloud backups, when off-siting meant taking a set of tapes home with you, I was managing a 486 SCO Unix HP box with a DAT drive, running a number of terminals. Everything was fine when I went away for a week but, just in case, I took the "mobile" phone with me, this was a Motorola car phone attached to a large lead acid battery pack (see https://www.tvfilmprops.co.uk/det/4016/Motorola-4800x-Partner-Retro-Mobile-Phone/ for a picture). I'd left clear instructions on swapping the DAT tape each day but in the middle of the week I got a call, "The backup tape isn't ejecting, what should we do?" My clear instructions were to leave things as they were, don't do anything and I'd sort it when I got back a few days later.
The general manager wasn't happy with this so, without telling me, he called in an engineer from the parent company's support people who was only familiar with DOS PCs and clearly had never encountered Unix before. Apparently without even looking at anything he just powered down the box and proceeded to fix the tape drive, which he did successfully, before powering back on again. At that point he got a load of error messages and not much else. With no idea what to do he just left, basically saying it wasn't his problem as he only did hardware!
I returned the following week to find a server with a trashed file system and the most recent working backup was a week old. It took a few days to rebuild everything, restore the last available data and re-enter nearly a week's work. Did I get any thanks or an apology? Of course I didn't! A while later they relocated the operation to the parent company's offices and I turned down all offers to go there.
Ouch!
I have a feeling someone was grounded for a considerable time.
My veterinarian's child was responsible for taking down the local Mom and Pop ISP for a weekend back in ~2000.
The kid pulled a prank on one of his friends, convincing him that he'd gotten his hands on 'Chrono Cross' for the Dreamcast with a trick involving a VCR and some sleight of hand. He then gave his friend a FTP link to a ~600MB CD of /dev/random garbage, expecting him to spend a two days tying up the phone while downloading it and a week trying to get it to work and wasting many, many CD-R discs in the process.
The friend, however, went straight to IRC. "Someone got a hold of some Bleem! code! == Chrono Cross, fully running, saw it in action myself!"
That was at 5pm on Friday night. In under an hour the FTP server fell over and took the network gateway, running on the same machine, with it.
Remember how I said they were a Mom and Pop? Well, Mom and Pop were on vacation until Monday.