Pop quiz: The network team didn't make your change. The server is in a locked room. What do you do?
- Reference: 1643013006
- News link: https://www.theregister.co.uk/2022/01/24/who_me/
- Source link:
Our story comes from Dave and takes us back to the Australia of the 1990s. It was the era of [2]Paul Keating and John Howard and, significantly, a time of advancement in telecommunications technology.
Riding that wave was our reader, "Dave" (no, not his real name) who was working in software and infrastructure for a government agency. His team had developed an imaging system ("back when that was hard," he said modestly) that could display trademark registrations on the new-fangled Windows desktops that were popping up all over the place.
[3]
"The application worked a treat," explained Dave, "and saved everybody heaps of time in their day job, and successfully displaced some dedicated SUN workstations that were used to display the coveted trademarks database images."
[4]
[5]
"Instead of queuing, examiners could simply use their desktop PCs to do their day job. Unfortunately, it was a victim of its own popularity."
It was indeed. The problem was that the servers were in one office and the examiners in another, about 12km away. Connecting the two was a 2Mbps link, which was frankly not up to the job as more and more users piled on.
[6]
Ever keen to keep the users happy, Dave had a brainwave. Why not simply put the servers in the same building as where the bulk of the work was done? A simple job – lug the physical hardware to the new location, set up the IP addressing for its new home and – Hey presto! Speedy images and happy users.
Change requests were approved and, at 9pm on Sunday, Dave and his team yanked out the server. An hour later it was racked in its new location, communicating its contentment via the medium of flashing lights and spinning fans.
Except none of the workstations in the building could see it. It was connected. Lights were on. But the all-important images were not accessible.
[7]
"After a couple of hours of fault-finding, we discovered that the networks team had not yet implemented the IP address change that was needed to allow the server to reside at the new location on the network," explained Dave.
It was now 1am. In seven hours' time, 600 users were due to turn up and expect things to be working. What to do? An urgent call was placed to the networks team, but nobody answered: "There was no such thing as 24x7 support back then," said Dave.
So again, what to do?
Dave, who had probably watched way too many heist movies, had an idea. Sure, the team could simply revert the change. Or he could become Dave The Action Hero. The securely locked network equipment room was on the other side of the partition wall. The door wasn't an option, but he found that if he slid a ceiling tile out of the way he could clamber through the ceiling cavity and over the partition...
"The landing on the floor hurt a bit," he said, "but within a couple of minutes I had rebooted the FreeBSD server that ran DNS into single user mode, made the necessary modifications to the DNS files and restarted the server."
A swift jab of the big green "Exit" button and Dave was out. It was now 2am. Six hours until the users were due and everything was up and running.
He slept in that morning and arrived in the office at lunchtime "to be greeted by some *very* angry network team members.
"It turns out that the correct solution did not involve jumping the partition."
So the users were happy, one set of Dave's management was happy. And one set was livid. Dave shrugged, the application was now working beautifully. And he had learned an important lesson.
[8]Planning for power cuts? That's strictly for the birds
[9]Hacking the computer with wirewraps and soldering irons: Just fix the issues as they come up, right?
[10]A time when cabling was not so much 'structured' than 'survival of the fittest'
[11]I would drive 100 miles and I would drive 100 more just to be the man that drove 200 miles to... hit the enter key
"Now I do security consulting work," he told us, "It is amazing how the adventures of my youth helps me make sure that my clients are secure –something something Floor-to-Ceiling mesh securing data centres..."
"He he."
For some reason we can't shake the image of Bruce Willis clambering through a duct in Die Hard from our minds after reading about Dave's adventures. Although the line "You asked for miracles? I give you FreeBSD" doesn't have quite the same ring to it.
Tell us about the time you had to channel your inner Tom Cruise in the name of DNS via an email to [12]Who, Me? ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/Who%2C%20Me%3F/
[2] https://www.theguardian.com/australia-news/2016/jul/08/paul-keating-says-john-howard-should-hang-his-head-in-shame-over-iraq-war
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Ye6G1Dj9UF8wzLznChg4rgAAABU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ye6G1Dj9UF8wzLznChg4rgAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ye6G1Dj9UF8wzLznChg4rgAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ye6G1Dj9UF8wzLznChg4rgAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ye6G1Dj9UF8wzLznChg4rgAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/01/17/who_me/
[9] https://www.theregister.com/2021/08/23/who_me/
[10] https://www.theregister.com/2022/01/03/who_me/
[11] https://www.theregister.com/2021/09/20/who_me/
[12] mailto:whome@theregister.com
[13] https://whitepapers.theregister.com/
Under the floor
I was working with a customer for a week or two. While we were working some men were installing a partition to split the room into "machines" and "humans". We arrived on Monday to find the workmen had finished building the partition, but we could not get into the "machine" side because no one knew the combination to the secure door. Up came the floor tiles and someone (small) went under the door, refixed the floor tiles and let us all in.
For all of my time there, that door was locked open. I do not know if anyone knew the combination. I found out later that the workmen did not know the combination - they just installed the lock. At another customer I found that these locks had a default combination code - and most customers did not bother changing it!
Re: Under the floor
We arrived on Monday to find the workmen had finished building the partition, but we could not get into the "machine" side
You were making life hard for yourselves, using fdisk to make the partitions would have been much easier
Re: Under the floor
Many years back, I watched a team come in to install a firedoor on my corridor. Did a very nice job, from concrete floor right up to ceiling tiles. Once they had re-instated and left, I went to the house services manager to ask if it was supposed to be a fire door. I then pointed out that there was a 30cm void above the ceiling tiles running the entire length of the corridor. There seems to have been a flaw in the tendering process....... Cue the return of the workmen.
For those right-ponders, think of how the the (1987?) Kings Cross fire (specifically the smoke) spread up the escalators.
Been there, done that.
I've gone under raised floors, too. And once knocked a hole through the sheetrock adjacent to the door, reached through & opened the door. And several times I simply picked the lock.
A buddy took out a cold chisel and mini-sledge (lump hammer to you Brits) and physically removed a couple cinder blocks once. He was even nice enough to repair it later (his Dad was a bricklayer and he earned pocket money as a teenager, helping him on weekends).
One does what one must to keep the bits flowing ... although these days one would likely be thrown in jail for even daring to voice such options.
In University, we were notorious for filching snacks from the residence kitchen at night. They never did figure out we were just climbing through the ceiling and ignoring their ever-more-obstructive locks as they kept upgrading them to try to keep us out... :)
They even dusted the window sills with flour to "figure out which window we were breaking in through." *LOL*
I once stayed somewhere there were double doors into the storage area for biscuits etc. Said doors were locked, but neither was bolted to the ground/frame, so you could just push them inwards, and then 'relock' by lining the lock up with the recess as you closed them again.
They added expensive extra locks, but no bolts in the time I was there.
One man's ceiling is another man's floor
> within a couple of minutes I had rebooted the FreeBSD server that ran DNS into single user mode, made the necessary modifications to the DNS files and restarted the server
No doubt the half of management who were pleased had realised that the network team could be replaced by a lone individual with the right motivation. I assume the other half of the managment who were livid had just realised the same thing.
Out of date building plans
Once ended up about 25ft up on a step ladder hastily attached to a pallet on a smallish forklift to fix (well power cycle) a "vital" router connecting accounts to the rest of the lan to get payroll done.
The comms cab used to be easily accessible before some bright spark decided to optimise the warehouse and have a mezzanine floor removed but neglected to involve site management or ops..., afterwards a large enough step ladder was purchased to access the cabinet, and an edict was issued to always consult ops and site management before undertaking "minor" changes like removing a floor. Of course the cabinet couldnt be moved due to lack of budget. It finally was moved following the warehouse being shut down for a week while a wasp infestation was dealt with. How did we find the wasp problem?, i was up my nearly tall enough ladder (i could reach but couldnt see) trying to install an extra switch in the cabinet but the thing wouldnt go in so an extra shove was needed *crunch* bzzzzzzzzzzzz and a cloud of wasps spew out of the bloody thing, never descended a ladder so quick in my life
Re: Out of date building plans
Reminds me of one of my favourite movie lines: "I got tired of coming up with last minute desperate solutions to impossible problems created by other fscking people."
.
Bonus points for anyone who can remember the film/star without practicing their Google Fu!
Re: Out of date building plans
"Bonus points for anyone who can remember the film/star without practicing their Google Fu!"
I got the right film but wrong character :)
Re: Out of date building plans
It's one of the Margaret Rutherford Miss Marples, but I can't remember which one.
Re: Out of date building plans
I've done ~40ft up on a ladder from a farm forklift.
One of those jobs that looks dangerous, but was actually perfectly safe - the forklift provided a much more reliable base than a ladder could on the ground.
Locked in at night
I was doing some work late one evening for a bank in Newcastle, we finished about 10pm.
In the huge foyer the length of the building, the customer said as I was a visitor, I had to go out via the main entrance and they would go out via the staff only revolving doors that had to be used with a staff pass.
They duly left and I walked down to the main doors to find there was no-one there, main doors locked and the customer had legged it and I didn't have their mobile numbers to contact them to let me out.
Waited there for 20 minutes thinking I was going to be there all night locked in at bank.
Then I thought I heard music, thought I was hearing things it was so quiet, took me a few minutes to identify where it was coming from, had to go upstairs and pinpoint where it was coming from, found it finally when I found 2 guys replacing floor tiles at the other end. Told them what happened and asked if they had a pass that would get me out of the door.
Finally made it out after being stuck for 30 minutes and still no security showed up - even waved my arms in the air a bit in case there was security monitoring cameras!
The customer did laugh the following day and said that should have probably made sure I could get out before they left!
Fun times
Re: Locked in at night
Where I used to work, I was often the last person to leave the buildings and lock up. I had a policy of checking every building before locking it and setting the alarm (the buildings weren't large). Occasionally, you would find someone in deep-thought unaware that everyone else had left. No-one should be left alone in a building without at least knowing they are the last one there.
When the staff get older, there is an increasing chance that they may need attention. I'll declare a personal interest in this.
Re: Locked in at night
I've been stuck in the service stairwell in a bank's high rise after following a manager's explicit instructions. After hours, no-one on any floor, no way back out. In desperation, I descended 30-odd floors, no way out on the ground floor, eventually about three floors below ground level there was a loading dock, kettle, sofa, and night security sitting around having a cuppa, all within sight of the doors to the stairs. They jumped a mile when I knocked on the window, because no-one had told them we were in the building.
I think that one was the first major security audit I triggered in my career - I shouldn't even have been in the building without a proper pass that would have let me out of the stairwell, it was a serious problem.
Re: Locked in at night
I used to work in a university office in Edinburgh which was supposed to be locked up and alarm set at 9pm. However the security company who did the final check and set the alarm (a) often arrived anything up to an hour early and (b) rarely bothered to check the building as they were supposed to.
As a result I set off the alarm on several occasions while leaving. The first time I was good and rang a keyholder. Thereafter, the problem having been left unsolved, I just thought "Fuck it" and walked away. Not my problem.
Careful with that axe, Eugene!
I went out to a restaurant to reboot a router, only to find it was behind a thick wooden door that no one had the key to.
Not a big deal, the hinges were facing outwards.
On my way out the door to my truck for a hammer and punch, I mentioned the locked door to a bartender and told him I'd be back. I didn't end up needing them, because by the time I made it back the he had opened the door for me.
With a fire axe. And boy, did he look happy with himself.
Dave?!
Is your name not Bruce, then?
Obligatory XKCD
[1]Devotion to Duty
[1] https://xkcd.com/705/