News: 1730698208

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Relocation is a complete success – right up until the last minute

(2024/11/04)


Who, Me? Welcome to another working week, loyal readers, and another dose of Who, Me? – the Reg 's weekly safe space in which readers submit stories of times when tech support went not quite so well as they might have hoped.

This week's hero we'll Regomize as "David" and his story has something in common with recent tales of finding out about the importance of redundancy.

You see David worked for a small but growing company. Small enough that all of its computer kit – web systems, database, backoffice, etc. – fit on a few racks in a server room. Growing fast enough that it was apparent this one room wasn't going to be big enough for long.

[1]

So the plan was hatched to move everything to a larger room in the same building. Step one was to build new racks in the larger room, wired up with cabling and patches for power and networking. Step two was to install redundant systems, where possible, in the new location.

[2]

[3]

Border Gateway Protocol was employed to make sure that customers would be routed to whichever location worked best for them, so there wasn't even a need for downtime while these redundant systems were moved.

[4]I made this network so resilient nothing could possibly go wro...

[5]Linux admin asked savvy scientist for IT help and the boffin blew it

[6]Compression? What's that? And why is the network congested and the PCs frozen?

[7]After we fix that, how about we also accidentally break something important?

Then it came time to move the one system for which there was no redundancy: the Oracle database on which the business relied. For that, a tight downtime window was scheduled – overnight – during which David and his team could physically move the machine from one room to the other.

The plan went like clockwork – the system was installed and powered up and tested with several minutes to go before the downtime window closed.

Congratulations all around, and all the exhausted techies breathed a sigh of relief – until, finally relaxing after a hard night's effort, David leaned back against the big power button on the rack. The rack with the database on it. The database of which there was only one. On which the entire enterprise depended.

[8]

Cue frantic activity as the seconds ticked down to the moment when the business would start actually losing money if all systems were not up and running.

David could not recall if it all actually came back online in time, or maybe a couple of minutes after. We can imagine time was a bit of a blur just then. What he did recall is that a plan was immediately drawn up to ensure that the database was also redundant, and that no single button could ever again take the entire company offline.

Have you ever found yourself as the dreaded "single point of failure?" Don't worry – we've all been there. [9]Tell us about it in an email to Who, Me? and we might share your tale with other readers for some future Monday morn. ®

Get our [10]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZyipWO8-7pcEO11KTVVf2wAAAIc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZyipWO8-7pcEO11KTVVf2wAAAIc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZyipWO8-7pcEO11KTVVf2wAAAIc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://www.theregister.com/2024/10/28/who_me/

[5] https://www.theregister.com/2024/10/21/who_me/

[6] https://www.theregister.com/2024/10/14/who_me/

[7] https://www.theregister.com/2024/10/07/who_me/

[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZyipWO8-7pcEO11KTVVf2wAAAIc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[9] mailto:whome@theregister.com

[10] https://whitepapers.theregister.com/



Single point of failure

Anonymous Anti-ANC South African Coward

These buggers are hidden in plain sight.

Grunchy

It seems to me Mr. Bean explored countless such scenarios.. Benny Hill suffered many similar mishaps but they were usually precipitated due to distraction by someone’s spectacular bosom!

Anonymous Coward

Yes, most of them went t..

No, wait, not going there. It's (a) Monday and (b) I haven't had my coffee yet.

Prst. V.Jeltz

Douglas had it sorted

"What happened?"

"A light lit up saying 'please do not press this button again' "

Chasxith

My favourite single point of failure so far had to be discovering that an entire office of 25 desks, PC's et al had been wired through a single domestic lightswitch by the entrance door. Walk in, flick the switch and everything sprang to life with a nice loud -crack- from the swtich.

Turned out the previous tenant of the space was rather lazy and/or ignorant of basic electrical practice.

Convincing the area manager to sign off the rewiring job was difficult until there were a few "full office shutdowns" in the middle of the working day due to someone flicking the switch by mistake.

John Riddoch

Was the flicking of the switch a mistake or a "mistake"?

Mostly, I'm surprised that running a ring mains for 25 PCs didn't blow the fuses for the lighting circuit. Certainly doesn't sound like it would pass any electrical inspection.

Will Godfrey

Depends on how old the setup is. If it's the older rewireable fuses all bets are off. I've seen a 15A fusewire in a 5A fuse carrier simply because it was the only fuse wire available and the people didn't know any better.

Sparkypatrick

I see your 15A fuse wire and raise you a nail (and sundry other conductors guaranteed to offer no overcurrent protection).

SVD_NL

Depends on the time period and location. Modern-ish PCs will tend to use around 50W, add about 25W for a monitor so 75W per setup, for 1875W in total.

In europe most circuits are 230V 16A, so the PCs would only use about half of what the circuit allows.

A quick google search shows even €3 light switches are rated for 10A (i rarely see stuff rated for less around here), so that would pass as well.

Is it a good idea? absolutely not. Would it pass inspection? Most likely.

I do hope they don't have too many outlets to attach space heaters or vacuum cleaners though...

Chasxith

Was a 1980s era office building with a "fuck it, that'll do" attitude to maintenance from the landlord. By the early 2010s when our firm were there, it had received rather a lot of that treatment.

Of course the switch-offs were...uh, mistakes.

ibmalone

Would it pass inspection? Most likely.

Genuine question, as I'm not an electrician, but would it really pass inspection? If the PCs ("et al"!) are plugged into wall sockets and not hard-wired in in some way then that 10A switch is controlling 13A sockets (in the UK at least), don't you have to take into account that something else might get plugged in? In any case my laptop on USB PD will happily pull 80W regularly and 100W at times (have measured with a metered cable), and that's only because it uses a lower energy mode when on USB, the normal power supply is 230W. Tower workstations can easily use more, and all computers will tend to pull maximum power during startup (except maybe the GPU isn't fully engaged).

In the UK number of sockets on a ring main is unlimited (!, but a ring is meant to be limited in the area it serves) and a ring is 32A. It looks like a smaller radial can be 20A, but even then don't you need an actual isolator rather than a light switch? OTOH, isolators do make a bigger snap when toggled than single pole switches, so maybe what was wired in was actually an isolator, in which case probably fine electrically and just an unwise choice practically.

Sparkypatrick

EU lighting circuits are typically 10A. UK may be 10A or commonly, 6A. You don't use a 16A circuit breaker for protection of a circuit with switches or other items rated at 10A for obvious reasons.

PCScreenOnly

I know that the main fuses take bigger load than rated, but 32 desks on 5a, or even if royally bodged onto a 33a is bit of a push

Doctor Syntax

"33a is bit of a push"

Nobody said the switch wasn't overloaded. Far from it. The "nice loud -crack- from the switch" tells the reality.

Will Godfrey

It's worse when they switch it off, immediately realise something is wrong and switch it back on again. If the loss of power didn't kill anything, the switch on surge would!

Dave K

That setup would have lasted even less time at a previous workplace I was at. This particular workplace had access-controlled doors for every floor/wing of the building. You know, the usual scan RFID card to open the door setup. Then when leaving, there was a button to press which opened the door, all standard stuff. Except that the door-release button looked remarkably like a light-switch, and was in fact right next to the actual light switch.

I've honestly lost count of how many times people tried to open the door and just switched the lights off instead because they hit the wrong button...

Anonymous Coward

I've seen emergency shutdown buttons mounted next to an maglocked exit door which looked remarkably similar, and yes, the obvious happened.

Apparently it took 3 full shutdowns before someone got permission to put a shield over the "do not touch" switch.

Imagine the feeling hitting the exit button and hearing all the fans spin down to the sort of deathly silence usually experienced in graveyards, and that in the days before widely available defib units..

blackcat

There is something quite otherworldly about a server room losing all power.

Many moons ago a fire suppression system test resulted in such a situation. The person testing it had removed the solenoid from the huge CO2 bottle so that would not fire, had switched the interlock with the main building fire alarm system so that would not trigger BUT had not turned the magic key for the UPS main contactor.

He triggered the alarm, it does everything as expected and suddenly we are all stood in the dark in absolute silence. And this was mid morning on a work day!

Crt

Gerhard den Hollander

At least you didn’t have those old crt monitors.

It was the first thing we did after a power failure. Run around and try to switch them all of before power came back on. Even a few of them could blow a single fuse with their startup surge. And too many of them could blow the main fuse even if the local fuses didn’t.

The other fun thing was to leave 2 identical monitors side by side to poweron at the same time and watch them trying to degauss each other.

Mind you these were 21’ sun or ibm monitors. The buggers officially required 2 people to handle them.

Re: Crt

Anonymous Coward

I kinda miss the solid degauss 'DUNK' when you powered them up, though, you knew the job was done right when you heard that :)

El blissett

Basically had this at my first (and last) custodian job looking after a UK school computer cluster in the 90s (ring networked). I had to come in in the morning, switch everything on, clean etc. Check on it during the day and switch off lights and a single power bank by the door at the end of the day - leaving the computers to their server commanded start up and shutdown jobs. They didn't tell me or someone they got in to do a machine-by-machine backup and update to do anything differently the day he was in. He'd moved the lighting controller to another socket to make room for the enormous plug for his backup trolley, so when I left for the night and did what I was told to do by switching that off, I wiped out minimum 6 hours work and possibly totalled a machine or two.

My contract wasn't renewed at the end of the year, luckily, I've never studied or worked anywhere since that had electrical and network setups that easy to destroy. The school itself was shut down a couple of years later - no idea what the fate of the pricey OS X kit they had just purchased along with a new on site BOFH. I hope they both found a new home.

Michael H.F. Wilkinson

I recall a visit to our HPC centre with students from all over Europe in the 1990s. Our Cray J932 was of course a one of the centrepieces of the kit on display, and it boasted a large power led and somewhat recessed power button below. One student asked what would happen if he pressed that button. My deadpan reply was that a small claw would come out of the recess containing the button and would clip off hos finger, and if that failed I had some plyers that would perform the same task manually.

"I had some plyers that would perform the same task manually."

Bebu sa Ware

I had some plyers [sic] that would perform the same task manually.

You must have amassed a considerable digital collection over the years although I favour bolt cutters or poultry shears myself.

Re: "I had some plyers that would perform the same task manually."

El blissett

A good long pair of garden shears lets you keep yourself and your clothes a reasonable distance from any blood spurts.

Do not press this button again

Pete 2

> David leaned back against the big power button on the rack

The moral being that if you have a critical button, exposed on a vital server, make doubly sure it is not connected to anything.

Re: Do not press this button again

Benegesserict Cumbersomberbatch

MAGIC o-- MORE MAGIC

Anonymous Coward

Previous employer - who will remain nameless - had a medium-ish sized data room with servers for several councils, itself and other medium sized businesses. Unfortunately, the emergency power off button was next to the door. And a poor HP engineer carrying some failed components out after a repair reached out to press the button he thought would open the door. Cue immediate silence as the entire place just powered down...

He wasn't popular, but it should at least have had a cover over it.

Pascal Monett

I've already told this story, but it seems fitting to tell it again.

I was once (long ago) an operator for a Bull DPS 7. Back then, that meant an entire room, with four dishwasher-sized things for holding one HDD each, HDD which could hold a magnificent (at the time) 40MB of data. Not to mention the four backup tape arrays. It was a pretty impressive site at the time, is what I'm saying.

There were no less than three engineers on site, and one day, two of them entered the server room in the middle of a lively discussion. It wasn't an argument, from where I was sitting (the other side of the room, in front of the console), they were just talking about something and it was obviously interesting.

The point is, the big red button was right next to the door, and one of the engineers was finishing what was obviously his big story. To emphasize the end of his story, he spread his arms wide and - whack - hit the big red button.

I believe that he didn't expect his story to end with the total silence of the server room . . .

Quick Change

Sam not the Viking

We had been discussing the replacement of some old machinery for 25 years!!! The customer (a statutory 'Authority') got round to updating and finalising the specification as the technology had moved on over the years and after a long process, we were fortunate to win the order. Big contract, four large machines controlled by modern wizardry with all the data acquisition and control you could imagine. Because of the importance of the site, a full spare machine was included in the quotation on the basis that 'At least three machines must be available at all times'. Also, a changeover had to be completed within 48 hours and would be demonstrated as part of the contract. Our proposal was to exchange two machines and their kit to prove the concept.

On the day of the 'demonstration' the customer, consultant and all the hangers-on assembled for the breakfast we had purchased for these big-wigs. As the clock was started, they dashed for the best cuts and fresh coffee whilst we got on with the job. An hour later they came out to see how we were getting on and were surprised to find the job complete.

"Impossible" they said, so we repeated the task before they went off for a self-congratulatory lunch, again at our expense.

We never did get the order for the spare.....

Badges? We ain't got no badges! We don't need no badges. I don't have
to show you any stinkin' badges!
-- "The Treasure of the Sierra Madre"