News: 1690788667

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Nobody would ever work on the live server, right? Not intentionally, anyway

(2023/07/31)


Who, Me? Greetings and felicitations, dear reader-folk, and welcome to the soft landing to the working week that we at The Reg call Who, Me? in which we share tales of readers like you who found themselves in unfortunate circumstances – often of their own making.

This week's tale is a lamentable one, to be sure, as it is a story of that oldest character flaw: overconfidence. Who among us can honestly say they have never fallen victim to that same boogeyman? Our hero we shall call "Han", that's who.

Han worked for a company that supplied Unix systems to a firm that processed photographs from film.

[1]

Younger readers may not believe it, but before smartphones came along cameras used "film" - thin plastic ribbons coated in light-sensitive substances. After taking photos, camera users took those films to a retailer that sent them to a remote laboratory in which arcane processes turned them into images on paper.

[2]

[3]

Anyway, that was the sort of company Han was visiting on this particular fateful day. The computer room was on a mezzanine floor, with a window overlooking the factory floor where staff busied themselves processing orders and printing pricing labels. Dozens of dot-matrix printers buzzed and whirred constantly, and the noise was deafening even in the lofty heights of the mezzanine.

There were three servers in the computer room: Live (which ran the terminals), Backup/Test (in case Live fell over), and Reporting. There is wisdom in that configuration, obviously. You should always have a backup, and the existence of the Backup server gave Han confidence.

[4]

This fateful day, Han was to install a software update. But, of course, he was not going to install his update on the Live server. No, that would be foolish. His plan was to install the update on the Backup/Test server and, you know, test it, before making any changes to the Live system. This was sensible, and Han was rightly proud of his plan. It gave him confidence.

[5]Linux lover consumed a quarter of the network

[6]Network died, hard, during company Christmas party, leaving lone techie to fix it

[7]Turning a computer off, then on again, never goes wrong. Right?

[8]Hacking a Foosball table scored an own goal for naughty engineers

Han left home early to get on with the job. When he arrived he confirmed with the manager on duty that staff were aware of what he was doing. A few staff were using the Backup/Test server, but he should be OK to shut it down in about 15 minutes.

Armed with the knowledge that he would not be disrupting anyone's work and everyone was fully informed of his disruptive plan, Han set up to work in the computer room. He went to the backup server and typed in the command to shut it down in 15 minutes:

&gt shutdown -h -y 15

And with that, he felt confident he would have time to head to the factory canteen and get a bit of breakfast before he had to start in earnest on the upgrade. A bacon and egg roll, perhaps?

[9]

As he left he could see that the users on the Test server had received their warnings that the server was about to shut down, and this only added to his confidence. All was going well.

You can imagine, then, the sense of deflation he experienced a few minutes later when, while munching away on his breakfast, he heard the sound coming from the factory floor change. The buzzing and whirring ceased, replaced by the excited and confused exclamations of workers whose computers had mysteriously ceased to compute.

The manager appeared at the door. "What have you done?" he pleaded.

Well, it seems that at some point in the morning the Live server had encountered a fault, and the system – as it was designed to do – had fallen over to the Backup/Test server. And Han, in his overconfidence that everything was going swimmingly, had failed to check in case something was not.

As a result, he had shut down the whole business – the very thing his carefully laid plan was supposed to avoid.

It took half an hour to get everything back up and running. Thirty minutes in which Han learned an important lesson: Don't get cocky.

Have you ever been so sure you were right, only to find out you were oh so wrong? Tell us about it in [10]an email to Who, Me? and we'll lend your story a sympathetic ear.

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZMeGRhJCarbLiPg-uku0OwAAAhc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZMeGRhJCarbLiPg-uku0OwAAAhc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZMeGRhJCarbLiPg-uku0OwAAAhc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZMeGRhJCarbLiPg-uku0OwAAAhc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2023/07/24/who_me/

[6] https://www.theregister.com/2023/07/17/who_me/

[7] https://www.theregister.com/2023/07/10/who_me/

[8] https://www.theregister.com/2023/07/03/who_me/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZMeGRhJCarbLiPg-uku0OwAAAhc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Don't let that guy work without supervision

Andy Non

Han Solo?

Re: Don't let that guy work without supervision

Korev

Looks like it's time for some film puns...

Re: Don't let that guy work without supervision

Korev

Let's see what develops...

Re: Don't let that guy work without supervision

b0llchit

It can't see the light of day.

Re: Don't let that guy work without supervision

Mr Crisps

Maybe should have expected a negative reaction?

Re: Don't let that guy work without supervision

Dinanziame

"I have a bad feeling about this"

Re: Don't let that guy work without supervision

Admiral Grace Hopper

Someone has to save our skins. Into the garbage chute, fly boy.

Re: Don't let that guy work without supervision

Wanting more

I expect Han felt very exposed.

Korev

Younger readers may not believe it, but before smartphones came along cameras used "film" - thin plastic ribbons coated in light-sensitive substances.

I know this is a joke, but it does make me feel rather old...

Grey beard icon -->

wolfetone

I don't have a grey beard, but I remember having to go to Boots with a disposable camera or a roll of 35mm (actually my first film camera was a disc format!) and handing it in, waiting days to get it back. Then digital cameras happened and I didn't bother with film.

Until 4 years ago, where I bought a cheap russian 35mm camera and a roll of Ilford XP2 (black and white, but can be developed with the colour process). I've not used a digital camera since for serious photography. Don't get me wrong, I whip my iPhone out to take a photo if I have to, but if I am on holiday or just fancy a day of taking photos, I will use film.

Why? The development of it. It's alchemy mixed with meditation. It provides excitement at the same time. Load the film in to the container, mix the developer, swirl it at set intervals (or just leave it for 1 hour for interesting results), stop bath, rinse, leave them to dry. Then stick them in the scanner while taking a sneak peak of your shots looking at the negatives. Only reason I scan them is because I don't have enough room for a dark room, but if I did I would probably do that as well.

On paper it's an awful process. It's time consuming. It's expensive. It's not instant. But given we live a life of being in a rush and having instant gratification, film photography offers something different. Like I said, it's like meditation. Plus, given you have 36 chances to get a photo right it forces you to look at something and decide if it's worth it. Before I got in to film again I would happily snap away on my digital camera and I would get maybe 1 good photo and the rest would be garbage. And would I even look at that one good photo again? No, it'd stay on the device to languish with the other photos I took in the heat of the moment that never get looked at again.

Doctor Syntax

" stop bath, rinse, leave them to dry"

No fixer?

Flightmode

I too never used stop bath on film, only on prints. For (BW) film I'd do developer + agitation, rinse, fix, rinse thoroughly, squeegee.

My father, who is an avid photography enthusiast, told me when I got started that you can't go wrong with Tri-X film - the worst thing that can happen is that the emulsion itself cracks, and if it does it'll crack absolutely evenly and you'll get a cool graphical effect from it. Well, I proved him wrong when developing my first roll in the school lab one night. I did the last rinse in WAY too warm water, so when I went to dry it off, large swaths of the emulsion came off the substrate and stuck to the squeegee. It wasn't very even, but the resulting pictures were pretty ghostly and kind of cool anyway. So I guess half a point to dad for that one.

I can still remember the smell of Agfa's Rodinal developer combining with the 1:63 dilution of Kodak's Indicator Stop Bath. Fondly.

Natalie Gritpants Jr

Bet you enjoy cleaning your vinyl before warming up your tube amp. Me, I'd rather not faff if I don't have to. If I want to meditate, I have a snooze and let my brain clear out the fluff.

Have you thought of wood turning? Great hobby for guys of a certain age.

Contrex

I wouldn't equate liking film photography with being an audiophool. You can see that a decent photo is good, but a lot of audio nonsense is just hocus pocus woo-woo.

Colouring Books

Anonymous Coward

As a student in the 80s I worked in production engineering for a small electronics company. They did their visual aids by taking B&W photos of the assemblies then colouring in the relevant components in felt tip to highlight them. So if the VA was for RTVing a capacitor bank I'd take a before pic of the capacitor bank with someone holding the RTV gun in the right position, then take after pics of the finished assy. They had a darkroom on site so I learned to develop and print then I'd use felt-tip pens to colour the capacitors in blue, the tie-wraps in red and the RTV in yellow then write the instructions. The VAs looked really good and the colouring made the conmponents stand out really well.

david 12

waiting days to get it back.

... someday my prints will come ...

(Snow White. 1937)

GlenP

I took a conscious step back a couple of years ago with digital photography. Like most people* I got into the habit of shooting lots of pics in the hope I'd get a good one rather than thinking about, and setting up, the shot carefully as I used to do when shooting expensive slide film. I've stayed with digital but tried to apply a film mindset if that makes sense.

The one thing I would go back to film photography for is B&W - for some reason digital monochrome, whether produced in-camera or post processed, just never has the same look.

*I met someone at the North Norfolk railway who shot absolutely everything on 4K video then spent days searching through for the few stills shots he wanted!

Flightmode

Like most people I got into the habit of shooting lots of pics in the hope I'd get a good one rather than thinking about, and setting up, the shot carefully as I used to do when shooting expensive slide film.

Aaron Johnson noted this phenomenon in his fantastic What The Duck comic strip (2006-2016). Forgive the links to the merch pages, but the comic archive search comes up with 0 hits.

- https://www.cafepress.com/mf/15360703/shoot-like-hell_mugs?productId=83198697

- https://www.cafepress.com/mf/36095082/2spray-and-pray-black_tshirt?productId=414186889

Korev

Han worked for a company that supplied Unix systems to a firm that processed photographs from film.

[1]It's a Unix system I know this

[1] https://www.youtube.com/watch?v=dFUlAQZB9Ng

HPUX

Bebu

It's a Unix system....

HPUX from the shutdown command but I would think the root shell prompt would be '#' rather than '>' unless it was to indicate it was the backup server having taken over the production role :)

I always liked the BSD 'fake' shutdown option (-k) with 5 mins grace (to get the users off or avoid Han's mistake) followed with a real shutdown.

Anonymous Coward

Or, $deity forbid, design systems such that the only way to test the output of a change is to actually deploy it in production.

Yes, such madness is far too common.

Stuart Castle

An old friend of mine shutdown the wrong server once. Nothing major, just the primary web server for NTL.com. An outage that, IIRC, was even mentioned on El Reg at the time. I never found out what happened, but he was looking for a new job very quickly after that..

Shouldn't pwople tell him there was a problem?

Joe W

When he arrived he confirmed with the manager on duty that staff were aware of what he was doing. A few staff were using the Backup/Test server, but he should be OK to shut it down in about 15 minutes.

Ok. He told them about the plan, they said this was good with them, even telling him that some people were using the backup/test server. Sure, he could have checked the number of active connections to the machine (confirm stuff - like when working on electrical installations: you check for yourself the fuse is out and secured against being turned on again), but the production machine could have keeled over "just then", basically when he shut down the test system. It would have been better to have both a backup and a test system, but who has the ressources for that... (I hope that manglement learned this lesson at least).

Dinanziame

One of the most pathetic letter I received was from a film developement business. They had used to send ultra-glossy magazines, boasting the quality of their colors and their crisp detail. This one was a single black and white sheet of paper, acknowledging that they were mostly dying since everything was going digital, but that maybe we still had old films which we'd like more copies of, or perhaps even digital pictures to print, and they were fully ready to do the job. Like receiving a letter from your grandmother in the hospital saying that maybe you can visit her, whenever you have time, while there's still time...?

Pete Sdev

Sounds like a UX problem. There should have been a large blinking red light to indicate "live is currently down, using backup server".

Sequin

I found about 10 old rolls of undeveloped film in boxes while having a clearout and decided to have them developed to negatives as I have a negative scanner packed away in another box.

They came back with a warning that because the films were old and may have been stored in less than ideal conditions they could be degraded. Some were, but I could make out what they were. Others were fine, and the latest ones that were ok were from my daughter's 4th birthday party - she's now 31!

Hain't we got all the fools in town on our side? And hain't that a big
enough majority in any town?
-- Mark Twain, "Huckleberry Finn"