News: 1700465527

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Why have just one firewall when you can fire all the walls?

(2023/11/20)


Who, Me? To quote the ancient philosophers: "Monday Monday, dah dah dah, can't trust that day." And so it is, dear reader, that we find ourselves yet again betrayed by the beginning of the working week and its requiremnet to spend the next five days exchanging your labour for currency. Fear not, though, for we can rely on The Reg to soften the blow with a dose of Who, Me? in which readers share their own tales of the treachery of tech.

For example, meet a reader we'll Regomize as "Charles" who once worked as a "support monkey" in a university Biology department.

Fear not! Charles's role may have had a simian name, but he was there to fix tech, not to have it tested on him.

[1]

Fixing things was challenging as Charles told us this team had a very broad remit indeed. "We did everything computer related – managed the servers and desktops, produced bespoke hardware and software packages for experiments, data recovery, software support, and generally acted like the computer concierge for the department's employees," Charles told Who, Me?, adding "If it had a CPU it was our responsibility."

[2]

[3]

One particularly challenging assignment followed the installation of an electron microscope. The microscope was controlled by a PC, and that PC in turn had to communicate with one other workstation, which was to be the "staging location" for data from the microscope. From there, data would be doled out as needed to whichever boffin needed it.

The PCs and workstations in the department were all networked, and it was vital that this PC controlling the microscope was not accessible from any other machine on the network. It was very new and very fancy, you see, so its usage had to be tightly controlled.

[4]

As it happens, the support monkeys had recently rolled out an antivirus/security package that included a firewall, and as Charles had helped with that project he was deemed a natural selection to set up the 'scope.

He sensibly began by studying the relevant documentation and scripting some firewall rules to achieve the required outcome.

Charles noted that the documentation was not exactly helpful: "Basically it was a list of the available variables and functions without any explanations or examples."

[5]

No problem, though – Charles was sure he was fit to survive this particular environmental change.

Having drafted a first pass at the firewall rules, he assigned it to a test group using the remote policy management server. Then he began testing the rules and found that, indeed, he could not connect to the microscope PC. Success.

Then, he found that he could not in fact connect to anything. At all. Suddenly Charles felt small. Very small. So small the availability of an electron microscope felt apt.

Then the phones started ringing and Charles realized with horror what had happened. In defining the test group via the management server he had accidentally applied his draft rules to every PC on the network.

Nobody could connect to anything.

Thankfully one of Charles's ancestors in the monkey colony had anticipated just such an undesirable mutation, and had permanently and irrevocably white-listed the policy management server – so it alone still had access to the rest of the network. Charles revoked his draft rule and the network came back to life before too many users went bananas.

Of course Charles did not go on to complete the project – it was handed to someone more suited to the task. (He thinks that ultimately a hardware solution was found rather than a firewall rule.)

If you have an anecdote like this – a time when your tech skills were not quite so brilliant as you might have hoped – we'd love to hear about it and turn it into a yarn to brighten some future Monday morn. Let us know in [6]an email to Who, Me? and we'll make you anonymously famous.

Get our [7]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZVs8VElRoh8OdTAFZLskDwAAAMs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZVs8VElRoh8OdTAFZLskDwAAAMs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZVs8VElRoh8OdTAFZLskDwAAAMs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZVs8VElRoh8OdTAFZLskDwAAAMs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZVs8VElRoh8OdTAFZLskDwAAAMs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] mailto:whome@theregister.com

[7] https://whitepapers.theregister.com/



Anonymous Coward

I know of someone who did something similar on a mobile phone network.

Whilst anyone can (and ultimately will) make a mistake at some point in their career, the thing that caused his 'fast exit, stage right' was that there was no authorized change request covering the work. The icing on the cake was there had been an email sent to everyone reminding people that no changes were to be made without a change request barely a week before.

John Riddoch

Yeah, you can get away with major outages caused by an approved change record with a minor slap on the wrist and the agony of a post incident/change review, but a minor outage without a change record often results in a swift exit from the company.

Korev

I know of someone who did something similar on a mobile phone network.

You mean he was firewalled?

See icon -->

SVD_NL

"Thankfully one of Charles's ancestors in the monkey colony had anticipated just such an undesirable mutation, and had permanently and irrevocably white-listed the policy management server"

The true hero of this story!

Undoubtedly this firewall rule is written in blood. (or a lot of overtime coffee, at the very least!)

Can tell it's a Monday!

Serg

"requiremnet" indeed...

Re: Can tell it's a Monday!

Will Godfrey

Oh come on! That's the the commonest mistake in the world. I find the fatser you type the moer of them you craete :P

P.S. I still upvoted you for spotting it so quickly - want a job proof reading?

Pascal Monett

In this case, a simple spell checker would have caught the mistake.

So, are The Vultures using one over on that side of the pond now ?

TonyJ

Weirdly I always used to struggle to type administrator properly. It was usually mangled to some variation of adminini... or such.

The other favourite of mine to mangle was GPUPDATE... and even now, typing this, it almost became GPUDPATE. A weirdly incorrect muscle memory.

We've all been there.

Kildare

Let he who has never sawn off the branch he was switting on, cast the first stone - I think that's mixing metaphors but you know what I mean

Re: We've all been there.

jmch

We'll burn that bridge when we come to it!

Michael H.F. Wilkinson

Didn't affect an entire network, but I do recall a scary experience with a computer that was controlling a 1.5m diameter IR telescope high up in the mountains of Switzerland. We we testing a new IR spectrograph, and one of the instructions I got from the engineer was that I should not move the telescope below -10 degrees declination, or else the liquid nitrogen and liquid helium might get poured out of the system, and various things might fail dramatically. The software controlling the telescope was, lets say, "interesting" in that an English language user interface was a late addition (afterthought is the correct phrase). It was very basic: it would prompt you for the coordinates of the object of interest, show the coordinates on the screen, and ask for confirmation by asking "Is this OK?".

At one point, I noticed I made a typo in the coordinates of the object of interest, entering -16 deg declination rather than -6 deg. At the prompt "Is this OK?" I dutifully entered "N" for no, just as I had successfully entered "Y" for yes previously. I was horrified to see the cheerful response I had seen so often before "Then I go!", and could hear the telescope motors start humming. There was no way to stop this before it pointed to this low position in the sky. Apparently, the user interface would consider any character input as a thumbs up, except for Cntrl-D (Unix EOF). We rushed upstairs expect all kinds of damage caused by this action. Luckily, the spectrograph survived this abuse, and worked fine for the rest of the session. I did suggest to our Italian hosts that they might want to update their UI and manuals.

Korev

> I should not move the telescope below -10 degrees declination, or else the liquid nitrogen and liquid helium might get poured out of the system, and various things might fail dramatically.

Does that include the humans that got in the way?

"could hear the telescope motors start humming"

Pascal Monett

So, that is proof that 1) the programmer didn't bother with actually checking inputs, and 2) zero quality control or even proper testing.

I would venture that this would be grounds for a serious dress-down, if not a pink slip outright. When you're writing code for expensive equipment, the least you can do is make sure your code is not going to risk breaking anything and that means testing, testing and testing some more.

Because if the telescope had been damaged, it could be proven that the code did wrong and that company would have been obliged to send a new telescope free of charge.

If that had happened, I'd wager the CEO would have suddenly been very interested in testing procedures . . .

Re: "could hear the telescope motors start humming"

Michael H.F. Wilkinson

Note that the spectrograph was ours, bolted on the back of the scope, and the telescope software had nothing to do with the limits on the spectrograph, and there was no way for the coders to know that this limit applied.

imanidiot

Holy hell, with consequences that high how is anything OTHER than "Y" not taken as cancel/nope/oh-shit-no?

As soon as...

Bebu

Windows loomed on the horizon in "antivirus/security package that included a firewall" it was pretty much an iceberg looking for a Titanic :)

In a very similar environment and possibly in the same era, faced with a similar requirement I went for a custom linux/ebtables/iptables (perhaps ipchains) screening bridge between the polloi and *the asset*.

Basically a discarded PC and two network adaptors and custom 2.4 kernel + ebtables userland. [std Python "Four Yorkshireman" sketch ... "you try and tell the young people today that"...]

Worked a treat until it wasn't needed a few years later.

Re: As soon as...

SVD_NL

I've never been a fan of firewalling all PCs seperately for rules like this.

Basic security rules like blocking certain services? sure.

Network flow rules? sounds like a nightmare!

L3+ managable switches, or VLAN segmentation is a much easier to control solution, with the added benefit that you enforce compliance for all devices.

(Or back then, a network bridge like you set up)

America is the country where you buy a lifetime supply of aspirin for one
dollar, and use it up in two weeks.