News: 1633332604

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Firewalls? Pfft – it's no match for my mighty spares-bin PC

(2021/10/04)


Who, Me? Start your week with a warning about those temporary emergency hacks that all too often end up permanent in today's edition of [1]Who, Me?

Our story comes from "Gary" - for that is not his name - who worked for a firm that offered a variety of services to the UK banking sector. Back at the turn of the century, our hero was tasked with testing the software to support a contract the company had with one of the bigger banks.

It was a simple enough task. Data had to go from point A to point B. However, the bit between A and B involved some proprietary encryption software and a couple of obscure ports. Not ideal, but still – should be easy enough to get the data flowing?

[2]

"Except it didn't," Gary told us, "because after several fruitless months, the new shiny 'Big Red Firewall' still flatly refused to pass packets on two ports from one host to the other."

[3]

[4]

Over some adult beverages, he'd joked with the project manager that, heck, even with his basic Linux skills he could lash up something with an old PC and a pair of network cards that would do that job... and a few months later the PM asked him if he was serious.

How hard could it be? A PC and some NICs were scavenged from the spares bin. A Linux distro was installed and tweaks were made with iptables and... hey presto! A working solution!

[5]

Frankenstein's computer was slotted into a spare rack in the main server room, effectively bypassing all that costly firewall technology. "I offered a large warning that it was a very temporary fix," said Gary, "not on UPS, no disc redundancy, crap hardware, etc. etc."

It did, however, work. It worked through the busy Christmas period. It worked into the new year. Weeks turned into months. The months rolled by, and still the data flowed.

[6]Don't touch that dial – the new guy just closed the application that no one is meant to close

[7]I would drive 100 miles and I would drive 100 more just to be the man that drove 200 miles to... hit the enter key

[8]You want us to make a change? We can do it, but it'll cost you...

[9]A practical demonstration of the difference between 'resilient' and 'redundant'

"The bank (presumably) never knew any wiser except it was working as they needed."

As for Gary's reward, "I was made redundant in this time in a 'cost saving exercise' (massive cull)." He also forgot all about the hacked-together link until his phone rang and he heard the voice of an ex-colleague.

The link was down and, after a monitor was connected, the only text on the screen was "Loading vmlinuz..."

[10]

"Now, this may have been down to a cocked up kernel upgrade," said Gary, "ensuring it was as patched as it could have been after it was in place (oh, how I don't miss LILO and needing to rerun it for kernel upgrades); presumably the power had cycled at some point and triggered the issue."

He was, however, currently in Europe and unable to collect the consultancy payday for getting the PC to boot once more.

Instead, it was time for his former colleagues to rip off the temporary sticking plaster and make things work as designed, othewise data (and cash) could not flow.

"Amazingly, the proper firewall was made to work inside of 48 hours. Amazing how attention was somewhat focused..."

Patch your conscience regarding a hack that hung around far longer than it should have with a confession to the understanding vultures staffing the [11]Who, Me? desk. The Regomiser awaits. ®

Get our [12]Tech Resources



[1] https://www.theregister.com/Tag/who-me

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YVrQ7GMo@hiseOnYMeeVggAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVrQ7GMo@hiseOnYMeeVggAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVrQ7GMo@hiseOnYMeeVggAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVrQ7GMo@hiseOnYMeeVggAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2021/09/27/who_me/

[7] https://www.theregister.com/2021/09/20/who_me/

[8] https://www.theregister.com/2021/09/13/who_me/

[9] https://www.theregister.com/2021/09/06/who_me/

[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVrQ7GMo@hiseOnYMeeVggAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[11] mailto:whome@theregister.com

[12] https://whitepapers.theregister.com/



I could post my history of 'temporary' bodges .....

KittenHuffer

..... but I do need to get some w**k done this week!

I seem to have spent much of my w**king life w**king for businesses that would prefer not to spend money on a proper solution. But are willing to accept a cobbled together 'temporary' bodge that only ends up costing them more in the long run.

Re: I could post my history of 'temporary' bodges .....

hutchism

Don't w**k too hard....

Re: I could post my history of 'temporary' bodges .....

KittenHuffer

I always w**k furiously!

Re: I could post my history of 'temporary' bodges .....

gryphon

Short term fixes rarely are.

Basic rules of engineering

Anonymous Custard

As the old mantra goes, there's nothing more permanent than a temporary fix...

You can't fix that...

Dave Null

"...that's *load bearing* technical debt"

Anonymous Coward

So relatable, I remember cobbling together a firewall and FTP server (FTP! ... so a very long time ago)

This was also an old desktop PC, spare NIC and good ole Linux

This was a "stop gap" solution, only for a few weeks. ha!

My story ended up happier though, after 6 months I was able to convince the then MD that this had become "mission critical" and in fairness they invested in a hardware based firewall and a rack mount server for the FTP services

This was a medium sized business turning over nothing like a bank, I guess the larger the organisation the harder it is to speak to someone with the hope of changing things

That reminds me...

Anonymous Coward

Of the time I built a quick and dirty Oracle DB server for the devs on a project (story I've told before, but bears repeating...) to get them out of a delivery delay bind. Making it extremely clear, in writing, that the Fujitsu desktop PC with a single PII-350 (it was quite a while ago), 4 GB RAM and non-resilient IDE hard drive (and no backups) was for development of their stuff only, I let them get on with it. As you would already have guessed time passed, I forgot all about the PC, and about a year later I was contacted by the programme manager wanting me to look into improving the performance of the "production database server as it was just so horribly slow".

Re: That reminds me...

Piro

This is an odd machine. Pentium II with 4GB of RAM?

I never have heard of such a thing. Is there even a chipset that supports that combination?

It would have been in any case, unbelievable, and I refuse to believe that any desktop PC could have had space for the number of RAM slots/sticks needed.

Re: That reminds me...

Lord Elpuss

Wasn't it something like 512MB max for a PII? I know some chipsets could address 4GB in a server config but only 'see' 512MB of it at any given time - or something like that. Was a long time ago...

Re: That reminds me...

Anonymous Coward

https://en.wikipedia.org/wiki/List_of_Intel_chipsets#Pentium_Pro/II/III_chipsets would suggest the 450GX and 450NX supported up to 8GB. Granted, it would probably be a special kind of desktop PC to use one.

Sequin

I once worked as the IT manager at a tourist attraction, and was given a miniscule budget to keep the back office and point of sale systems working. After several staff losses, I also took on the reponsibility of keeeping the Audio visua systems running for the punters. I did this on the cheap, scrapping obsolete VHS machines and replacing them with 20 quid DVD players from ASDA.

I used several old PCs from home for vital tasks, and an old Epson dot matrix of mine was used to print everyone's payslips. One day I was called in to see the boss to be told that I was being made redundant and should pack up my things and leave the building, serving out my notice on gardening leave.

At this point I requested the return of all of my equipment, with the exception of a 22inch Dell CRT monitor that I personally gifted to a lovely colleague with very poor eyesight.

Panic ensued and I cared not a jot - most of the kit I dropped off at the tip on my way home.

Ken G

Why didn't you either resell the equipment to your employer or contract yourself as a service to them? As IT manager you shouldn't have exposed them to that concentration of risk.

Anonymous Coward

Because at that point he was no longer the IT manager, and couldn't reasonably be expected to give a toss

Anonymous Coward

I've done that. "You're no longer needed". Ok, uninstalls all my software, removes all my data, loads car with all my equipment, locks up, pops key through letterbox.

Eh?

jake

"how I don't miss LILO and needing to rerun it for kernel upgrades"

Because it was such a chore?

MAybe a dozen or so keystrokes ... you must have been typing your fingers to the bone, poor thing.

Re: Eh?

Lord Elpuss

Without having any further knowledge of his config, you're not qualified to make that statement.

This is every IT project

Anonymous Coward

There’s always sellotape involved in any project. I don’t even see it as a bad thing anymore. First you get it to work, then you come back and fix…

…ideally you fix in a controlled fashion, but sometimes the system makes the decisions for you.

Let’s face it: this was just the hack that failed. I’m sure there are dozens of other hacks in that deployment that never caused an issue and were never fixed.

Re: This is every IT project

Admiral Grace Hopper

"There’s always sellotape involved in any project."

Please, we're professionals, we use duct tape.

Re: This is every IT project

jake

Gaffer tape ... don't leave home without it.

Fursty Ferret

I think the big mistake here is offering his former colleagues help on the phone call, other than just a bit of fake sympathy. All this has served to do is reinforce to the company that employed him that you can extract useful work from people and then kick them out, knowing that they'll still help out for nothing if it all goes tits up.

Wrote a q&d in 1991

Diogenes

Wrote a quick and dirty in 1991 specced to run on Intel 286 machines that did all the things its mission critical bigger brother did and a lot more. I was made redundant in 2003, and heard that it was only decommissioned in 2018 along with mission critical big brother.

Guess what I found!

Kildare

When I took on the role of Network Manager at a large university hospital just beforeY2K. Amongst the other horrors i discovered was an old (even then) 386 desktop with no monitor or keyboard sitting on a desk in the main comms room. I was told that it was the DNS - I knew this to be wrong as there was another - rack mounted - box which was. No one knew the password for this box so there it sat, until my curiosity got the better of me.

I disconnected it and waited for the call! - Didn't wait too long before I heard that people were complaing about not being able to access from the Uni with whom we had links. I connected a monitor and keyboard and found it was running some ancient version of RedHat (nothing but the best here!) , Not having the password, I rebooted the box in maintenance mode and set a new root password to get in.

It was running Bind but it's sole purpoose was to act as a reverse lookup - i.e.provide the name from a given IP. I assume that this was someone's idea of security!

A new zone file on the "real" DNS provided some extra space on the desk:)

Re: Guess what I found!

Anonymous Coward

A packet sniffer on the wire would have been a better way of getting info on what it was doing

executive assistance can sometimes be a good thing.

ColinPa

Some executive assistance, such as hourly phone calls about a critical problem is not welcome. You spend more time on the call or the pre-call call (to make sure you did not say the wrong things to the executive).

We had an executive planning to come to the UK and wanted to see a demo of our bright new shiny software. We were stuck trying to get two products to talk to each other, and could not fix it. In our call with the executive assistant who was setting up the visit, we mentioned the problem. Within 10 minutes of the end of the call we had a phone call from the top guru of the other product. Within another 10 minutes we were told which fix we needed, and what the bypass was.

The executive visit was a success!

Temporary hacks aren't.

jake

If you are reading this, you're probably using the hack that I put together in 4.1BSD (now called 4.1aBSD) for part of the TCP/IP stack to be included in 4.2BSD[0]. It was supposed to be one of those "just get us through the demo, dammit" hacks. I got 'er done over Christmas/NewYears break in 1981. Virtually every version of TCP/IP since has used it. Not too bad for a quick hack ...

[0] Just to cut the usual pack of idiots putting words into my mouth off at the socks, no, I didn't write the whole stack. That's why I said "part of". It was only about 120 lines of C in total.

Anonymous Coward

Not just in software. Many years ago a local bigwig had expanded his office into the adjoining garage. As a sparks I was roped in to wire it up. However, his office shared the leccy with the plumbers next door, and they would only agree to the power being off for a prior arranged 30 minutes at some weird time of day.

So, wired up the extension, tested everything in situe, and prepared to patch into the supply. The 30 minute window opened, I patched in and tested, and everything passed. However, the linking cables passed over an area where a shelving unit was to be inserted, so I had to leave the cables and a temporary junction box which was to be replaced with a socket outlet, exposed there waiting for the shopfitter to finish off so I could then finish off.

So I waited for an opportunity for the power to be off. and trim that last cable and put in the last socket box. And waited. And waited. About five years or so later I popped in for something and the temporary connection point was *still* there. The next time I passed the office it had been demolished!

"Security is mostly a superstition. It does not exist in nature... Life is
either a daring adventure or nothing."
-- Helen Keller