An important system on project [REDACTED] was all [REDACTED] up
- Reference: 1684135806
- News link: https://www.theregister.co.uk/2023/05/15/who_me/
- Source link:
This week meet a reader we'll Regomize as "Red" – short for [REDACTED]. A decade or so back (we can't be too specific about the time) Red worked for a US defense contractor on a "special access" project.
What was the project? Red can't tell us. What did it do? Red can't tell us. Who was it for? Red can't tell us. Or he could, but then, you know …
[1]
It's not worth the risk.
[2]
[3]
Only a handful of individuals apparently had the clearances and accreditations required to work on this hush-hush project, so staff turnover was very limited. It operated in a secure space, with its own walled-off access and control system (which was initially set up by a third-party contractor) as well as a dedicated security administrator. Red's job – to the extent he can tell us about it at all – was "to provide accredited secure systems to fulfill contract requirements."
Sounds like "Universal Exports" to us.
[4]
Anyway, this whole system apparently ran quite well for quite some time with few changes required – helped, in no small part, by the very low turnover of staff.
Then, the dedicated security administrator left, and unleashed havoc. The secure space became, if anything, too secure.
"One day," Red tells us, "the system would not allow anybody to enter." In order for anyone to get into the building, "a manual high security lock had to be opened by the facility security officer." The facility security officer wasn't part of the project, but was permitted access. Like, what choice was there?
[5]
At this point Red was called in. His first task was to identify what had gone wrong. And his first diagnostic discovery was that the access control software was running on a PC with Windows 95.
We mentioned this was only about a decade or so ago, right? Windows 95 was not exactly the latest and greatest thing. Or secure. Or supported.
[6]Owner of 'magic spreadsheet' tried to stay in the Lotus position until forced to Excel
[7]Your security failure was so bad we have to close the company … NOT!
[8]Support chap put PC into 'drying mode' and users believed it was real
[9]Student requested access to research data. And waited. And waited. And then hacked to get root
Before you start nodding sagely and sheeting home all blame to Microsoft, it turns out Win95 was not the sole culprit. The access control software had littered the drive with temporary files that neither it nor Win95 managed to clean up, with the result that the PC's drive was both highly fragmented and almost 95 percent full. To add to the fun, the access control software demanded ten percent of the drive be free in order for it to create its temp files – which is madness.
Red tried calling the contractor who had set up the system, only to learn that it was no longer under contract and thus not supported. Plus the super-secret project's super-secret budget had no spare pennies in it to pay for a new contract.
The fix was therefore up to Red, and Red alone. He did some manual knuckle work to free up some space on the disk so the access control system worked and everyone could get in and do their super-secret jobs. Yay Red.
Then he tried to update the access control software, and found another barrier: the update required an internet connection. Do you think this super-secret computer in a super-secret facility on a super-secret project for a US defense contractor was allowed to connect to the internet?
Yeah, nah.
So Red decided to leave the old system running on Windows 95, but cobbled together a backup and cloning system for it to keep it stable, and wrote some scripts to get rid of temp files on a daily basis. Not exactly what you'd call a permanent solution, but enough to keep it going.
And keep going it did, at the very least until Red left the project – in 2013 – to make it someone else's problem.
Ah, rugs, and the things we sweep under them. Aren't they wonderful? If you've got a tale of kicking a problem down the road for someone else to deal with, we'd love to hear about it in [10]an email to Who, Me? And we promise, we'll keep you as anonymous as our old pal [REDACTED]. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZGICwplU-vWG-BXUSVrrKQAAAMk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZGICwplU-vWG-BXUSVrrKQAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZGICwplU-vWG-BXUSVrrKQAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZGICwplU-vWG-BXUSVrrKQAAAMk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZGICwplU-vWG-BXUSVrrKQAAAMk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/05/08/who_me/
[7] https://www.theregister.com/2023/05/01/who_me/
[8] https://www.theregister.com/2023/04/24/who_me/
[9] https://www.theregister.com/2023/04/17/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
"To add to the fun, the access control software demanded ten percent of the drive be free in order for it to create its temp files – which is madness."
Madness that Microsoft copied for Exchange? Hit a fault where incoming emails were bouncing, turns out if you have less then 10% free space on the boot drive, some versions of Exchange refuses to accept incoming emails, regardless of how much space is free on the drive/partition where Exchange itself is installed.
Reminds me of one of the first versions of internet explorer for solaris (yes there was such a thing) that could only set the cache in % disk space and minimum of 1% which on a quota based unix system was not such a good idea when 1% was way more disk then you had quota for... It was promptly banned...
Support of such systems was a nightmare
[Site] your board is doing something strange
[Me] can you swap it for another board?
[Site] it's only coming out of the bunker in 2" squares
[Me] can you tell me what it is doing strange?
[Site] no!
[Me] can you give me a clue?
[Site] NO!!!
[Me] if I test my system doing A might it show the issue?
[Site] no
(I had a long list of companies bidding for contracts, and those who won it (if it had got to that stage and was announced), so I did have a bit of an idea about what the customer might be trying to do.)
...
[Me] if I test my system doing H might it show the issue?
[Site] partially
...
[Me] if I test my system doing H and M might it show the issue?
[Site] maybe
Add to this, [Site] is in a different time zone, so I have done a day at work, I have been contacted at home just before my bed time and this is now 4 a.m. Code is something approaching 6 million lines with approaching 90 concurrent (and to some extent interacting) threads if all the options are in use.
[Site Manager] this is costing us 1 million an hour when will you have it fixed?
[Me] did you trial this before today?
[Site Manager] do you think I'm "£$%^ (= no, of course we did not)
How I loved working on those issues!
Not exactly what you'd call a permanent solution,...
There is nothing more permanent than a temporary solution.
That is so true. [1]Obligatory xkcd.
[1] https://xkcd.com/2730/
Security not
Red/black network communications was controlled by a secure router. Only a limited set of commands for configuration, only a limited number of protocols allowed through (and could be tightened by command) and even such things as 'data diodes'. The UI was as tight as tight and would explain at length why it would not allow you to do what you were trying to do, and if possible, how you might achieve it by another method.
Supplier of said router was taken over by a VC group and new manglement brought in.
The new router that would be supplied in place of the super secure one came from a different section of the company, located on the same physical site where manglement presided. i.e. they had the ear!
New router was Linux based, requiring root access to the command line to manually modify text configuration files.
Supplier wondered why the [security conscious] customers left for a competitor. Plus all the engineers responsible for supporting the super secure router would have nothing to do with the new system and left.