What DARPA wants, DARPA gets: A non-hacky way to fix bugs in legacy binaries
- Reference: 1692356294
- News link: https://www.theregister.co.uk/2023/08/18/darpa_legacy_binary_patching/
- Source link:
A couple of years into a five-year DARPA project and we're perhaps well on our way there, thanks to the smart cookies at Georgia Tech. According to the US university, the GT team has, with $10 million in Pentagon funding, [1]developed a prototype pipeline that can "distill" binary executables into human-intelligible code so that it can be updated and deployed in "weeks, days, or hours, in some cases."
Hold on a moment
We know what you're thinking: Uncle Sam is reinventing decompilation. It certainly sounds like it. There are lots of decompilation and reverse-engineering tools out there for turning executable machine-level code into corresponding source code in human-readable high-level language like C or C++. That decompiled source, however, tends to be messy and hard to follow, and is typically used for figuring out how a program works and whether any bugs are exploitable.
From what we can tell, this DARPA program seeks a highly robust, automated method of converting executable files into a high-level format developers can not only read – a highly abstract representation, or HAR, in this case – but also edit to remove flaws and add functionality, and reassemble it all back into a program that will work as expected. That's a bit of a manual, error-prone chore even for highly skilled types using today's reverse-engineering tools, which isn't what you want near code going into things like aircraft.
DARPA instead seems to want a decompilation-and-recompilation system that is reliable, easy enough to use, and incorporates stuff you'd expect from a military research nerve center, such as formal verification of a program's modifications.
Ah-HAR!
With that said, let's look at this DARPA-backed work. After running an executable through the university's "distillation" process, software engineers should be able to examine the generated HAR, figure out what the code does, and make changes to add new features, patch bugs, or improve security, and turn the HAR back into executable code, says GT associate professor and project participant Brendan Saltaformaggio.
This would be useful for, say, updating complex software that was written by a contractor or internal team, the source code is no longer or never was to hand and neither are its creators, and stuff needs to be fixed up. Reverse engineering the binary and patching in an update by hand can be a little hairy, hence DARPA's desire for something a bit more solid and automatic. The idea is to use this pipeline to freshen up legacy or outdated software that may have taken years and millions of dollars to develop some time ago.
[2]
"The US government has this tremendous problem where they put tons of research and development into cutting-edge software, and then two years down the line, it needs to be updated," he said.
[3]
[4]
Yes, even after two years; it's not just for code that was finished a decade or more ago. Saltaformaggio told The Register it's still the case that software in executable form gets handed over to the Pentagon to deploy, and no one is tasked with maintaining the source code or making it available as needed, even after that short a time.
"In an ideal world someone would be hanging on to that source, and I'm sure that's sometimes the case. But not always," Saltaformaggio said.
[5]
Dare we say, a team or contractor may not be inclined to help with an update if there is no budget or agreement requiring it to do so. Rather than go through months or years of bidding, negotiations, and finally some engineering, Uncle Sam might want to skip ahead to that last part if all it wants is a bug fix, especially if it needs a critical update, stat. And if the source code is no longer available in any case, it doesn't have to be recreated from scratch: a binary update will be possible.
Indeed, GT touts its work as a way for the Dept of Defense to save millions of dollars in time and money.
A legacy code wizard, complete with spells
And so, enter DARPA's Verified Security and Performance of Large Legacy Software, or V-SPELL program, which [6]kicked off in late 2020.
The GT team is one of [7]just two groups given a grant to work on all three research thrusts for the project. Its [8]goals include decoding binary executables into a human-readable representation, making it possible for changes to the readable code, and recomposing it back into a binary executable that can be slotted into place where the old one was without issue.
Here's the pitch direct from DARPA:
The goal of the V-SPELLS program is to create a developer-accessible capability for piece-by-piece enhancement of software components with new verified code that is both correct-by-construction and compatible-by-construction, ie, safely composable with the rest of the system.
V-SPELLS will create practical tools for developers to gain benefits of formal software verification in incremental software (re)engineering rather than only in clean-slate introduction. V-SPELLS tools will enable developers to deliver assured incremental modernization of legacy systems in a manner that leverages verification technologies and reduces rather than raises risk.
V-SPELLS aims to radically broaden adoption of software verification by enabling incremental introduction of superior technologies into systems that cannot be redesigned from scratch and replaced as a whole.
Saltaformaggio told El Reg his team has the entire process working from start to finish, and with some level of stability, too. "DARPA sets challenges they like to use to test the capabilities of a project," he told us over the phone. "So far we've handled every challenge problem DARPA's thrown at us, so I'd say it's working pretty well."
Saltaformaggio said his team's pipeline disassembles binaries into a graph structure with pseudo-code, and presented in a way that developers can navigate, and replace or add parts in C and C++.
[9]DARPA wants interoperability standard for Moon living
[10]Don't shoot! DARPA wants to capture future spy balloons in one piece
[11]DARPA tells AI world: Make a model that secures software, there's $25M in it for you
[12]NASA, DARPA enlist Lockheed to build nuclear-powered spacecraft
Sorry, Java devs and Pythonistas: Saltaformaggio tells us that there's no reason the system couldn't work with other programming languages, "but we're focused on C and C++. Other folks would need to build out support for that."
Along with being able to deconstruct, edit, and reconstruct binaries, the team said its processing pipeline is also able to comb through HARs and remove extraneous routines. The team has also, we're told, baked in verification steps to ensure changes made to code within hardware ranging from jets and drones to plain-old desktop computers work exactly as expected with no side effects.
[13]
Saltaformaggio told us the V-SPELLS program ends in 2025, and his team's software is already at the stage where partners are being lined up for experiments, and the US Navy is likely first among them. Other transition partners, including companies working in the aerospace industry, are also interested in testing the pipeline, Saltaformaggio said.
As to when the civilian world can expect its own magic pipe that ingests legacy binaries and spits out something useful - that's going to take a while, but it's still likely, Saltaformaggio told us.
"DARPA programs are always way forward looking, and we're still in the very fundamental research stage," Saltaformaggio said. "But the government loves to take technology that it feels comfortable with and redeploy it for civilian uses."
"It might be a decade, but it'll happen," Saltaformaggio predicted. ®
Get our [14]Tech Resources
[1] https://coe.gatech.edu/news/2023/08/distilling-outdated-software-could-save-defense-dept-millions-time-and-money
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZN@VqdU0D4IbQ-6EyPceLAAAA5c&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZN@VqdU0D4IbQ-6EyPceLAAAA5c&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZN@VqdU0D4IbQ-6EyPceLAAAA5c&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZN@VqdU0D4IbQ-6EyPceLAAAA5c&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://sam.gov/opp/99e10fef8fe941e0813ea4220d262306/view?keywords=v-spells&sort=-relevance&index=&is_active=true&page=1
[7] https://galois.com/project/polymorph/
[8] https://www.darpa.mil/program/verified-security-and-performance-enhancement-of-large-legacy-software
[9] https://www.theregister.com/2023/08/16/luna_10_darpa/
[10] https://www.theregister.com/2023/08/10/darpa_wants_to_capture_future/
[11] https://www.theregister.com/2023/08/09/darpa_aixcc/
[12] https://www.theregister.com/2023/07/27/nasa_darpa_nuclear_spacecraft/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZN@VqdU0D4IbQ-6EyPceLAAAA5c&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
Well, first of all, it's Da Gubbermint. You are doing your patriotic duty in letting it reverse-engineer and modify binaries for which there is no code.
Second, those EULAs, apparently, all belong to defunct companies whose programmers are likely retired today (if they are still alive), and whose IP has not been bought by some company still in activity today.
Third, even if that IP had been bought, there's a fair chance that the accounting department (or internal library) has no clue that they have the rights to that code. Also, there is a non-zero chance that, even if they know, they have no desire to tell anyone because they don't want to put a finger into the update process of a software for which they might have the source code (having the IP rights doesn't mean you have the code) but don't have anyone capable of modifying it.
Because if they show up and make a fuss about IP, then they run the very real risk of being liable for the changes.
I know I wouldn't want that.
They'd probably wont be liable for any changes due to convoluted successor-of-interest clauses in the contracts.
It probably is a great opportunity for the IP holder to overcharge the government for a small change with the exclusion of any assurances and liability because it is classified as legacy code.
How did they allow themselves to get into this position anyway?
As part of the contract it should have been mandatory for the source code to have been handed over, or at the very least held in escrow in case the developers moved on, died etc. I've developed many business critical applications over the years and the more savvy companies insisted on either having the source code or the source to be held in escrow with agreed circumstances and procedures for making it available to the company.
Re: How did they allow themselves to get into this position anyway?
Maybe it's not really their code they want to fiddle with. Maybe it's some code running in some gear they've compromised that they'd like to run with some convenient new features, then reinsert without drawing a lot of attention?
Clearly, 'maybe' is doing some heavy lifting there. But who wouldn't want to patch others' militaries' code if the opportunity arose?
I can easily believe they've lost the source (or toolchains) for an awful lot of stuff, too.
Re: How did they allow themselves to get into this position anyway?
That is the marvel of the software industry.
You need to realize that source code used to be treated like gold. You had the rights to the final product, but if you wanted the source code, you paid extra (and I mean HEAVY extra).
And that's not too long ago either. About ten years ago I was working for a European Institution that had a business-critical Notes application for which it had paid a ungodly amount of money to have the rights to the source code (and that was accompanied by an iron-clad contract in which, if ever said source code was published anywhere, a number of first-borns would have to be sacrificed on a night of a new moon).
Even today, having the source code is not a given. Go ask some web designer to do you a website and get the quote for having the website and the source code for all of it. I'm thinking you'll be looking at a different figure compared to just the website and support.
Personally, I've always programmed in Lotus Notes. For me, it was a given that the applications I was writing for the client would not be code-locked.
I know of quite a few software houses in Luxembourg that don't do that today.
Re: How did they allow themselves to get into this position anyway?
Yes I agree. When I've been required to submit the source code it has always been at a premium cost. If they want exclusivity i.e. ownership of the source, then that costs much more.
Tesla Autopilot
This could be one way to fix flaws
icon =>
Java devs and Pythonistas?
Java is shipped as byte code that (unless obfuscated) that can already be converted back into reasonable source code with minimal effort. I expect python is similarly easy. Good job too as I'm pretty sure that the tool would only be easily applicable to other statically compiled languages not interpreted/JIT-ed languages.
Any tool that can cope with C will probably do a reasonable job on Fortran binaries etc. but more modern languages like RUST might be more of a challenge
Re: Java devs and Pythonistas?
As the article says: Those tools have been around for a long time. There are languages which make it easy, and others making it hard. I suspect it is about the latter languages...
Americans and their love for cool sounding abbreviations...
V-SPELL = Verified Security and Performance of Large Legacy Software
They are masters of that art. Proven so many times. Including many examples where the abbreviation and the long version contradict 100% to mask the true intention.
Beware the perils of space
Or, rather, lack thereof.
I've done rather more than my fair share of staring blankly at hex dumps of assorted ROMs trying to work out what was going on and one of the issues with more "mature" systems is that a considerable amount of ingenuity was often expended to get the code to fit the space available. I vaguely recall that the Commodore PET ROM relied rather heavily on jumping into the middle of instructions (often operands whose values were also valid opcodes).
Even if your decompiler can solve those kinds of oddities, I suspect the recompiled code stands a good chance of being bigger than the original as it's unlikely to make use of such arcane optimisations. Perhaps not even the expletive-powered encouragement of the deputy chairman of the Conservative party would be sufficient to make it go back where it's come from.
Double Edged Sword
Miscreants will also gain this increased ability to modify executable code. Security research should be done in parallel with this.
The past rises up
I imagine there is a lot of embedded code out there which was originally Ada.
So... what iif all government contracts forced you to document and comment your code?
Hold on a moment,... Isn't this the same country that has a multitude of software companies with EULAs and other contracts that specifically name reverse-engineering as a prohibited activity? I guess reverse-engineering will makes them liable for considerable damages. The military must be threatening with sending drones to those companies or they will be required to attend the inevitable sueballs, regardless of your name being "DARPA".