DARPA nails cash to project 'FENCE' — a smart camera that only sends pics when pixels change
- Reference: 1625549346
- News link: https://www.theregister.co.uk/2021/07/06/darpa_open_sources_bug_bounty_platform/
- Source link:
The Agency last week announced last week that Raytheon, BAE Systems and Northrop Grumman will develop the new snapper under the Fast Event-based Neuromorphic Camera and Electronics (FENCE) program.
The research and development agency [1]solicited proposals for the project in October 2020, when it sought help to build a camera that can sense motion and determine its importance, with low latency and consuming minimal energy.
[2]
The technology was originally conceived for tactical applications like robotics or autonomous vehicles in rural areas where network access is not guaranteed. Or theaters of war where scenes are highly dynamic and data-heavy.
[3]
[4]
These neuromorphic cameras have silicon circuits that mimic brain function, operating with sparse output in addition to the low latency and high energy efficiency. Dr Whitney Mason, the program manager leading the FENCE program, issued a [5]canned statement , explaining:
Event-based cameras operate under these same principles when dealing with sparse scenes, but currently lack advanced ‘intelligence’ to perform more difficult perception and control tasks.
The improvement is attributable to the neuromorphic camera’s asynchronous operation and the transmission of data only on pixels that have changed — detected by the thermal detector in the IR camera known as the focal plane array (FPA) and machine learning algorithms.
By selecting and transmitting only the images that change, neuromorphic cameras produce less than one per cent as much data in sparse scenes relative to traditional FPAs, as demonstrated in the following video:
[6]Youtube Video
[7]
DARPA said Raytheon, BAE Systems, and Northrop Grumman will work to develop the low-latency asynchronous read-out integrated circuit (ROIC) and a ROIC-integrated processing layer that identifies relevant change signals. The ROIC and processing layer combination facilitates a sensor to operate at less than 1.5 Watts.
[8]An anti-drone system that sneezes targets to death? Would that be a DARPA project? You betcha
[9]DARPA adds RISC-V to its Toolbox: Defense researchers can get special access to SiFive chip designs
[10]The FETT seeks to defeat SSITH defenses as US military goes hard on bug bounties and its Star Wars issues
DARPA’s been busy lately, as last week it also [11]announced it will be open-sourcing its Finding Exploits to Thwart Tampering (FETT) bug bounty-hunting evaluation platform and tools.
For FETT, DARPA’s first ever bug bounty program, the organization provided hundreds of researchers and engineers access to a virtual platform to search for vulnerabilities and security flaws — known as the System Security Integration Through Hardware and firmware (SSITH) program. The open-sourcing includes the back-end management of emulated systems, user-facing front-end components and the evaluation tools used for testing processor power, performance, area, security, and security properties.
The organization announced that it is also open-sourcing the baseline RISC-V processor designs used by the SSITH program and the tools for instantiating and interacting with the baseline processors on FPGA development boards and Amazon AWS F1 cloud. The open-sourcing excludes the program’s secure architectures.
DARPA said the available processor designs “provide a jumping-off point for developers that are exploring novel hardware protections and are interested in a means of evaluating them in a virtual environment”.
[12]
The FETT platform, tools, demonstrators, and supporting assets are available on [13]GitHub . ®
Get our [14]Tech Resources
[1] https://sam.gov/opp/a67d04b96be141df8064ef8fce06ad90/view
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YOQpwaGHB0z8FF8DEhktrwAAAAg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YOQpwaGHB0z8FF8DEhktrwAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YOQpwaGHB0z8FF8DEhktrwAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.darpa.mil/news-events/2021-07-02
[6] https://youtu.be/bTRM0jHKOsY
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YOQpwaGHB0z8FF8DEhktrwAAAAg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2021/06/10/darpa_antidrone_system/
[9] https://www.theregister.com/2021/04/07/sifive_darpa_chips/
[10] https://www.theregister.com/2020/06/09/darpa_bug_bounty/
[11] https://www.darpa.mil/news-events/2021-06-30a
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YOQpwaGHB0z8FF8DEhktrwAAAAg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://github.com/GaloisInc/BESSPIN.
[14] https://whitepapers.theregister.com/
It does indeed sound like normal compressed video - but that uses key frames where the whole image is sent every few frames, with frames between them showing the difference as things change. Plus lots of other goodies as well, but that's the basis.
This FENCE will have to resolve the same issue: if it gets one bit wrong in its transmission, everything afterwards is confused... so they'll have to send some sort of key frame to cope with inevitable disconnection issues, and key frames hold a shedload more data that difference frames. So it's going to be a juggling act, I feel.
Perhaps they've got a perfect transmission path, guaranteed?
But video encoders only encode what you give them.
All this camera needs to do is keep the previous raw frame in memory, and compare the current frame to that. If there are no differences (or very subtle differences), don't forward it to the video encoder. The result will be a variable frame rate (read: super low frame rate) video stream.
With a simple way to turn this mode on or off, this would take me a few hours to write, test, and ship.
this would take me a few hours to write, test, and ship.
Yes but that would not feed $$$ to the hungry military mega-corps.
Differential compression....
.. is the term DARPA doesn't seem to know, and want to reinvent, despite being like half a century old.
I myself have written remote control software for slow modems (1200-9600 bauds) that used it and only sent those regions of the screen over the cable that have actually changed, reducing typical bandwidth usage by >95%.
Re: Differential compression....
Clearly you need millions in research cash to reinvent the wheel. Like you I also came up with the same obvious idea in the 1990s when trying to write Remote Access Tools for use over 56k modems.
Just goes to show how bad their cameras must have been before this.
Re: Differential compression....
Came here to say something very similar. The compression used by Sun Microsystems Sun Ray thin clients used this approach and I guess many other implementations as you have mentioned. It seems that reinventing the wheel can be lucrative though.
Edit. Having looked up event cameras there is a twist. They are good at spotting fast moving things which would seem to be useful in a military situation. From wikipedia;
"Image reconstruction from events has the potential to create images and video with high dynamic range, high temporal resolution and minimal motion blur. Image reconstruction can be achieved using temporal smoothing, e.g. high-pass or complementary filter. Alternative methods include optimization and gradient estimation followed by Poisson integration."
"Open sourced"
> The open-sourcing excludes the program’s secure architectures.
So like open-sourcing the Linux kernel, except you just leave out the low-level parts. What's the point, besides buzzword bingo?
"detected by the thermal detector [..] and machine learning algorithms"
I have a problem understanding that. Does that mean that the camera has a statistical analysis machine sitting behind it, judging what has changed and what to send ?
Or is it that they're going to ML the thing thoroughly and put the resulting code in the camera's software ? That sounds more likely.
Oh, and I like the video that starts with the mention that it is comparing actual "normal" camera output with a simulation of what a "neuromorphic" camera would produce (because anything high-tech these days is either quantum or neuro-something, obviously). In other words, their fancy video is just a pie-in-the-sky, we-have-no-proof PR puff piece.
Low hanging fruit
Seems like they picked all the low hanging fruits and now are beating around the bush. Slap some trending keywords on the proposal and let that funding flow in.
Why does this system seem like a variation of mpeg video compression minus regular keyframes.
As I understand mpeg it first sends a complete picture (keyframe), then for subsequent frames it only sends what has changed between frames until something (time based or amount of difference) triggers a new keyframe