Facebook has developed promising asymmetric compression technology and aspires to share it with the world.

Dubbed "Superpack" and [1]detailed in a post by Facebook Engineering software engineer Sapan Bhatia, the tech "combines compiler analysis with data compression to uncover size optimizations beyond the capability of traditional compression tools".

Bhatia explains that Facebook needs compression because its apps keep bloating, but users don't want bigger apps. Algorithmic options such as Deflate and LZMA can certainly do a job, but Bhatia said they "work well with monolithic data" but "weren't enough to offset the pace of growth we were seeing in our apps".

[2]

So Facebook did its own thing.

[3]

[4]

"Superpack is based on an insight in Kolmogorov's algorithmic measure of complexity, which defines the information content of a piece of data as the length of the shortest program that can generate that data," Bhatia wrote. "In other words, data can be compressed by representing it as a program that generates the data."

Bhatia admits that the idea of reducing Kolmogorov complexity in itself "is not new to the domain of compression". But Superpack puts it to work in novel ways "by combining compiler methods with modern compression techniques".

[5]Search 'middle finger' on Giphy: Basically Facebook's response to UK competition concerns over merger

[6]Facebook apologises after its AI system branded Black people as primates

[7]DRAM-as-cache is too expensive for even Facebook – Zuck now blending it with NVM SSD drives

[8]Facebook used facial recognition without consent 200,000 times, says South Korea's data watchdog

The project works rather well. Bhatia reports that Superpack "has helped reduce the size of our fleet of Android apps, which are substantially smaller in comparison to regular Android APK compression, with average savings of over 20 per cent compared with Android's default Zip compression".

Which may be a mixed blessing given Facebook's apps have proven problematic in so many ways.

[9]

But Facebook is already contemplating other uses for Superpack, because it is good at compressing code and data.

"We are working on a new on-demand executable file format that saves disk space by keeping shared libraries compressed and decompressing them at load time," Bhatia wrote. Delta compression of code to reduce the size of software updates is another option.

Longer-term data storage is also on the agenda.

[10]

"We are also investigating using Superpack as a cold-storage compressor, to compress log data and files that are rarely used," Bhatia wrote.

The project is also applicable to iOS, and Bhatia said a port is on the cards – as is sharing it outside Facebook.

"We aspire to bring the benefits of Superpack to everyone. To this end, we are exploring ways to improve the compatibility of our compression work with the Android ecosystem," Bhatia wrote, adding "We may someday consider open-sourcing Superpack."

For now, Bhatia and Facebook are content to have blogged about Superpack. The [11]post describing the tech includes more mathematical detail and compression wonkery than it's sensible to include in a news story but will probably go some way towards satisfying the curious. ®

Get our [12]Tech Resources



[1] https://engineering.fb.com/2021/09/13/core-data/superpack/

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YUBy5iKVjAcx1mviHQ1ZSAAAAJE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YUBy5iKVjAcx1mviHQ1ZSAAAAJE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YUBy5iKVjAcx1mviHQ1ZSAAAAJE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2021/09/09/facebook_cma_giphy/

[6] https://www.theregister.com/2021/09/06/in_brief_ai/

[7] https://www.theregister.com/2021/09/03/facebook_cachelib_nvm_not_dram/

[8] https://www.theregister.com/2021/08/26/facebook_fined_by_south_korea/

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YUBy5iKVjAcx1mviHQ1ZSAAAAJE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YUBy5iKVjAcx1mviHQ1ZSAAAAJE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[11] https://engineering.fb.com/2021/09/13/core-data/superpack/

[12] https://whitepapers.theregister.com/