Google: Linux kernel and its toolchains are underinvested by at least 100 engineers
- Reference: 1628080149
- News link: https://www.theregister.co.uk/2021/08/04/google_linux_kernel_security/
- Source link:
Kees Cook, a Google software engineer who has devoted much of his time to security features in the Linux kernel, has [1]posted about continuing problems in the kernel which he said have insufficient focus.
"The stable kernel releases ('bug fixes only') each contain close to 100 new fixes per week," he said. This puts pressure on Linux vendors – including those who support the countless products which run Linux – to "ignore all the fixes, pick out only 'important' fixes, or face the daunting task of taking everything," he said.
[2]
Cook partly blames the C programming language. "With Linux written in C, it will continue to have a long tail of associated problems," he said. He added that the Mitre CVE (Common Vulnerabilities and Exposures) list, used by professionals to assess the importance of bugs, is not up to the task since "not all security flaws have CVEs assigned, nor are they assigned in a timely manner."
[3]
[4]
The only solution is to continually update to the latest version of the stable release used, but Cook said that "performing continuous kernel updates... faces enormous resistance within an organization due to fear of regressions – will the update break the product?" Another issue is that many vendors use old kernels and backport the fixes, which means redundant work as multiple engineers at different companies fix the same problem.
Cook references Google's fuzzing tool, [5]Syzkaller , which is currently reporting nearly 1,000 possible issues in the Linux kernel: about 400 a year are fixed, he said, but the number is growing by 100 per year as new ones are found.
[6]
Google's fuzzing tool is finding increasing numbers of potential bugs in the Linux kernel
What is the solution? Cook has a number of proposals, including moving away from the email-only workflow used for Linux development, introducing more automated testing and fuzzing, continuous integration, and other steps to make the development process "more efficient." Currently too much kernel testing happens after a version is released, he said.
Cook also proposed improving the Linux toolchain, not least with making sure "Linux can be written in memory-safe languages like Rust."
[7]Following Torvalds' nudge, Paragon's NTFS driver for Linux is on track for kernel
[8]Thinking about upgrading to Debian Bullseye? Watch out for changes in Exim and anything using Python 2.x
[9]Make-me-admin holes found in Windows, Linux kernel
[10]Linux kernel sheds legacy IDE support, but driver-dominated 5.14 rc1 still grows
According to Cook, "based on our most conservative estimates, the Linux kernel and its toolchains are currently underinvested by at least 100 engineers." He suggested that companies move in-house engineers working on kernel code and security to work on the upstream kernel instead. "This is the only solution that will ensure a balance of security at reasonable long-term cost."
Reasonable long-term cost? Linux, which is a free operating system, largely powers many of the world's most profitable companies, not least Google itself whose parent company Alphabet [11]reported $19.36bn operating profit in its quarter ending 30 June. The company could employ an additional 100 Linux security engineers without blinking, as could Amazon, which likewise runs mostly on Linux and [12]reported revenue for its last quarter of $113.1bn.
[13]
In February this year, Google [14]said it was sponsoring two full-time developers to work on upstream kernel security. ®
Get our [15]Tech Resources
[1] https://security.googleblog.com/2021/08/linux-kernel-security-done-right.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YQq5tjmrCAp64oWaTBYbYwAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQq5tjmrCAp64oWaTBYbYwAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YQq5tjmrCAp64oWaTBYbYwAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://github.com/google/syzkaller
[6] https://regmedia.co.uk/2021/08/04/syzbot.png
[7] https://www.theregister.com/2021/08/02/paragon_ntfs_linux_kernel/
[8] https://www.theregister.com/2021/07/26/debian_bullseye_release_set_for/
[9] https://www.theregister.com/2021/07/21/windows_linux_privilege_escalation/
[10] https://www.theregister.com/2021/07/12/linux_5_14_rc1/
[11] https://www.theregister.com/2021/07/28/alphabet_q2_2021/
[12] https://www.theregister.com/2021/07/30/amazon_q2_2021/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQq5tjmrCAp64oWaTBYbYwAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2021/02/24/google_ups_linux_security_effort/
[15] https://whitepapers.theregister.com/
Fool
When I was younger I was seriously considering doing Kernel development, but who would pay me?
Doing work so that then big corporations can make billions off of it without paying a penny is a fool's errand.
The way Google is addressing the issue also sounds disrespectful. Engineers are not cattle.
Re: Fool
Engineers are not cattle.
Unfortunately they're rarely treated as pets.
Re: Fool
> Doing work so that then big corporations can make billions off of it without paying a penny is a fool's errand.
I disagree. I contribute to free software because I consider it a public good.
And there's nothing inherently wrong with making billions of dollars in of itself if you are providing something of value. There are many, many companies out there making a more moderate income from free software, and that's perfectly fine with me.
Re: Fool
It's wrong to use someone's work without payment though. I get some people are privileged and they can afford to donate their time, but this creates inequality.
If you are able to commit time, you are enforcing a pattern where a big corporation does not have to pay workers for some of the work and that creates an expectation that developers' work is not worth any money.
People from poor background then are excluded from participation in those areas, because they have to work for money to put food on the table.
This is similar problem to unpaid internships, where the places were snapped up by children coming from wealth (as they didn't have to work for money and parents paid rent, food etc) and that excluded working class children from having a path to getting better jobs.
Re: Fool
Not everything in life is valued in 'money', and not all valuable things have a monetary value.
A reductionist approach to life seldom leads to happiness, and certainly doesn't lead to a wholesome understanding.
'Linux' (kernel and stack) is, of course, not 'free' of cost, whether financial or otherwise, but the user certainly has freedom to access it, adapt it, share it and use it in ways that those tied to proprietary systems are largely excluded from. Perhaps that is what makes it worth the 'cost' for many of its users.
Re: Fool
Yes open source was targeted at users (not corporations) who couldn't afford to pay extortionate price of enterprise software.
They didn't predict that the work of so many contributors will be appropriated by corporations, sold and not reimbursed.
Basically if Google wanted to incorporate e.g. Linux in their product, they should pay all contributors for use.
If you wanted to incorporate Linux to control temperature in your shed, then you should be free to do so.
Corporations should never be allowed to use someone's work without payment.
Time for Google to pay for all the stuff it uses for free?
Some hundred of millions, just a little dent in their revenues, would do.... but let's be sincere, the move to Linux was made also to have someone else pay the costs of development as much as possible....
Has anyone asked Linus what he thinks?
And as a follow up, how far away from the blast was the reporter found?
:D
Re: Has anyone asked Linus what he thinks?
He would probably agree with a lot of the guts of the comments. He is also reportedly enthusiastic with the recent moves to make Rust a via language for the kernel.
The trick is if one company starts donating engineers, they're essentially providing value for competitors. The practicalities are a little tricky.
The mythical man hour
While I think the guy makes a reasonable point, I'm not convinced by the proposed solution. Firstly, kernel engineers aren't lying around waiting to be called up and secondly, they don't necessarily scale.
In the near future I think we'll see more proof of concepts as to whether kernel code can be ported to Rust and whether this brings any of the hoped for improvements in memory safety, etc. In a sense, the work might be valuable in itself as an exercise in shining a light on less illuminated parts of the code base. If this does work, it also provides a model for how more companies might contribute usefully by sponsoring specific kernel projects (as is already done in FreeBSD). And, of course, moving away from the monolithic kernel towards a microkernel might also become possible (and reasonable now that x86_32 and its poor context swapping no longer dominates). This is turn should make rolling releases easier for all too handle.
For companies it's much easier to argue for enlightened self-interest when talking about specific projects rather than providing engineers but being told you can't keep the code.
Im sure I read something about workmen and their tools, maybe it was work-persons being tools - either way it appears the project got cancelled after a couple of weeks / months / yrs anyway so...
Elreg, you've run this article before
A couple of months ago 'Linux shouldn't be using their build process and e-mail, they should be using a (oh shock and surprise) Google developed solution instead'
What was said then applies now. If Google are so concerned, employ 100 developers and get them to supply fixes to upstream.
I would expect that given the wide number of platforms Linux supports that whilst Rust could be useful in some instances it's not going to handle all the necessary cases.
Well Google had better stop leeching and start getting more involved.
They pay for a relatively tiny number of developers and invest vastly more into their Android and Fuschia platforms which don't exactly help anything in the long run.
This is a little like those charity adverts where some rich celebrity has the audacity to tell us little peasants to "help out" when their wealth dwarfs any contribution of ours.
So, fsck off Google. The world really would be a better place without you.
>> In February this year, Google said it was sponsoring two full-time developers to work on upstream kernel security.
That's mighty generous of them. I wonder how many they employ internally to work on kernel security?