FreeBSD 13.0 to ship without WireGuard support as dev steps in to fix 'grave issues' with initial implementation
- Reference: 1616500837
- News link: https://www.theregister.co.uk/2021/03/23/freebsd_130_no_wireguard/
- Source link:
WireGuard, created by Jason A Donenfeld, was among the most warmly anticipated new features in FreeBSD 13.0. Netgate, sponsors of the popular FreeBSD firewall pfSense, [1]funded a "well-known member of the FreeBSD community in 2019 to bring kernel-mode WireGuard support to FreeBSD."
[2]
The work was completed in August 2020 and upstreamed into FreeBSD in November 2020.
[3]
Over 45,000 lines of code were removed from the FreeBSD 13.0 release
[4]
Unfortunately, Donenfeld reported last week that he had been disappointed by the implementation, [5]saying that "a popular firewall vendor tasked a developer with writing a WireGuard implementation for FreeBSD. They didn't bother reaching out to the project."
When it came to the code, he said:
There were random sleeps added to "fix" race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren't careful when they write C.
Donenfeld got together with core FreeBSD developer Kyle Evans along with Matt Dunwoodie, who had worked on a WireGuard implementation for OpenBSD. The three "dug in and completely reworked the implementation," working fast in an effort to get the code into FreeBSD 13.0, which is scheduled to ship next month.
A new implementation was completed by the trio, using far fewer lines of code, and Donenfeld said: "I think we've mostly succeeded in producing something that behaves like WireGuard. The net result certainly isn't perfect, though – the Linux and OpenBSD implementations were long, careful, slow projects by comparison – but it is at least a base on which to build and improve over time."
Rethinking VPN: Tailscale startup packages Wireguard with network security [6]READ MORE
That said, he expressed reservations about "additional systems coding issues to work out – locking, lifetimes, races, and that sort of thing."
Although the original hope was to ship this new implementation in FreeBSD 13.0, Donenfeld said: "Fortunately, and contrary to our plans, it looks exceedingly likely that given the grave issues we found in the existing code, they'll in the end just disable the module from the release, and revisit for 13.1, rather than merging our fixes a few short days before the release," which he added was "probably a very wise decision that takes some courage to make."
This is correct and in FreeBSD 13.0 RC3 "the if_wg(4) pseudo driver has been removed," according to the [7]release notes .
Public dispute over code quality
Donenfeld's critique of the original code was not appreciated by Netgate's director of engineering, Scott Long, who [8]said : "My team and I were proud of the work, proud of the results, and eager to share it with the pfSense and FreeBSD communities."
Long added: "Right now, we have not found any issues that would result in a remote or unprivileged vulnerability for pfSense users who are running WireGuard."
Donenfeld then both defended his remarks and expressed the wish to de-escalate the friction between them, in a comment to his original post. Evans also contributed to the discussion, posting about his disappointment in how the communication was handled.
A further consequence is that WireGuard (using the implementation removed from FreeBSD) has also been [9]stripped from pfSense "out of an abundance of caution," according to Netgate co-owner Jim Thompson. He added that "should WireGuard again be accepted into FreeBSD, we will re-evaluate it for inclusion in a future version of pfSense software."
[10]
Despite some harsh words, it seems that all parties have had the best intentions in trying to add this important protocol to FreeBSD, and that a strong implementation will appear in version 13.1. In the meantime, it is missing. Another question is around how the removed version came so close to being included in the 13.0 release, if it is as flawed as Donenfeld suggests. ®
Get our [11]Tech Resources
[1] https://www.netgate.com/blog/pfsense-plus-pfsense-ce-dev-insights-direction.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YFoeq8zW3VQD5BJ5t134zgAAABQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://regmedia.co.uk/2021/03/23/remove.jpg
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YFoeq8zW3VQD5BJ5t134zgAAABQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://lists.zx2c4.com/pipermail/wireguard/2021-March/006494.html
[6] https://www.theregister.com/2020/04/02/rethinking_vpn_tailscale_startup_packages/
[7] https://lists.freebsd.org/pipermail/freebsd-stable/2021-March/093340.html
[8] https://www.netgate.com/blog/painful-lessons-learned-in-security-and-community.html
[9] https://www.netgate.com/blog/wireguard-removed-from-pfsense-ce-and-pfsense-plus-software.html
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YFoeq8zW3VQD5BJ5t134zgAAABQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
To be honest the description of the code in the article...
There were random sleeps added to "fix" race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren't careful when they write C.
... looks like just the kind of outsourced code that I have the pleasure of working with (which means fixing it after it blows up in live). And as in the article it just appears in the source tree one day with a ticket number and if you open the ticket there's nothing that says how or why.
Someone will be along in a moment to blame C for all that, by the way.
re: Someone will be along in a moment to blame C for all that, by the way.
I aim to please.
You say that like C isn't behind the vast majority of vulnerabilities and exploits, but it is.
If it's so difficult to write C well then surely some of the fault must be with the language.
"Programs must be written for programmers to read, and only incidentally for machines to execute." - Abelson and Sussman, Structure and Interpretation of Computer Programs
C is the tail wagging the dog.
Re: re: Someone will be along in a moment to blame C for all that, by the way.
If it's so difficult to write C well
It isn't. Problems like:
random sleeps added to "fix" race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code
are just bad coding, no matter what the language. "sleeps to fix race conditions" FFS? What sort of crappy junior-school coder thinks that's a solution?
the most spectacular buffer overflows
OK, that's easy to do in C if you're not paying attention, but there are functions you can use to avoid it, and test tools to catch it.
I was about to
sweep in and complain about poor coding.
Whenever I write kernel modules in C (Linux ugh), I find myself spending far too long detangling unintuitive preprocessor crap that has no place in 2021. When implementing secure protocols, most ciphers allow in-place operations, but usually headers need to be prepended and you will never find a solution to this problem that permits for the headers to remain memory aligned or for the buffers to be encrypted do.
This means to effectively make protocols which encapsulate higher level data, good buffer management is necessary. And while C allows you to do anything you want, as efficiently as you want, almost all solutions to the problem tends to lead towards reimplementing object oriented language features... usually in preprocessor macros or using crazy compiler extensions for tracking offsets into buffers based on their positions in structs.
There is also the whole game of kernel buffers. Since the kernel is in privileged mode, performing allocs and frees is frowned upon. The structure of the kernel memory space is expensive and dangerous to randomly allocate memory, especially if it may trigger the kernel to need to further allocate memory beyond its initial pool. Since the MMU is mostly bypassed in this mode and since C memory management generally is not relocateable, the only real solution is to overprovision needlessly.
I could (and probably should) write a book on all the numerous problems related to kernel development as well as the endless caveats of coding kernels in C, but let me simply say that while good C code is possible, it’s rare and far too many trivial tasks are managed manually and repetitively when coding C.
I don’t particularly care for the syntax of Rust or Go. But both languages run with a great concept which is... if it’s a very common task that can be added to the language with no real additional cost, do it. As such, both languages understand strings, buffers and data structures. There is no need for disgusting hacks to implement things like macros that are all hacks to support something as trivial as foreach.
C could fix these things as well. But it’s a conscious decision by the people steering the language to keep it as it is and to leave it to libraries and compiler extensions to do it instead.
I love C because if I want to write a new C compiler, I can make something usable and likely self-hosting within a few hours. But this isn’t the characteristic of a programming language I would want to use in 2021. If I were to spend my time on such a project, the first thing I’d do is build extensions for strings, buffers and data structures ... and it wouldn’t be C anymore.
Oh... and most importantly, I would drop the preprocessor and add support for domain specific language extensions. And I’d add proper RTTI. And I’d add an extension for references. And of course relocatable memory. And probably make ...
You know what... I don’t think I’d do anything other than bootstrap the new language with C and then basically just ignore the standard from there :)
Re: I was about to
You're conflating "things that are hard to do in a kernel" and "things that C doesn't do well".
Any good kernel implementation will have kernel memory allocation/free functions that are safe to use, and just because a concept has been added to a new language to hide such mechanisms doesn't necessarily mean it's usable in kernel mode. If the designers of the language support system chose to provide a kernel-safe implementation of the feature, you can use it, but not otherwise. The same applies to any language, including C.
Kernel code, by its nature, often needs to directly access hardware and/or exact memory locations. The kind of languages that are designed to hold the programmers hand and hide such hardware features tend to make that difficult, and have to provide "special" features to bypass their inherent protections. Those can also create problems when used incorrectly.
Hardware engineers vs Software Engineers
Netgate recently "updated" pfSense only to cause a lot of problems, I've used both pfSense and Netgate hardware for a great many years - as independent engineers they were both good, but once Netgate started writing software everything started to slide downhill. To quote a friend of mine from 40 years ago who used to design boards for PDP-11's; "One software writer thinks he can keep a dozen engineers busy but in fact one hardware engineer keeps 20 programmers busy."
Check out NetBSD mailing lists about NB wireguard. Same kind of story, but NetBSD maintained they had a quality implementation.
http://mail-index.netbsd.org/tech-net/2020/08/22/msg007842.html
As an outsider I got the strong feeling that the only issue was that Donenfeld was not involved in the port. He hinted about many issues, but did not seem.to give a single concrete example.
No idea what happened about the meeting the participants agreed upon.
Dig deeper than this story. Netgate has a history of stepping on it's own peepee and when it gets called out it acts like it did nothing wrong...everybody else is attacking them...and they are simply trying to help. the fact of the matter is..netgate tried tor release crappy code that exceeds the poor quality of most software these days...the project maintainer called them on it..and netgate went whining to the internet. Get over it Netgate...hire a competent engineer and work with the wireguard project maintainer.