LWN.net Weekly Edition for March 12, 2020
- Reference: 0000813894
- News link: https://lwn.net/Articles/813894/
- Source link:
[1]Welcome to the LWN.net Weekly Edition for March 12, 2020 This edition contains the following feature content:
[2]Handling attacks on a community : as Debian contends with ongoing attacks, the question of when members of a community can be expelled arises.
[3]The short and long-term future of community conferences : the Covid-19 pandemic is wreaking havoc on community conferences, but it is not the only problem.
[4]openSUSE's board turmoil : the abrupt resignation of two openSUSE board members raises questions.
[5]The Let's Encrypt certificate revocation scare : what is to be done when a certificate authority makes a mistake involving millions of sites?
[6]Two new ways to read a file quickly : the kernel community considers two ways to optimize reading a file's contents.
This week's edition also includes these inner pages:
[7]Brief items : Brief news items from throughout the community.
[8]Announcements : Newsletters, conferences, security updates, patches, and more.
Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.
[9]Comments (none posted)
[10]Handling attacks on a community
By Jake Edge
March 11, 2020
A recent [11]message to the debian-project mailing list by Debian project leader (DPL) Sam Hartman is about a proposal to moderate the mailing list. There have been repeated attacks on various project members and the distribution itself posted to the list over the last few years, many from sock-puppet, throwaway email accounts, which spawned a recent discussion on the debian-private mailing list; Hartman was summarizing that discussion for those who are not on the private list. But the problems on debian-project (and other Debian public lists) are kind of just the tip of the iceberg; there is an ongoing, persistent effort to roil the distribution and its community.
The discussion on debian-private happened while Hartman was taking a vacation; his summary was partly from catching up with his email and also a continuation of the [12]consensus building work he has done during his DPL term. The debian-private discussion led to a [13]bug report that asked for debian-project to become a moderated list " for the time being ", which is where Hartman suggested the technical discussion on how to do the moderation should go. In parallel with the private discussion, there was also a [14]thread on the project list about how distributed moderation of a mailing list might work. The idea is that multiple moderators could help filter out messages from new subscribers to the list.
Moderating a mailing list can be controversial in free-software communities, but Hartman's summary and the thread on how to actually go about doing it indicate that there is a fairly strong consensus that the problems need to be addressed. As he put it:
There was strong although not universal support for the idea that we need to be doing something about abusive mails we're receiving. People describe the current climate as toxic and several people said that if we did not succeed in doing better, it would drive them away from Debian lists as a communications medium. Others indicated through words and actions that they were already cutting back their Debian involvement as a result.
Moderation of problematic lists was by far the best supported option for responding. Many people spoke in favor. Many people volunteered to help. I am not aware of anyone who favored any technical choice over moderation.
It is not hard to find examples of the kinds of messages that are being targeted (e.g. [15]here , [16]here , [17]here , and [18]here for fairly recent examples). It is also clear that many participants on the mailing list have concluded who is behind at least some of the anonymous/pseudonymous attacks: Daniel Pocock. In fact, Pocock was the subject of a different [19]message from Hartman on debian-project; while he did not directly connect the dots between the messages and his action expelling Pocock from the Debian project entirely, it is hard not to come to the conclusion that the two are related.
Hartman's message was meant to explain why he felt his action as DPL was reasonable under the [20]Debian Constitution but more detail on the expulsion was provided in Hartman's [21]reply to an "intent to package" (ITP) [22]bug filed by Pocock. Hartman noted that normally the project " avoids discussing expulsions in public ", but that Pocock's use of the term "Debian Developer" and attempts to act as one after having that status revoked caused the project to make an exception. Hartman also described the expulsion in some detail:
In response to some of the same actions that ultimately ended up in your key being removed from the maintainer's keyring, you were banned from all our lists. I reviewed how to respond to this ITP with members of ftpmaster, members of the account manager team and members of the community team. As part of that discussion, the question came up as to whether you were welcome at all in our community. With the concurrence of members of the account managers, ftpmaster, and community team, I conclude that you are not welcome in the Debian Project. Please stop all interactions with our lists, our BTS [bug-tracking system], our forums, salsa.debian.org, and any other Debian Project communications channels. Allowing your activity and presence in our community would only support behavior that is not welcome in our community--behavior that you have declined to stop despite multiple requests from multiple parties over an extended period of time.
It is also hard not to connect the dots between the moderation effort, expulsion, and Hartman's comments in his [23]announcement that he would not be seeking a second consecutive term as DPL. That message, which is well worth reading in its entirety, is a sober reflection on how his term has matched up with his campaign platform. It is a balanced look at his successes and failures as DPL. Tucked into the end of that message, though, is a cautionary tale for his successor.
Throughout my entire term as DPL, Debian has been subjected to a campaign of harassment from a former project member. The primary thing I'm doing my last few months as DPL is dealing with lawyers. This is not a new thing: this issue persisted through much of my predecessor's term as well.
[...] Whoever steps forward as DPL is going to need to spend some significant energy continuing to defend our community. You won't be alone. There are a number of people who are spending significant energy on this problem inside Debian and in the greater Free Software community. But I won't lie: it is a real emotional drag.
[...] My overall reaction to this situation is disappointment and horror thinking about how much damage a single motivated person can do to a community.
The damage wrought to the community is really the crux of the matter. Projects and their communities do have the right, and, some would say, the responsibility, to determine who is allowed to be a part of their community. Regardless of the merit of the accusations, a community can conclude that someone has stepped over the line and can no longer be associated with it. [24]Freedom of association cuts both ways.
It is also worth noting that, so far at least, no one has thrown their hat into the ring to run for DPL. The [25]call for nominations went out on March 7 and the nomination period ends March 14. If the current situation persists, the nomination period may need to be extended, as it [26]was last year . Hartman's warning may be giving potential candidates pause.
Debian is not the only organization to have complained about Pocock's behavior. In May 2019, the list of subscribers to [27]Free Software Foundation Europe (FSFE) mailing lists was obtained and used inappropriately, [28]according to FSFE President Matthias Kirschner :
In brief, your email addresses were used by a third party to create another mailing list, unaffiliated with and without the consent and prior knowledge of the FSFE, on the web infrastructure of another company. Shortly afterwards, the third party then ran automation scripts to unsubscribe all members of the FSFE's list, which resulted in you receiving emails requesting your confirmation to unsubscribe from the FSFE's lists.
Said third-party is identified as " Daniel Pocock and/or Ready Technology (UK) Limited ", but the message makes it clear that the FSFE believes that Pocock was, at a minimum, involved in this.
Additionally, unsubscribe requests for all members of the FSFE Discussion List were automatically generated on two separate occasions: on 2 May 2019 and 5 May 2019 (one of them proven to be from Pocock), regardless of whether or not they had requested to be unsubscribed from the FSFE Discussion List. This resulted in members receiving emails requesting them to confirm their unsubscribe request from the FSFE Discussion List.
We have gathered enough evidence to be confident that these are the events that transpired, and also to identify the parties involved in the breach. Accordingly, we have banned all relevant email addresses from the FSFE web infrastructure.
More background, at least from the FSFE side, can be found in a [29]lengthy message from [30]General Assembly member Florian Snow. There may well be legitimate concerns that Pocock has with the FSFE organizational structure, as [31]noted in this message , for example, but his tactics are seemingly not welcome in that community. As Michael Kesper [32]put it in response to a [33]message from Pocock, where he pointedly does not deny the list-subscription manipulation: " We as a community want to communicate with respect to each other as otherwise no community can survive. "
The situation is undoubtedly messy, but it is the case that several organizations have determined that the behavior is not something they want in their communities. Fedora also [34]removed Pocock's blog from its [35]Planet Fedora blog aggregator due to Code of Conduct violations. Clearly Pocock believes he is being unfairly treated by these projects, which is not surprising, but is also not really germane to the question at hand. Communities must set their own standards and individuals need to either stay within the bounds—or go elsewhere. Continuing to engage, or attack, communities that have, wrongly or rightly, excluded you is as clearly wrong as it is counterproductive. On the flipside, communities must try to ensure that they are even-handed and reasonable; sometimes reputations and even employment can be seriously affected by actions of this sort.
Dissent within a community is to be expected and should be welcomed—as long as the manner of dissenting stays within the community bounds. Opinions and complaints, even if they are not shared widely, are typically not "censored" or otherwise hindered so long as they are respectfully presented. Personal attacks, veiled threats, innuendo, and the like, however, are generally seen as "not respectful" even in a community as notoriously fractious as Debian. Dissent is important, but so is community. There is a balance to be struck and it is up to the community to do so for itself.
[36]Comments (50 posted)
[37]The short and long-term future of community conferences
By Jonathan Corbet
March 10, 2020 The Linux development community is spread out over the planet and interacts primarily through email and online systems. It is widely felt, though, that there is great value in getting people together in person occasionally to talk about current issues and get to know each other as people. This year, though, the coronavirus pandemic is disrupting the conference schedule to an extent that won't be known for some time. But there are longer-term concerns as well, to the point that the head organizer for one of the kernel community's most successful events is questioning whether it should continue to exist.
Short-term disruption
While some conferences scheduled in March have proceeded as planned — [38]SCaLE 18x being one example — many others have been canceled or postponed to a later date. There are two obvious reasons for this: gathering large numbers of people together into close quarters while a pandemic is growing is ill-advised in general, and many of the participants and speakers have decided not to participate in any case. This is creating some short-term economic pain for conference organizers, and it hurts the communities that have been deprived of their gatherings. But, by most accounts, canceling these events is the only right thing to do.
It will be interesting to see what will happen in the near future. Many of the canceled events are attempting to reschedule during the (northern hemisphere) summer months when, it is hoped, there will be a respite from the pandemic, company travel bans will be rescinded, and the mood will be generally better. It is worth noting that the authorities have been clear that this respite may or may not happen; if things do not get better, a lot of organizers and attendees are going to be disappointed a second time. In that case, regular events may not return for some time.
If the situation does improve in the coming months, then conference-goers can expect a rather busy summer as all of the postponed events land on top of the other conferences that were already scheduled during that time. That, too, may affect the success of some events; there are only so many conferences one can go to over a short period of time. Trust us, here at LWN we've explored those limits fully.
Community events in the longer term
Even before the current difficulties, community conferences have been experiencing a number of challenges. High on that list is the fact that many events are having a harder time finding sponsors; since corporate sponsors cover a large portion of the expense of running a typical event, that is an existential problem. Corporate mergers have reduced the number of sponsor prospects to begin with; a conference that once got support from both IBM and Red Hat will likely find one of those streams drying up. Companies seem to be tightening their grip on this sort of expenditure in general. Additionally, conference sponsorship money is often controlled by a company's marketing department; since a development conference tends to generate little in the way of sales leads, enthusiasm for sponsoring it will be limited.
The sheer number of conferences being run is a challenge for both sponsorship budgets and attendees. A quick look at the [39]LWN events calendar shows that there is almost always a relevant event happening somewhere. That makes for a nice range of choices for attendees — until one starts to feel the need to attend several of them. And even companies with generous sponsorship budgets can only support so many events.
There is also the fundamental question of what our events are for. In that context, it is worthwhile to take a look at [40]this message from Josef Bacik, the lead organizer for the 2020 [41]Linux Storage, Filesystem, and Memory-Management Summit . LSFMM is one of the community's most focused and technical events; it's a place where a lot of work gets done. So when its lead organizer says that it has outlived its usefulness and should be killed off, people tend to take notice.
Bacik's note describes some of the tensions that have affected a number of community events over the years. One of those has to do with size and inclusivity. Developers tend to prefer smaller events with a high concentration of people they know, and who are working on similar problems. In such a setting, it is easy to have conversations with the relevant people to get specific problems solved; this kind of event can be highly productive. As a conference gets larger, though, the number of people who are not directly involved with the topics of interest grows, and it gets harder to have focused discussions or stumble across the right people in the hallway.
Conferences like LSFMM use an invitation process in order to create the desired environment; only people who are active in the relevant areas get in, and not too many of those even. The problem with an invitation process is that, even in a world where it is run entirely fairly, it is exclusive. Established developers can get in; new developers are going to have a much harder time.
This kind of system tends to create a group of "in" participants that can be difficult for others to break into. You need to be known to the top-tier developers to get an invitation, but if you're unable to meet them at an event, it's that much harder to become known. The community absolutely needs new developers; there is no shortage of work to be done, even if the existing developers continue working indefinitely. Putting up barriers hurts the community in the long run.
One can argue for letting more people in, but that runs up against Bacik's next concern: that the conference is already getting too big. Making it even larger threatens to dilute the working nature of the conference, which Bacik fears is already reduced from what it once was. The need for sponsorships, which companies often buy because they want the attendee slots that are usually part of the deal, makes things worse. A sponsorship is an attractive product because it allows a company to bypass the invitation process and get access to the development community, but each sponsorship sold makes the conference that much bigger.
Bacik also complains about presentations — a complaint that is heard frequently around working conferences. A good conference involves a lot of discussion and problem solving; sitting passively and listening to somebody describe their work is not what attendees are looking for. Presentations, too, fill an important role; even avid LWN readers cannot be current on what is happening in all corners of the community, so that information needs to be communicated somehow. But there doesn't always seem to be a consensus on where that should happen.
Bacik suggested that, if LSFMM were to be abolished, developers could go to the [42]Linux Plumbers Conference instead. There is no invitation system there, and that event is good at getting a lot of work done as well. But LPC has a similar problem that reveals itself in a different way: the event tends to sell out quickly. In 2019, the only way in was to register during the few hours when seats were available, or to get your company to sponsor the event. That is not an entirely inclusive system either; trying to absorb an event like LSFMM would not improve the situation.
What should our conferences be?
It is clear that we have some conflicting expectations regarding our community events. We want small, focused events where established developers can solve problems without a lot of distractions around, but we also want to be open to new developers. We want to communicate our work to others and learn what others are up to, but we have limited patience for sitting through presentations. We want to let new developers — or would-be developers — meet established developers and learn about our community, but we don't want our events to get big and, to a real extent, we don't want to go the effort of meeting and teaching those people.
The community has a number of high-quality working conferences now. The pressures of fundraising and conference burnout may well reduce the number of such events in the future, though. There is no shortage of more commercially oriented events for those who want to go to them; the development community has generally chosen to sit those out in recent years. What we seem to be missing are events where the development community opens itself up to outsiders. The Linux Plumbers Conference fills that role to an extent, but only for those who can get seats. FOSDEM arguably works that way, but it is also a demonstration of what happens when there is no control over the size of the event at all.
We don't really have a model for the sort of event that achieves our many conflicting goals.
How all of this will play out remains to be seen. Perhaps, after a period when all conferences have been canceled, we will all decide that we're better off without all those airline and hotel ballroom experiences anyway, and they'll never come back. But the truth is that there is value in getting the development community together; it makes everything run far more smoothly when we're all back behind our keyboards. So we will need to figure out ways to have events that meet all of our needs; happily, figuring things out is something we are good at.
[43]Comments (17 posted)
[44]openSUSE's board turmoil
By Jonathan Corbet
March 5, 2020 Like many larger free-software projects, [45]openSUSE has [46]an elected board that is charged with handling various non-technical tasks: organizing events, dealing with conduct issues, managing the project's money, etc. Sitting on such a board is usually a relatively low-profile activity; development communities tend to pay more attention to technical contributions than other types of service. Every now and then, though, board-related issues burst into prominence; that is the case now in the openSUSE project, which will be holding a special election after the abrupt resignation of one-third of its board.
The openSUSE project has, in fact, [47]just held a board election that closed on January 31. There were four candidates for the two available seats; in the end, Simon Lees was returned to the board for another term and Sarah Julia Kriesch won the other seat. The discussion over the course of the election was perhaps a bit more contentious than usual, with Kriesch in particular stirring things up by [48]claiming to be the driving force behind the in-progress [49]openSUSE foundation effort and seemingly [50]overlooking the existence of openSUSE contributors in China (something she later [51]apologized for ). That all settled down, though, and it appeared that the new board was set to get to work after the [52]announcement of the results on February 1.
Resignations
On February 11, though, Kriesch [53]resigned her newly won board seat , citing " personal reasons ". In response, the board [54]appointed Vinzenz Vietzke — who had narrowly come in third during the election — to fill that seat.
Then, on February 27, Christian Boltz also [55]resigned from the board , stating that " some things happened in the board that are completely against my principles and beliefs ". His message did not go into details, and included a request to the community: " please don't speculate about the details or ask for them ". Unfortunately, due to the fact that there were humans involved, such a request was doomed to be ignored. Numerous community members expressed concerns about these two resignations, especially when Kriesch [56]confirmed that they were related.
One area of particular concern was the idea that the board would, again, appoint a new member to fill the empty seat. Given the fact that there was some obvious conflict happening within the board, it was not entirely unreasonable to worry that the board might be using these appointments to suppress whatever disagreement was taking place. As Jim Henderson [57]put it :
But there's a possible appearance of impropriety here, and an appointment now without general information as to the nature of the conflict is going to appear to be tainted, whether it is or isn't.
What's clear to me is this: Something is seriously wrong in the board - serious and intractable enough that two board members felt *resignation* was their best course of action.
The pressure quickly grew to the point where the board evidently felt the need to respond; that [58]happened two days later, with a statement that it was all the result of a code-of-conduct issue:
For the second time Sarah significantly breached the guiding principles and how she chose to deal with this situation led to 2/3rds of board members believing that she should step down from her role within the board.
As all of you can imagine, this decision was not an easy one, and the board gave her the opportunity to try to solve what happened. Unfortunately the ultimate outcome was to offer her to step down instead of publicly removing her from the board per the openSUSE election rules.
Boltz then [59]confirmed that the board's handling of this situation was behind his resignation, after grumbling for a bit about the posting of (previously) confidential information.
As one might imagine, there is some disagreement with how the board has represented the situation; Kriesch [60]denies any breach of the project's guiding principles, for example. And not everybody is happy with how it was handled. For the most part, though, the community appears to have accepted the explanation and is ready to get back to its real work.
Aftermath
Perhaps as a result of the concerns that were expressed about the remaining board members making another appointment, the board has [61]announced that Boltz's seat will be filled by a special election instead. Details on how that election will be run are to be posted in the near future.
Early in the discussion there were some suggestions that the entire board could resign and a new one elected, but that is not going to happen. As former board chair Richard Brown [62]pointed out , the project [63]has struggled to find candidates for elections in the past; finding enough people to fill an entirely new board in the current environment, where board members are taking fire publicly for their decisions, would be a challenge.
There was also a call for tightening the rules regarding who can serve on the board after an [64]allegation of a " special relationship " between Kriesch and Boltz. It [65]appears , though, that said relationship is not particularly special, and that there is little appetite for regulating how board members can relate to each other outside of the project in any case. So no changes are to be expected on that front.
Finally, there were also concerns about this whole drama playing out in public; that led to a [66]suggestion for the creation of a private list to keep dirty laundry out of public view. It seems that suggestion [67]may well be acted upon . Private lists are certainly not unheard of in the free-software community, and they may help to avoid some public pain. They can also keep relevant information away from stakeholders who are not on those lists, though.
Meanwhile, the work of creating the openSUSE distributions continues as usual; the board has little to do with that. Hopefully the tasks that the board does concern itself with — such as the ongoing foundation work — continue to move forward as well. Before long the board should be back up to full strength with community-elected members and this whole episode will fade into memory.
[68]Comments (none posted)
[69]The Let's Encrypt certificate revocation scare
By Jake Edge
March 10, 2020
The [70]Let's Encrypt project has made real strides in helping to ensure that every web site can use the encrypted HTTPS protocol; it has provided TLS certificates at no charge that are accepted by most or all web browsers. Free certificates accepted by the browsers are something that was difficult to find prior to the advent of the project in 2014; as of the end of February, the project has [71]issued over a billion certificates . But a bug that was recently found in the handling of [72]Certificate Authority Authorization (CAA) by the project put roughly 2.6% of the active certificates—roughly three million—at risk of immediate revocation. As might be expected, that caused a bit of panic in some quarters, but it turned out that the worst outcome was largely averted.
Let's Encrypt allows web-site operators to sign up for its service to sign their TLS certificates, so that browsers will recognize the certificate as valid. Let's Encrypt acts as a Certificate Authority (CA) and its keys are signed by a CA (IdenTrust) that is carried in the root certificate store for the browsers. That means a browser can follow the signature chain from a root certificate it trusts all the way to the certificate of the site, thus establishing the validity of the keys contained in the certificate.
In order for a site to get a certificate from Let's Encrypt, its administrator needs to show that they control the domain in question. That's typically done by adding a challenge value provided by Let's Encrypt to either the DNS information for the domain or via a URL that can be retrieved from the domain's web server. The administrator proves that they have the needed access, thus show that the domain is under their control.
Administrators who wish to restrict the kinds of certificates that can be issued for their domains can add CAA records to their DNS configuration. Those can be used to disallow certain providers, such as Let's Encrypt, from issuing certificates for a domain or portion of one. For example, the web site administrator at "subdomain.example.com" could not receive a certificate from Let's Encrypt or some other CA simply by adding a web page to the server they control if the administrator of the top-level "example.com" domain disallowed that with CAA records. Some sites may also want to restrict the CAs that can be used; some CAs offer services beyond just signing, which may be required for security or regulatory compliance.
So when Let's Encrypt is checking a site's validity, it needs to consult the CAA records as well, which turns out to be where the bug was. Let's Encrypt allows users to wait up to 30 days after proving they control the domain before requesting a certificate. But the CAA information needs to be checked within eight hours of issuance, so a recheck is done if needed. As [73]reported by Josh Aas, the executive director of the [74]Internet Security Research Group (the entity behind Let's Encrypt), the [75]Boulder CA server had a problem in the recheck code:
The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.
Before the bug was fixed, certificates issued by Let's Encrypt based on a certificate request with multiple domains in it may not have had their domain's CAA records checked properly. Those affected certificates were thus not in compliance.
That led to a [76]message on March 3 from a Let's Encrypt staff member saying that any of the affected certificates that had not been renewed by March 5 would be revoked. That would mean browsers would stop accepting the certificates from those three million sites. But by March 4, Aas [77]said that 1.7 million certificates had been renewed, which meant the existing, possibly invalid, certificates for those sites could be revoked without causing any problems. Of the remaining certificates, only 445 were for sites where the CAA record would disallow certificates being issued by Let's Encrypt; those were forcibly revoked, but the rest would not be revoked, at least immediately.
Let's Encrypt certificates are only issued for 90 days and must be renewed before the end of that time period. In the worst case, it means that around 1.3 million sites would have invalid certificates, at least in a technical sense, for up to three months. The [78]CA/Browser Forum (CA/B), which sets the standards that CAs need to comply with, does not consider certificates to be valid if the CAA records were not checked within eight hours before issuance. So even though none of those sites currently have a CAA record prohibiting the issuance of Let's Encrypt certificates, the existing set are not valid under the rules. The timeline set by CA/B for revocations is what drove the original March 5 deadline.
A Mozilla [79]bug report was filed by Aas to request an exemption from the requirement to revoke all of the affected certificates. Wayne Thayer [80]pointed Aas at the Mozilla [81]guidelines on revocation , which notes that the company does not grant exceptions but recognizes that there may be times when " revoking misissued certificates within the prescribed deadline may cause significant harm ". He also said that Mozilla requests some more information if a CA decides not to revoke the certificates.
Jacob Hoffman-Andrews [82]replied with additional details to explain why Let's Encrypt felt that it would be detrimental to do the bulk revocation. He said that users who encountered an error when browsing to an affected site would likely " look up instructions on how to bypass revocation checks "; once doing so they might well forget to re-enable those checks, so they would miss other revocations. It could also trigger "warning blindness", where users see so many warnings that they stop paying attention to them. But he noted a larger problem, as well:
By reviewing previous incident reports and analyzing our current situation, a common root cause of failure to timely revoke is that Subscribers are not able to replace certificates on the BR-
Most Subscribers are not able to field round-the-clock incident response, so improving the speed of manual replacement processes cannot be the answer. Increasing public acceptance of revoked certificate errors also cannot be the answer, because that would undermine public faith in the web PKI. Reducing the incidence and scope of CA errors is an important part of the solution, and we have laid out some plans to that effect at [84]https://bugzilla.mozilla.org/show_bug.cgi?id=1619047 . However, responsible systems design requires layered responses, and it is possible that we, or another CA, will have a similar-sized incident in the future despite our best practices and best efforts.
He said that Let's Encrypt plans to work on an open protocol to notify users of automated CAs of an imminent revocation in such a way that those certificates can be automatically renewed. In a world where even the smallest web sites have TLS certificates so that they can offer encrypted communications to their users, it is certainly important for them to be able to maintain their certificates—even without staff dedicated to handling such things. Those who are wondering can consult a [85]site where users of Let's Encrypt certificates can check whether they need an update.
The browser makers have the final authority on what root certificates they will accept, but they need to be cognizant of the impact removing one would have. If one or more of the big players decides that the steps taken by Let's Encrypt were not sufficient, they could remove the IdenTrust root certificate from their root store, though that would affect far more than just Let's Encrypt certificates. In that unlikely scenario, IdenTrust might decide (or be pressured) to revoke the Let's Encrypt certificates instead. No actions of that sort have been mooted—at least publicly. The havoc caused by such a move would be monumental.
One possible downside of the widespread availability of gratis certificates from Let's Encrypt is the creation of a monoculture. Concentrating TLS certificate issuance in a single organization might be worrisome, whether it is Let's Encrypt or one of the commercial providers. We are far from that situation now, but this incident does show that a problem found in a large number of issued certificates may leave any CA in an unenviable position—certificates that do not expire for a year or more would only add to the mess.
Overall, Let's Encrypt did an excellent job in a rather compressed time frame to identify, fix, and partly mitigate what was, in truth, just a technical violation of the specifications for CAs. It seems rather unlikely that many—perhaps any—of the remaining unrevoked certificates were actually issued for domains that they should not have been. That is not to say that technicalities should be ignored, but it is clear that sometimes there are overarching considerations as well. The bug and the problems it caused are unfortunate, for sure, but things seem to be moving in the right direction at this point.
[86]Comments (20 posted)
[87]Two new ways to read a file quickly
By Jonathan Corbet
March 6, 2020 System calls on Linux are relatively cheap, though the mitigations for speculative-execution vulnerabilities have made them more expensive than they once were. But even cheap system calls add up if one has to make a large number of them. Thus, developers have been working on ways to avoid system calls for a long time. Currently under discussion is a pair of ways to reduce the number of system calls required to read a file's contents, one of which is rather simpler than the other.
readfile()
LWN recently [88]looked at the proposed fsinfo() system call, which is intended to return information about mounted filesystems to an application. One branch of the discussion delved into whether that information could be exported via sysfs instead; one concern that was expressed about that approach is that the cost of reading a lot of little files would be too high. Miklos Szeredi [89]argued that it would not be, but suggested that, if people were concerned, they could reduce this cost by introducing a new system call to read the contents of a file:
ssize_t readfile(int dfd, const char *path, char *buf, size_t bufsize,
int flags);
The dfd and path arguments would identify a file in the usual way. A successful readfile() would read the contents of the indicated file into buf , up to a maximum of bufsize bytes, returning the number of bytes actually read. On its face, readfile() adds nothing new; an application can obtain the same result with calls to openat() , read() , and close() . But it reduces the number of system calls required from three to one, and that turns out to be interesting to some users.
In particular, Karel Zak, the maintainer of the util-linux project, [90]offered " many many beers " for the implementation of readfile() . Many of the utilities in util-linux (tools like ps and top , for example) spend a lot of time reading information from small /proc and sysfs files; having a readfile() call would make them quite a bit more efficient.
People who complain that it's hard to get kernel developers to pay attention to their problems clearly have missed an important technique; Greg Kroah-Hartman quickly [91]responded with enthusiasm: " Unlimited beers for a 21-line kernel patch? Sign me up! ". He provided a first implementation, and went on to say that this system call might actually make sense to have. Naturally, the patch has grown past 21 lines once all of the details that need to be taken into account were dealt with, and there is still a manual page to write. But it seems likely that there will be a submission of readfile() in the near future.
Of course, some people are already talking about the need for a writefile() as well.
readfile() on the ring
As the conversation progressed, Jann Horn [92]pointed out that the developers working on [93]io_uring have also expressed interest in adding a readfile() -like capability. The whole point of io_uring is to be able to perform system-call actions asynchronously and without having to actually call into the kernel, so it might seem like a good fit for this use case. He did note that truly supporting that feature in io_uring is " a bit complicated ", since there is no way to propagate a file descriptor returned by openat() to a subsequent read() operation queued in the ring. Without that, the read() cannot be queued until after the file has been opened, defeating the purpose of the exercise.
The fact of the matter, though, is that "a bit complicated" is a good description of io_uring in general. It seems unlikely that the author of a tool like ps is going to want to go through all of the effort needed to set up an io_uring instance, map it into the address space, queue some operations, and start things running just to avoid some system calls when reading /proc . But the developers of other, more complex applications would, it seems, like to have this sort of capability.
In short order, perhaps in the hope of tapping into that "unlimited beer" stream, io_uring maintainer Jens Axboe [94]posted a patch that fills in the missing piece. It works by remembering the file descriptor returned by the last openat() call in a given chain of operations. To implement a readfile() , an application could set up an io_uring chain with three operations, corresponding to the openat() , read() , and close() calls. For the latter two, though, the usual file-descriptor argument would be provided as the special constant IOSQE_FD_LAST_OPEN , which would be replaced by the descriptor for the last opened file when the operation executes.
This approach works, at the cost of complicating the interface and implementation with the magic file-descriptor substitution. Josh Triplett had a different idea, which he first [95]posted in an LWN comment in January: let applications specify which file descriptor they would like to use when opening a file. He filled out that idea in March with [96]a patch series adding a new O_SPECIFIC_FD flag to the [97]openat2() system call. This feature is available independently of io_uring; if an application really wants to open a file on descriptor 42 and no other, the new flag makes that possible.
The patch set also adds a new prctl() operation to set the minimum file descriptor to use when the application has not requested a specific one. This minimum defaults to zero, preserving the "lowest available descriptor" semantics that Unix has guaranteed forever. A developer wanting to control the file descriptors used could raise this minimum and know that the kernel would not use any of the descriptors below that minimum without an explicit request.
It only took Axboe about three hours to come up with [98]a new patch series integrating this work. It mostly consists of delaying the checks of file-descriptor validity so that they don't happen ahead of the call that actually makes a given descriptor valid. There seems to be a general agreement that this approach makes more sense than magic file-descriptor substitution, so this is the version that seems likely to go ahead.
At this point, though, this work has only circulated on the io_uring list, which has a relatively limited readership. Axboe has [99]said that he plans to post it more generally in the near future, and that merging for 5.7 is within the realm of possibility. So it may well be that there will soon be two different ways for an application to read the contents of a file with a minimum of system calls — and Karel Zak may end up buying a lot of beer.
[100]Comments (76 posted)
Page editor : Jonathan Corbet
Inside this week's LWN.net Weekly Edition
[101]Briefs : x86 root of trust flaw; Explicit graphics synchronization; DNF 5 starts; PipeWire; systemd 245; Quotes; ...
[102]Announcements : Newsletters; conferences; security updates; kernel patches; ... Next page : [103]Brief items>>
[1] https://lwn.net/Articles/814596/
[2] https://lwn.net/Articles/814508/
[3] https://lwn.net/Articles/814420/
[4] https://lwn.net/Articles/813800/
[5] https://lwn.net/Articles/814389/
[6] https://lwn.net/Articles/813827/
[7] https://lwn.net/Articles/813896/
[8] https://lwn.net/Articles/813897/
[9] https://lwn.net/Articles/814596/#Comments
[10] https://lwn.net/Articles/814508/
[11] https://lwn.net/ml/debian-project/tslo8tgge89.fsf@suchdamage.org/
[12] https://lwn.net/Articles/790382/
[13] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952465
[14] https://lwn.net/ml/debian-project/20200223095553.xn3jhl7uulibpwdf@gpm.stappers.nl/
[15] https://lwn.net/ml/debian-project/D8yjhiI1gZEt8XLHtzQI8NuwVViFQYh3UNoa7FtSy4zrYhnxbcRzorRBnWCOpPZbSO2ebmHC3xsuctHdpVtXMuVgj80vOwaMjr2PpDdx23U=@protonmail.com/
[16] https://lwn.net/ml/debian-project/959afc2e-a0ae-bbdc-14bc-3d0b93de8903@debian.community/
[17] https://lwn.net/ml/debian-project/H3wJbOsvVY23EsMR7o0Y20StUbTWCkBFYV5KKlhez1xxWXyPnrd0Sr7Zxr_okyeIuEsADHXRhM-juXVUamswjXU70tCVXEpGIPNHkb4o3jY=@protonmail.com/
[18] https://lwn.net/ml/debian-project/ea-mime-5e072373-6227-653ea4a2@www-2.mailo.com/
[19] https://lwn.net/ml/debian-project/tsllfoamorb.fsf@suchdamage.org/
[20] https://www.debian.org/devel/constitution
[21] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953378#12
[22] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953378
[23] https://lwn.net/Articles/813831/
[24] https://en.wikipedia.org/wiki/Freedom_of_association
[25] https://lwn.net/ml/debian-vote/20200307210304.GA3399625@roeckx.be/
[26] https://lwn.net/Articles/782786/
[27] https://fsfe.org/index.en.html
[28] https://lists.fsfe.org/pipermail/discussion/2019-May/012802.html
[29] https://lists.fsfe.org/pipermail/discussion/2019-May/012740.html
[30] https://fsfe.org/about/team.en.html
[31] https://lists.fsfe.org/pipermail/discussion/2019-May/012760.html
[32] https://lists.fsfe.org/pipermail/discussion/2019-May/012778.html
[33] https://lists.fsfe.org/pipermail/discussion/2019-May/012776.html
[34] https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=04afe8249f895e6e07e2ebb25c286166e40dcbad
[35] http://fedoraplanet.org/
[36] https://lwn.net/Articles/814508/#Comments
[37] https://lwn.net/Articles/814420/
[38] https://www.socallinuxexpo.org/scale/18x
[39] https://lwn.net/Calendar/
[40] https://lwn.net/ml/linux-fsdevel/b506a373-c127-b92e-9824-16e8267fc910@toxicpanda.com/
[41] https://events.linuxfoundation.org/lsfmm/
[42] https://linuxplumbersconf.org/
[43] https://lwn.net/Articles/814420/#Comments
[44] https://lwn.net/Articles/813800/
[45] https://www.opensuse.org/
[46] https://en.opensuse.org/openSUSE:Board
[47] https://en.opensuse.org/openSUSE:Board_election
[48] https://lwn.net/ml/opensuse-project/trinity-317170fa-d68f-403c-8c25-30aee7ca98d2-1579257133244@3c-app-gmx-bap49/
[49] https://lwn.net/Articles/792053/
[50] https://lwn.net/ml/opensuse-project/trinity-3759152d-9c70-4e06-b666-5124dd3ad4fe-1580109123018@3c-app-gmx-bap28/
[51] https://lwn.net/ml/opensuse-project/trinity-d2a17e56-1245-45bd-808b-ff2397b084f2-1581275553522@3c-app-gmx-bs06/
[52] https://lwn.net/ml/opensuse-project/d047ab16-cce7-d644-c86d-3867669c31f5@lasentinelle.mu/
[53] https://lwn.net/ml/opensuse-project/trinity-66c4b99a-b377-4db3-a51f-f2ffc53d8c02-1581452806688@3c-app-gmx-bs51/
[54] https://lwn.net/ml/opensuse-project/nycvar.YFH.7.76.2002271214210.3123@naguvnf.csrvsre.pbz/
[55] https://lwn.net/ml/opensuse-project/4771625.CujlEKmei7@tux.boltz.de.vu/
[56] https://lwn.net/ml/opensuse-project/trinity-23babf9d-3f4a-48cf-9dc9-c1bb74832511-1582903965972@3c-app-gmx-bap23/
[57] https://lwn.net/ml/opensuse-project/r3bkre$cfl$1@ciao.gmane.io/
[58] https://lwn.net/ml/opensuse-project/CAKW7KQSW4s-Od8cLJjZXgigj6bNyQWYLUPzHB9nLkR7vtc40tQ@mail.gmail.com/
[59] https://lwn.net/ml/opensuse-project/1739744.xEoioXefQr@tux.boltz.de.vu/
[60] https://lwn.net/ml/opensuse-project/trinity-cbf67425-7b38-4515-b877-8b81bb084ea1-1583001359544@3c-app-gmx-bs12/
[61] https://lwn.net/ml/opensuse-project/nycvar.YFH.7.76.2003040012550.3212@naguvnf.csrvsre.pbz/
[62] https://lwn.net/ml/opensuse-project/BEAF6273-552C-4A04-952C-038342843016@suse.de/
[63] https://lwn.net/Articles/776324/
[64] https://lwn.net/ml/opensuse-project/1691563.mc3XgJ6J9P@t520.axxite.internal/
[65] https://lwn.net/ml/opensuse-project/8292793.7FSq4rZfRq@tux.boltz.de.vu/
[66] https://lwn.net/ml/opensuse-project/69e1ce9a-c3cb-5dda-0f23-dfb41945e110@dodin.org/
[67] https://lwn.net/ml/opensuse-project/CAKW7KQTt2cPNXBQchGzn3+JGi-WHw_sCnVgNPxZVyfWb6H=QeQ@mail.gmail.com/
[68] https://lwn.net/Articles/813800/#Comments
[69] https://lwn.net/Articles/814389/
[70] https://letsencrypt.org/
[71] https://letsencrypt.org/2020/02/27/one-billion-certs.html
[72] https://letsencrypt.org/docs/caa/
[73] https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591
[74] https://www.abetterinternet.org/
[75] https://github.com/letsencrypt/boulder
[76] https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864
[77] https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591/3
[78] https://cabforum.org/
[79] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179
[80] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c1
[81] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
[82] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c7
[83] https://cabforum.org/baseline-requirements-documents/
[84] https://bugzilla.mozilla.org/show_bug.cgi?id=1619047
[85] https://checkhost.unboundtest.com/
[86] https://lwn.net/Articles/814389/#Comments
[87] https://lwn.net/Articles/813827/
[88] https://lwn.net/Articles/813172/
[89] https://lwn.net/ml/linux-kernel/CAJfpegtOwyaWpNfjomRVOt8NKqT94O5n4-LOHTR7YZT9fadVHA@mail.gmail.com/
[90] https://lwn.net/ml/linux-kernel/20200303113814.rsqhljkch6tgorpu@ws.net.home/
[91] https://lwn.net/ml/linux-kernel/20200303130347.GA2302029@kroah.com/
[92] https://lwn.net/ml/linux-kernel/CAG48ez3Z2V8J7dpO6t8nw7O2cMJ6z8vwLZXLAoKGH3OnCb-7JQ@mail.gmail.com/
[93] https://lwn.net/Articles/776703/
[94] https://lwn.net/ml/io-uring/20200303235053.16309-1-axboe@kernel.dk/
[95] https://lwn.net/Articles/810495/
[96] https://lwn.net/ml/io-uring/20200304143548.GA407676@localhost/
[97] https://lwn.net/Articles/796868/
[98] https://lwn.net/ml/io-uring/20200304180016.28212-1-axboe@kernel.dk/
[99] https://lwn.net/ml/io-uring/ed5c490f-4faf-afc7-bfab-d58aed061fc6@kernel.dk/
[100] https://lwn.net/Articles/813827/#Comments
[101] https://lwn.net/Articles/813896/
[102] https://lwn.net/Articles/813897/
[103] https://lwn.net/Articles/813896/
[2]Handling attacks on a community : as Debian contends with ongoing attacks, the question of when members of a community can be expelled arises.
[3]The short and long-term future of community conferences : the Covid-19 pandemic is wreaking havoc on community conferences, but it is not the only problem.
[4]openSUSE's board turmoil : the abrupt resignation of two openSUSE board members raises questions.
[5]The Let's Encrypt certificate revocation scare : what is to be done when a certificate authority makes a mistake involving millions of sites?
[6]Two new ways to read a file quickly : the kernel community considers two ways to optimize reading a file's contents.
This week's edition also includes these inner pages:
[7]Brief items : Brief news items from throughout the community.
[8]Announcements : Newsletters, conferences, security updates, patches, and more.
Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.
[9]Comments (none posted)
[10]Handling attacks on a community
By Jake Edge
March 11, 2020
A recent [11]message to the debian-project mailing list by Debian project leader (DPL) Sam Hartman is about a proposal to moderate the mailing list. There have been repeated attacks on various project members and the distribution itself posted to the list over the last few years, many from sock-puppet, throwaway email accounts, which spawned a recent discussion on the debian-private mailing list; Hartman was summarizing that discussion for those who are not on the private list. But the problems on debian-project (and other Debian public lists) are kind of just the tip of the iceberg; there is an ongoing, persistent effort to roil the distribution and its community.
The discussion on debian-private happened while Hartman was taking a vacation; his summary was partly from catching up with his email and also a continuation of the [12]consensus building work he has done during his DPL term. The debian-private discussion led to a [13]bug report that asked for debian-project to become a moderated list " for the time being ", which is where Hartman suggested the technical discussion on how to do the moderation should go. In parallel with the private discussion, there was also a [14]thread on the project list about how distributed moderation of a mailing list might work. The idea is that multiple moderators could help filter out messages from new subscribers to the list.
Moderating a mailing list can be controversial in free-software communities, but Hartman's summary and the thread on how to actually go about doing it indicate that there is a fairly strong consensus that the problems need to be addressed. As he put it:
There was strong although not universal support for the idea that we need to be doing something about abusive mails we're receiving. People describe the current climate as toxic and several people said that if we did not succeed in doing better, it would drive them away from Debian lists as a communications medium. Others indicated through words and actions that they were already cutting back their Debian involvement as a result.
Moderation of problematic lists was by far the best supported option for responding. Many people spoke in favor. Many people volunteered to help. I am not aware of anyone who favored any technical choice over moderation.
It is not hard to find examples of the kinds of messages that are being targeted (e.g. [15]here , [16]here , [17]here , and [18]here for fairly recent examples). It is also clear that many participants on the mailing list have concluded who is behind at least some of the anonymous/pseudonymous attacks: Daniel Pocock. In fact, Pocock was the subject of a different [19]message from Hartman on debian-project; while he did not directly connect the dots between the messages and his action expelling Pocock from the Debian project entirely, it is hard not to come to the conclusion that the two are related.
Hartman's message was meant to explain why he felt his action as DPL was reasonable under the [20]Debian Constitution but more detail on the expulsion was provided in Hartman's [21]reply to an "intent to package" (ITP) [22]bug filed by Pocock. Hartman noted that normally the project " avoids discussing expulsions in public ", but that Pocock's use of the term "Debian Developer" and attempts to act as one after having that status revoked caused the project to make an exception. Hartman also described the expulsion in some detail:
In response to some of the same actions that ultimately ended up in your key being removed from the maintainer's keyring, you were banned from all our lists. I reviewed how to respond to this ITP with members of ftpmaster, members of the account manager team and members of the community team. As part of that discussion, the question came up as to whether you were welcome at all in our community. With the concurrence of members of the account managers, ftpmaster, and community team, I conclude that you are not welcome in the Debian Project. Please stop all interactions with our lists, our BTS [bug-tracking system], our forums, salsa.debian.org, and any other Debian Project communications channels. Allowing your activity and presence in our community would only support behavior that is not welcome in our community--behavior that you have declined to stop despite multiple requests from multiple parties over an extended period of time.
It is also hard not to connect the dots between the moderation effort, expulsion, and Hartman's comments in his [23]announcement that he would not be seeking a second consecutive term as DPL. That message, which is well worth reading in its entirety, is a sober reflection on how his term has matched up with his campaign platform. It is a balanced look at his successes and failures as DPL. Tucked into the end of that message, though, is a cautionary tale for his successor.
Throughout my entire term as DPL, Debian has been subjected to a campaign of harassment from a former project member. The primary thing I'm doing my last few months as DPL is dealing with lawyers. This is not a new thing: this issue persisted through much of my predecessor's term as well.
[...] Whoever steps forward as DPL is going to need to spend some significant energy continuing to defend our community. You won't be alone. There are a number of people who are spending significant energy on this problem inside Debian and in the greater Free Software community. But I won't lie: it is a real emotional drag.
[...] My overall reaction to this situation is disappointment and horror thinking about how much damage a single motivated person can do to a community.
The damage wrought to the community is really the crux of the matter. Projects and their communities do have the right, and, some would say, the responsibility, to determine who is allowed to be a part of their community. Regardless of the merit of the accusations, a community can conclude that someone has stepped over the line and can no longer be associated with it. [24]Freedom of association cuts both ways.
It is also worth noting that, so far at least, no one has thrown their hat into the ring to run for DPL. The [25]call for nominations went out on March 7 and the nomination period ends March 14. If the current situation persists, the nomination period may need to be extended, as it [26]was last year . Hartman's warning may be giving potential candidates pause.
Debian is not the only organization to have complained about Pocock's behavior. In May 2019, the list of subscribers to [27]Free Software Foundation Europe (FSFE) mailing lists was obtained and used inappropriately, [28]according to FSFE President Matthias Kirschner :
In brief, your email addresses were used by a third party to create another mailing list, unaffiliated with and without the consent and prior knowledge of the FSFE, on the web infrastructure of another company. Shortly afterwards, the third party then ran automation scripts to unsubscribe all members of the FSFE's list, which resulted in you receiving emails requesting your confirmation to unsubscribe from the FSFE's lists.
Said third-party is identified as " Daniel Pocock and/or Ready Technology (UK) Limited ", but the message makes it clear that the FSFE believes that Pocock was, at a minimum, involved in this.
Additionally, unsubscribe requests for all members of the FSFE Discussion List were automatically generated on two separate occasions: on 2 May 2019 and 5 May 2019 (one of them proven to be from Pocock), regardless of whether or not they had requested to be unsubscribed from the FSFE Discussion List. This resulted in members receiving emails requesting them to confirm their unsubscribe request from the FSFE Discussion List.
We have gathered enough evidence to be confident that these are the events that transpired, and also to identify the parties involved in the breach. Accordingly, we have banned all relevant email addresses from the FSFE web infrastructure.
More background, at least from the FSFE side, can be found in a [29]lengthy message from [30]General Assembly member Florian Snow. There may well be legitimate concerns that Pocock has with the FSFE organizational structure, as [31]noted in this message , for example, but his tactics are seemingly not welcome in that community. As Michael Kesper [32]put it in response to a [33]message from Pocock, where he pointedly does not deny the list-subscription manipulation: " We as a community want to communicate with respect to each other as otherwise no community can survive. "
The situation is undoubtedly messy, but it is the case that several organizations have determined that the behavior is not something they want in their communities. Fedora also [34]removed Pocock's blog from its [35]Planet Fedora blog aggregator due to Code of Conduct violations. Clearly Pocock believes he is being unfairly treated by these projects, which is not surprising, but is also not really germane to the question at hand. Communities must set their own standards and individuals need to either stay within the bounds—or go elsewhere. Continuing to engage, or attack, communities that have, wrongly or rightly, excluded you is as clearly wrong as it is counterproductive. On the flipside, communities must try to ensure that they are even-handed and reasonable; sometimes reputations and even employment can be seriously affected by actions of this sort.
Dissent within a community is to be expected and should be welcomed—as long as the manner of dissenting stays within the community bounds. Opinions and complaints, even if they are not shared widely, are typically not "censored" or otherwise hindered so long as they are respectfully presented. Personal attacks, veiled threats, innuendo, and the like, however, are generally seen as "not respectful" even in a community as notoriously fractious as Debian. Dissent is important, but so is community. There is a balance to be struck and it is up to the community to do so for itself.
[36]Comments (50 posted)
[37]The short and long-term future of community conferences
By Jonathan Corbet
March 10, 2020 The Linux development community is spread out over the planet and interacts primarily through email and online systems. It is widely felt, though, that there is great value in getting people together in person occasionally to talk about current issues and get to know each other as people. This year, though, the coronavirus pandemic is disrupting the conference schedule to an extent that won't be known for some time. But there are longer-term concerns as well, to the point that the head organizer for one of the kernel community's most successful events is questioning whether it should continue to exist.
Short-term disruption
While some conferences scheduled in March have proceeded as planned — [38]SCaLE 18x being one example — many others have been canceled or postponed to a later date. There are two obvious reasons for this: gathering large numbers of people together into close quarters while a pandemic is growing is ill-advised in general, and many of the participants and speakers have decided not to participate in any case. This is creating some short-term economic pain for conference organizers, and it hurts the communities that have been deprived of their gatherings. But, by most accounts, canceling these events is the only right thing to do.
It will be interesting to see what will happen in the near future. Many of the canceled events are attempting to reschedule during the (northern hemisphere) summer months when, it is hoped, there will be a respite from the pandemic, company travel bans will be rescinded, and the mood will be generally better. It is worth noting that the authorities have been clear that this respite may or may not happen; if things do not get better, a lot of organizers and attendees are going to be disappointed a second time. In that case, regular events may not return for some time.
If the situation does improve in the coming months, then conference-goers can expect a rather busy summer as all of the postponed events land on top of the other conferences that were already scheduled during that time. That, too, may affect the success of some events; there are only so many conferences one can go to over a short period of time. Trust us, here at LWN we've explored those limits fully.
Community events in the longer term
Even before the current difficulties, community conferences have been experiencing a number of challenges. High on that list is the fact that many events are having a harder time finding sponsors; since corporate sponsors cover a large portion of the expense of running a typical event, that is an existential problem. Corporate mergers have reduced the number of sponsor prospects to begin with; a conference that once got support from both IBM and Red Hat will likely find one of those streams drying up. Companies seem to be tightening their grip on this sort of expenditure in general. Additionally, conference sponsorship money is often controlled by a company's marketing department; since a development conference tends to generate little in the way of sales leads, enthusiasm for sponsoring it will be limited.
The sheer number of conferences being run is a challenge for both sponsorship budgets and attendees. A quick look at the [39]LWN events calendar shows that there is almost always a relevant event happening somewhere. That makes for a nice range of choices for attendees — until one starts to feel the need to attend several of them. And even companies with generous sponsorship budgets can only support so many events.
There is also the fundamental question of what our events are for. In that context, it is worthwhile to take a look at [40]this message from Josef Bacik, the lead organizer for the 2020 [41]Linux Storage, Filesystem, and Memory-Management Summit . LSFMM is one of the community's most focused and technical events; it's a place where a lot of work gets done. So when its lead organizer says that it has outlived its usefulness and should be killed off, people tend to take notice.
Bacik's note describes some of the tensions that have affected a number of community events over the years. One of those has to do with size and inclusivity. Developers tend to prefer smaller events with a high concentration of people they know, and who are working on similar problems. In such a setting, it is easy to have conversations with the relevant people to get specific problems solved; this kind of event can be highly productive. As a conference gets larger, though, the number of people who are not directly involved with the topics of interest grows, and it gets harder to have focused discussions or stumble across the right people in the hallway.
Conferences like LSFMM use an invitation process in order to create the desired environment; only people who are active in the relevant areas get in, and not too many of those even. The problem with an invitation process is that, even in a world where it is run entirely fairly, it is exclusive. Established developers can get in; new developers are going to have a much harder time.
This kind of system tends to create a group of "in" participants that can be difficult for others to break into. You need to be known to the top-tier developers to get an invitation, but if you're unable to meet them at an event, it's that much harder to become known. The community absolutely needs new developers; there is no shortage of work to be done, even if the existing developers continue working indefinitely. Putting up barriers hurts the community in the long run.
One can argue for letting more people in, but that runs up against Bacik's next concern: that the conference is already getting too big. Making it even larger threatens to dilute the working nature of the conference, which Bacik fears is already reduced from what it once was. The need for sponsorships, which companies often buy because they want the attendee slots that are usually part of the deal, makes things worse. A sponsorship is an attractive product because it allows a company to bypass the invitation process and get access to the development community, but each sponsorship sold makes the conference that much bigger.
Bacik also complains about presentations — a complaint that is heard frequently around working conferences. A good conference involves a lot of discussion and problem solving; sitting passively and listening to somebody describe their work is not what attendees are looking for. Presentations, too, fill an important role; even avid LWN readers cannot be current on what is happening in all corners of the community, so that information needs to be communicated somehow. But there doesn't always seem to be a consensus on where that should happen.
Bacik suggested that, if LSFMM were to be abolished, developers could go to the [42]Linux Plumbers Conference instead. There is no invitation system there, and that event is good at getting a lot of work done as well. But LPC has a similar problem that reveals itself in a different way: the event tends to sell out quickly. In 2019, the only way in was to register during the few hours when seats were available, or to get your company to sponsor the event. That is not an entirely inclusive system either; trying to absorb an event like LSFMM would not improve the situation.
What should our conferences be?
It is clear that we have some conflicting expectations regarding our community events. We want small, focused events where established developers can solve problems without a lot of distractions around, but we also want to be open to new developers. We want to communicate our work to others and learn what others are up to, but we have limited patience for sitting through presentations. We want to let new developers — or would-be developers — meet established developers and learn about our community, but we don't want our events to get big and, to a real extent, we don't want to go the effort of meeting and teaching those people.
The community has a number of high-quality working conferences now. The pressures of fundraising and conference burnout may well reduce the number of such events in the future, though. There is no shortage of more commercially oriented events for those who want to go to them; the development community has generally chosen to sit those out in recent years. What we seem to be missing are events where the development community opens itself up to outsiders. The Linux Plumbers Conference fills that role to an extent, but only for those who can get seats. FOSDEM arguably works that way, but it is also a demonstration of what happens when there is no control over the size of the event at all.
We don't really have a model for the sort of event that achieves our many conflicting goals.
How all of this will play out remains to be seen. Perhaps, after a period when all conferences have been canceled, we will all decide that we're better off without all those airline and hotel ballroom experiences anyway, and they'll never come back. But the truth is that there is value in getting the development community together; it makes everything run far more smoothly when we're all back behind our keyboards. So we will need to figure out ways to have events that meet all of our needs; happily, figuring things out is something we are good at.
[43]Comments (17 posted)
[44]openSUSE's board turmoil
By Jonathan Corbet
March 5, 2020 Like many larger free-software projects, [45]openSUSE has [46]an elected board that is charged with handling various non-technical tasks: organizing events, dealing with conduct issues, managing the project's money, etc. Sitting on such a board is usually a relatively low-profile activity; development communities tend to pay more attention to technical contributions than other types of service. Every now and then, though, board-related issues burst into prominence; that is the case now in the openSUSE project, which will be holding a special election after the abrupt resignation of one-third of its board.
The openSUSE project has, in fact, [47]just held a board election that closed on January 31. There were four candidates for the two available seats; in the end, Simon Lees was returned to the board for another term and Sarah Julia Kriesch won the other seat. The discussion over the course of the election was perhaps a bit more contentious than usual, with Kriesch in particular stirring things up by [48]claiming to be the driving force behind the in-progress [49]openSUSE foundation effort and seemingly [50]overlooking the existence of openSUSE contributors in China (something she later [51]apologized for ). That all settled down, though, and it appeared that the new board was set to get to work after the [52]announcement of the results on February 1.
Resignations
On February 11, though, Kriesch [53]resigned her newly won board seat , citing " personal reasons ". In response, the board [54]appointed Vinzenz Vietzke — who had narrowly come in third during the election — to fill that seat.
Then, on February 27, Christian Boltz also [55]resigned from the board , stating that " some things happened in the board that are completely against my principles and beliefs ". His message did not go into details, and included a request to the community: " please don't speculate about the details or ask for them ". Unfortunately, due to the fact that there were humans involved, such a request was doomed to be ignored. Numerous community members expressed concerns about these two resignations, especially when Kriesch [56]confirmed that they were related.
One area of particular concern was the idea that the board would, again, appoint a new member to fill the empty seat. Given the fact that there was some obvious conflict happening within the board, it was not entirely unreasonable to worry that the board might be using these appointments to suppress whatever disagreement was taking place. As Jim Henderson [57]put it :
But there's a possible appearance of impropriety here, and an appointment now without general information as to the nature of the conflict is going to appear to be tainted, whether it is or isn't.
What's clear to me is this: Something is seriously wrong in the board - serious and intractable enough that two board members felt *resignation* was their best course of action.
The pressure quickly grew to the point where the board evidently felt the need to respond; that [58]happened two days later, with a statement that it was all the result of a code-of-conduct issue:
For the second time Sarah significantly breached the guiding principles and how she chose to deal with this situation led to 2/3rds of board members believing that she should step down from her role within the board.
As all of you can imagine, this decision was not an easy one, and the board gave her the opportunity to try to solve what happened. Unfortunately the ultimate outcome was to offer her to step down instead of publicly removing her from the board per the openSUSE election rules.
Boltz then [59]confirmed that the board's handling of this situation was behind his resignation, after grumbling for a bit about the posting of (previously) confidential information.
As one might imagine, there is some disagreement with how the board has represented the situation; Kriesch [60]denies any breach of the project's guiding principles, for example. And not everybody is happy with how it was handled. For the most part, though, the community appears to have accepted the explanation and is ready to get back to its real work.
Aftermath
Perhaps as a result of the concerns that were expressed about the remaining board members making another appointment, the board has [61]announced that Boltz's seat will be filled by a special election instead. Details on how that election will be run are to be posted in the near future.
Early in the discussion there were some suggestions that the entire board could resign and a new one elected, but that is not going to happen. As former board chair Richard Brown [62]pointed out , the project [63]has struggled to find candidates for elections in the past; finding enough people to fill an entirely new board in the current environment, where board members are taking fire publicly for their decisions, would be a challenge.
There was also a call for tightening the rules regarding who can serve on the board after an [64]allegation of a " special relationship " between Kriesch and Boltz. It [65]appears , though, that said relationship is not particularly special, and that there is little appetite for regulating how board members can relate to each other outside of the project in any case. So no changes are to be expected on that front.
Finally, there were also concerns about this whole drama playing out in public; that led to a [66]suggestion for the creation of a private list to keep dirty laundry out of public view. It seems that suggestion [67]may well be acted upon . Private lists are certainly not unheard of in the free-software community, and they may help to avoid some public pain. They can also keep relevant information away from stakeholders who are not on those lists, though.
Meanwhile, the work of creating the openSUSE distributions continues as usual; the board has little to do with that. Hopefully the tasks that the board does concern itself with — such as the ongoing foundation work — continue to move forward as well. Before long the board should be back up to full strength with community-elected members and this whole episode will fade into memory.
[68]Comments (none posted)
[69]The Let's Encrypt certificate revocation scare
By Jake Edge
March 10, 2020
The [70]Let's Encrypt project has made real strides in helping to ensure that every web site can use the encrypted HTTPS protocol; it has provided TLS certificates at no charge that are accepted by most or all web browsers. Free certificates accepted by the browsers are something that was difficult to find prior to the advent of the project in 2014; as of the end of February, the project has [71]issued over a billion certificates . But a bug that was recently found in the handling of [72]Certificate Authority Authorization (CAA) by the project put roughly 2.6% of the active certificates—roughly three million—at risk of immediate revocation. As might be expected, that caused a bit of panic in some quarters, but it turned out that the worst outcome was largely averted.
Let's Encrypt allows web-site operators to sign up for its service to sign their TLS certificates, so that browsers will recognize the certificate as valid. Let's Encrypt acts as a Certificate Authority (CA) and its keys are signed by a CA (IdenTrust) that is carried in the root certificate store for the browsers. That means a browser can follow the signature chain from a root certificate it trusts all the way to the certificate of the site, thus establishing the validity of the keys contained in the certificate.
In order for a site to get a certificate from Let's Encrypt, its administrator needs to show that they control the domain in question. That's typically done by adding a challenge value provided by Let's Encrypt to either the DNS information for the domain or via a URL that can be retrieved from the domain's web server. The administrator proves that they have the needed access, thus show that the domain is under their control.
Administrators who wish to restrict the kinds of certificates that can be issued for their domains can add CAA records to their DNS configuration. Those can be used to disallow certain providers, such as Let's Encrypt, from issuing certificates for a domain or portion of one. For example, the web site administrator at "subdomain.example.com" could not receive a certificate from Let's Encrypt or some other CA simply by adding a web page to the server they control if the administrator of the top-level "example.com" domain disallowed that with CAA records. Some sites may also want to restrict the CAs that can be used; some CAs offer services beyond just signing, which may be required for security or regulatory compliance.
So when Let's Encrypt is checking a site's validity, it needs to consult the CAA records as well, which turns out to be where the bug was. Let's Encrypt allows users to wait up to 30 days after proving they control the domain before requesting a certificate. But the CAA information needs to be checked within eight hours of issuance, so a recheck is done if needed. As [73]reported by Josh Aas, the executive director of the [74]Internet Security Research Group (the entity behind Let's Encrypt), the [75]Boulder CA server had a problem in the recheck code:
The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.
Before the bug was fixed, certificates issued by Let's Encrypt based on a certificate request with multiple domains in it may not have had their domain's CAA records checked properly. Those affected certificates were thus not in compliance.
That led to a [76]message on March 3 from a Let's Encrypt staff member saying that any of the affected certificates that had not been renewed by March 5 would be revoked. That would mean browsers would stop accepting the certificates from those three million sites. But by March 4, Aas [77]said that 1.7 million certificates had been renewed, which meant the existing, possibly invalid, certificates for those sites could be revoked without causing any problems. Of the remaining certificates, only 445 were for sites where the CAA record would disallow certificates being issued by Let's Encrypt; those were forcibly revoked, but the rest would not be revoked, at least immediately.
Let's Encrypt certificates are only issued for 90 days and must be renewed before the end of that time period. In the worst case, it means that around 1.3 million sites would have invalid certificates, at least in a technical sense, for up to three months. The [78]CA/Browser Forum (CA/B), which sets the standards that CAs need to comply with, does not consider certificates to be valid if the CAA records were not checked within eight hours before issuance. So even though none of those sites currently have a CAA record prohibiting the issuance of Let's Encrypt certificates, the existing set are not valid under the rules. The timeline set by CA/B for revocations is what drove the original March 5 deadline.
A Mozilla [79]bug report was filed by Aas to request an exemption from the requirement to revoke all of the affected certificates. Wayne Thayer [80]pointed Aas at the Mozilla [81]guidelines on revocation , which notes that the company does not grant exceptions but recognizes that there may be times when " revoking misissued certificates within the prescribed deadline may cause significant harm ". He also said that Mozilla requests some more information if a CA decides not to revoke the certificates.
Jacob Hoffman-Andrews [82]replied with additional details to explain why Let's Encrypt felt that it would be detrimental to do the bulk revocation. He said that users who encountered an error when browsing to an affected site would likely " look up instructions on how to bypass revocation checks "; once doing so they might well forget to re-enable those checks, so they would miss other revocations. It could also trigger "warning blindness", where users see so many warnings that they stop paying attention to them. But he noted a larger problem, as well:
By reviewing previous incident reports and analyzing our current situation, a common root cause of failure to timely revoke is that Subscribers are not able to replace certificates on the BR-
[83]baseline requirements
mandated timelines (24 hours and 5 days, depending on the issue).Most Subscribers are not able to field round-the-clock incident response, so improving the speed of manual replacement processes cannot be the answer. Increasing public acceptance of revoked certificate errors also cannot be the answer, because that would undermine public faith in the web PKI. Reducing the incidence and scope of CA errors is an important part of the solution, and we have laid out some plans to that effect at [84]https://bugzilla.mozilla.org/show_bug.cgi?id=1619047 . However, responsible systems design requires layered responses, and it is possible that we, or another CA, will have a similar-sized incident in the future despite our best practices and best efforts.
He said that Let's Encrypt plans to work on an open protocol to notify users of automated CAs of an imminent revocation in such a way that those certificates can be automatically renewed. In a world where even the smallest web sites have TLS certificates so that they can offer encrypted communications to their users, it is certainly important for them to be able to maintain their certificates—even without staff dedicated to handling such things. Those who are wondering can consult a [85]site where users of Let's Encrypt certificates can check whether they need an update.
The browser makers have the final authority on what root certificates they will accept, but they need to be cognizant of the impact removing one would have. If one or more of the big players decides that the steps taken by Let's Encrypt were not sufficient, they could remove the IdenTrust root certificate from their root store, though that would affect far more than just Let's Encrypt certificates. In that unlikely scenario, IdenTrust might decide (or be pressured) to revoke the Let's Encrypt certificates instead. No actions of that sort have been mooted—at least publicly. The havoc caused by such a move would be monumental.
One possible downside of the widespread availability of gratis certificates from Let's Encrypt is the creation of a monoculture. Concentrating TLS certificate issuance in a single organization might be worrisome, whether it is Let's Encrypt or one of the commercial providers. We are far from that situation now, but this incident does show that a problem found in a large number of issued certificates may leave any CA in an unenviable position—certificates that do not expire for a year or more would only add to the mess.
Overall, Let's Encrypt did an excellent job in a rather compressed time frame to identify, fix, and partly mitigate what was, in truth, just a technical violation of the specifications for CAs. It seems rather unlikely that many—perhaps any—of the remaining unrevoked certificates were actually issued for domains that they should not have been. That is not to say that technicalities should be ignored, but it is clear that sometimes there are overarching considerations as well. The bug and the problems it caused are unfortunate, for sure, but things seem to be moving in the right direction at this point.
[86]Comments (20 posted)
[87]Two new ways to read a file quickly
By Jonathan Corbet
March 6, 2020 System calls on Linux are relatively cheap, though the mitigations for speculative-execution vulnerabilities have made them more expensive than they once were. But even cheap system calls add up if one has to make a large number of them. Thus, developers have been working on ways to avoid system calls for a long time. Currently under discussion is a pair of ways to reduce the number of system calls required to read a file's contents, one of which is rather simpler than the other.
readfile()
LWN recently [88]looked at the proposed fsinfo() system call, which is intended to return information about mounted filesystems to an application. One branch of the discussion delved into whether that information could be exported via sysfs instead; one concern that was expressed about that approach is that the cost of reading a lot of little files would be too high. Miklos Szeredi [89]argued that it would not be, but suggested that, if people were concerned, they could reduce this cost by introducing a new system call to read the contents of a file:
ssize_t readfile(int dfd, const char *path, char *buf, size_t bufsize,
int flags);
The dfd and path arguments would identify a file in the usual way. A successful readfile() would read the contents of the indicated file into buf , up to a maximum of bufsize bytes, returning the number of bytes actually read. On its face, readfile() adds nothing new; an application can obtain the same result with calls to openat() , read() , and close() . But it reduces the number of system calls required from three to one, and that turns out to be interesting to some users.
In particular, Karel Zak, the maintainer of the util-linux project, [90]offered " many many beers " for the implementation of readfile() . Many of the utilities in util-linux (tools like ps and top , for example) spend a lot of time reading information from small /proc and sysfs files; having a readfile() call would make them quite a bit more efficient.
People who complain that it's hard to get kernel developers to pay attention to their problems clearly have missed an important technique; Greg Kroah-Hartman quickly [91]responded with enthusiasm: " Unlimited beers for a 21-line kernel patch? Sign me up! ". He provided a first implementation, and went on to say that this system call might actually make sense to have. Naturally, the patch has grown past 21 lines once all of the details that need to be taken into account were dealt with, and there is still a manual page to write. But it seems likely that there will be a submission of readfile() in the near future.
Of course, some people are already talking about the need for a writefile() as well.
readfile() on the ring
As the conversation progressed, Jann Horn [92]pointed out that the developers working on [93]io_uring have also expressed interest in adding a readfile() -like capability. The whole point of io_uring is to be able to perform system-call actions asynchronously and without having to actually call into the kernel, so it might seem like a good fit for this use case. He did note that truly supporting that feature in io_uring is " a bit complicated ", since there is no way to propagate a file descriptor returned by openat() to a subsequent read() operation queued in the ring. Without that, the read() cannot be queued until after the file has been opened, defeating the purpose of the exercise.
The fact of the matter, though, is that "a bit complicated" is a good description of io_uring in general. It seems unlikely that the author of a tool like ps is going to want to go through all of the effort needed to set up an io_uring instance, map it into the address space, queue some operations, and start things running just to avoid some system calls when reading /proc . But the developers of other, more complex applications would, it seems, like to have this sort of capability.
In short order, perhaps in the hope of tapping into that "unlimited beer" stream, io_uring maintainer Jens Axboe [94]posted a patch that fills in the missing piece. It works by remembering the file descriptor returned by the last openat() call in a given chain of operations. To implement a readfile() , an application could set up an io_uring chain with three operations, corresponding to the openat() , read() , and close() calls. For the latter two, though, the usual file-descriptor argument would be provided as the special constant IOSQE_FD_LAST_OPEN , which would be replaced by the descriptor for the last opened file when the operation executes.
This approach works, at the cost of complicating the interface and implementation with the magic file-descriptor substitution. Josh Triplett had a different idea, which he first [95]posted in an LWN comment in January: let applications specify which file descriptor they would like to use when opening a file. He filled out that idea in March with [96]a patch series adding a new O_SPECIFIC_FD flag to the [97]openat2() system call. This feature is available independently of io_uring; if an application really wants to open a file on descriptor 42 and no other, the new flag makes that possible.
The patch set also adds a new prctl() operation to set the minimum file descriptor to use when the application has not requested a specific one. This minimum defaults to zero, preserving the "lowest available descriptor" semantics that Unix has guaranteed forever. A developer wanting to control the file descriptors used could raise this minimum and know that the kernel would not use any of the descriptors below that minimum without an explicit request.
It only took Axboe about three hours to come up with [98]a new patch series integrating this work. It mostly consists of delaying the checks of file-descriptor validity so that they don't happen ahead of the call that actually makes a given descriptor valid. There seems to be a general agreement that this approach makes more sense than magic file-descriptor substitution, so this is the version that seems likely to go ahead.
At this point, though, this work has only circulated on the io_uring list, which has a relatively limited readership. Axboe has [99]said that he plans to post it more generally in the near future, and that merging for 5.7 is within the realm of possibility. So it may well be that there will soon be two different ways for an application to read the contents of a file with a minimum of system calls — and Karel Zak may end up buying a lot of beer.
[100]Comments (76 posted)
Page editor : Jonathan Corbet
Inside this week's LWN.net Weekly Edition
[101]Briefs : x86 root of trust flaw; Explicit graphics synchronization; DNF 5 starts; PipeWire; systemd 245; Quotes; ...
[102]Announcements : Newsletters; conferences; security updates; kernel patches; ... Next page : [103]Brief items>>
[1] https://lwn.net/Articles/814596/
[2] https://lwn.net/Articles/814508/
[3] https://lwn.net/Articles/814420/
[4] https://lwn.net/Articles/813800/
[5] https://lwn.net/Articles/814389/
[6] https://lwn.net/Articles/813827/
[7] https://lwn.net/Articles/813896/
[8] https://lwn.net/Articles/813897/
[9] https://lwn.net/Articles/814596/#Comments
[10] https://lwn.net/Articles/814508/
[11] https://lwn.net/ml/debian-project/tslo8tgge89.fsf@suchdamage.org/
[12] https://lwn.net/Articles/790382/
[13] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952465
[14] https://lwn.net/ml/debian-project/20200223095553.xn3jhl7uulibpwdf@gpm.stappers.nl/
[15] https://lwn.net/ml/debian-project/D8yjhiI1gZEt8XLHtzQI8NuwVViFQYh3UNoa7FtSy4zrYhnxbcRzorRBnWCOpPZbSO2ebmHC3xsuctHdpVtXMuVgj80vOwaMjr2PpDdx23U=@protonmail.com/
[16] https://lwn.net/ml/debian-project/959afc2e-a0ae-bbdc-14bc-3d0b93de8903@debian.community/
[17] https://lwn.net/ml/debian-project/H3wJbOsvVY23EsMR7o0Y20StUbTWCkBFYV5KKlhez1xxWXyPnrd0Sr7Zxr_okyeIuEsADHXRhM-juXVUamswjXU70tCVXEpGIPNHkb4o3jY=@protonmail.com/
[18] https://lwn.net/ml/debian-project/ea-mime-5e072373-6227-653ea4a2@www-2.mailo.com/
[19] https://lwn.net/ml/debian-project/tsllfoamorb.fsf@suchdamage.org/
[20] https://www.debian.org/devel/constitution
[21] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953378#12
[22] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953378
[23] https://lwn.net/Articles/813831/
[24] https://en.wikipedia.org/wiki/Freedom_of_association
[25] https://lwn.net/ml/debian-vote/20200307210304.GA3399625@roeckx.be/
[26] https://lwn.net/Articles/782786/
[27] https://fsfe.org/index.en.html
[28] https://lists.fsfe.org/pipermail/discussion/2019-May/012802.html
[29] https://lists.fsfe.org/pipermail/discussion/2019-May/012740.html
[30] https://fsfe.org/about/team.en.html
[31] https://lists.fsfe.org/pipermail/discussion/2019-May/012760.html
[32] https://lists.fsfe.org/pipermail/discussion/2019-May/012778.html
[33] https://lists.fsfe.org/pipermail/discussion/2019-May/012776.html
[34] https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=04afe8249f895e6e07e2ebb25c286166e40dcbad
[35] http://fedoraplanet.org/
[36] https://lwn.net/Articles/814508/#Comments
[37] https://lwn.net/Articles/814420/
[38] https://www.socallinuxexpo.org/scale/18x
[39] https://lwn.net/Calendar/
[40] https://lwn.net/ml/linux-fsdevel/b506a373-c127-b92e-9824-16e8267fc910@toxicpanda.com/
[41] https://events.linuxfoundation.org/lsfmm/
[42] https://linuxplumbersconf.org/
[43] https://lwn.net/Articles/814420/#Comments
[44] https://lwn.net/Articles/813800/
[45] https://www.opensuse.org/
[46] https://en.opensuse.org/openSUSE:Board
[47] https://en.opensuse.org/openSUSE:Board_election
[48] https://lwn.net/ml/opensuse-project/trinity-317170fa-d68f-403c-8c25-30aee7ca98d2-1579257133244@3c-app-gmx-bap49/
[49] https://lwn.net/Articles/792053/
[50] https://lwn.net/ml/opensuse-project/trinity-3759152d-9c70-4e06-b666-5124dd3ad4fe-1580109123018@3c-app-gmx-bap28/
[51] https://lwn.net/ml/opensuse-project/trinity-d2a17e56-1245-45bd-808b-ff2397b084f2-1581275553522@3c-app-gmx-bs06/
[52] https://lwn.net/ml/opensuse-project/d047ab16-cce7-d644-c86d-3867669c31f5@lasentinelle.mu/
[53] https://lwn.net/ml/opensuse-project/trinity-66c4b99a-b377-4db3-a51f-f2ffc53d8c02-1581452806688@3c-app-gmx-bs51/
[54] https://lwn.net/ml/opensuse-project/nycvar.YFH.7.76.2002271214210.3123@naguvnf.csrvsre.pbz/
[55] https://lwn.net/ml/opensuse-project/4771625.CujlEKmei7@tux.boltz.de.vu/
[56] https://lwn.net/ml/opensuse-project/trinity-23babf9d-3f4a-48cf-9dc9-c1bb74832511-1582903965972@3c-app-gmx-bap23/
[57] https://lwn.net/ml/opensuse-project/r3bkre$cfl$1@ciao.gmane.io/
[58] https://lwn.net/ml/opensuse-project/CAKW7KQSW4s-Od8cLJjZXgigj6bNyQWYLUPzHB9nLkR7vtc40tQ@mail.gmail.com/
[59] https://lwn.net/ml/opensuse-project/1739744.xEoioXefQr@tux.boltz.de.vu/
[60] https://lwn.net/ml/opensuse-project/trinity-cbf67425-7b38-4515-b877-8b81bb084ea1-1583001359544@3c-app-gmx-bs12/
[61] https://lwn.net/ml/opensuse-project/nycvar.YFH.7.76.2003040012550.3212@naguvnf.csrvsre.pbz/
[62] https://lwn.net/ml/opensuse-project/BEAF6273-552C-4A04-952C-038342843016@suse.de/
[63] https://lwn.net/Articles/776324/
[64] https://lwn.net/ml/opensuse-project/1691563.mc3XgJ6J9P@t520.axxite.internal/
[65] https://lwn.net/ml/opensuse-project/8292793.7FSq4rZfRq@tux.boltz.de.vu/
[66] https://lwn.net/ml/opensuse-project/69e1ce9a-c3cb-5dda-0f23-dfb41945e110@dodin.org/
[67] https://lwn.net/ml/opensuse-project/CAKW7KQTt2cPNXBQchGzn3+JGi-WHw_sCnVgNPxZVyfWb6H=QeQ@mail.gmail.com/
[68] https://lwn.net/Articles/813800/#Comments
[69] https://lwn.net/Articles/814389/
[70] https://letsencrypt.org/
[71] https://letsencrypt.org/2020/02/27/one-billion-certs.html
[72] https://letsencrypt.org/docs/caa/
[73] https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591
[74] https://www.abetterinternet.org/
[75] https://github.com/letsencrypt/boulder
[76] https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864
[77] https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591/3
[78] https://cabforum.org/
[79] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179
[80] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c1
[81] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
[82] https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c7
[83] https://cabforum.org/baseline-requirements-documents/
[84] https://bugzilla.mozilla.org/show_bug.cgi?id=1619047
[85] https://checkhost.unboundtest.com/
[86] https://lwn.net/Articles/814389/#Comments
[87] https://lwn.net/Articles/813827/
[88] https://lwn.net/Articles/813172/
[89] https://lwn.net/ml/linux-kernel/CAJfpegtOwyaWpNfjomRVOt8NKqT94O5n4-LOHTR7YZT9fadVHA@mail.gmail.com/
[90] https://lwn.net/ml/linux-kernel/20200303113814.rsqhljkch6tgorpu@ws.net.home/
[91] https://lwn.net/ml/linux-kernel/20200303130347.GA2302029@kroah.com/
[92] https://lwn.net/ml/linux-kernel/CAG48ez3Z2V8J7dpO6t8nw7O2cMJ6z8vwLZXLAoKGH3OnCb-7JQ@mail.gmail.com/
[93] https://lwn.net/Articles/776703/
[94] https://lwn.net/ml/io-uring/20200303235053.16309-1-axboe@kernel.dk/
[95] https://lwn.net/Articles/810495/
[96] https://lwn.net/ml/io-uring/20200304143548.GA407676@localhost/
[97] https://lwn.net/Articles/796868/
[98] https://lwn.net/ml/io-uring/20200304180016.28212-1-axboe@kernel.dk/
[99] https://lwn.net/ml/io-uring/ed5c490f-4faf-afc7-bfab-d58aed061fc6@kernel.dk/
[100] https://lwn.net/Articles/813827/#Comments
[101] https://lwn.net/Articles/813896/
[102] https://lwn.net/Articles/813897/
[103] https://lwn.net/Articles/813896/