Red Hat strikes a crushing blow against RHEL downstreams
- Reference: 1687537811
- News link: https://www.theregister.co.uk/2023/06/23/red_hat_centos_move/
- Source link:
A superficially modest [1]blog post from a senior Hatter announces that going forward, the company will only publish the source code of its CentOS Stream product to the world. In other words, only paying customers will be able to obtain the source code to Red Hat Enterprise Linux… And under the terms of their contracts with the Hat, that means that they can't publish it.
In the opinion of the Reg FOSS Desk, the blog post itself is so full of corporate language that it borders on obfuscatory. However, we've contacted the Red Hat press office, and the company confirmed that the release does say what we got out from reading between the lines. This is very bad news for downstream projects which rebuild the RHEL source code to produce compatible distributions, [2]such as AlmaLinux, Rocky Linux, EuroLinux, and Oracle Unbreakable Linux.
[3]
The core difference is that CentOS Stream is upstream of RHEL: it's what will become the next point release of RHEL. At risk of sounding uncharitable, it's a sort of continuous rolling beta of the next version of RHEL. Alma, Rocky, and so on, and the former CentOS Linux, were downstream of RHEL: they were rebuilds from the same source code, guaranteeing perfect compatibility. So you could run one of the rebuilds, without paying Red Hat anything, while using the same drivers and getting perfect compatibility with RHEL apps.
[4]
[5]
You don't get that with CentOS Stream: It's a preview of the future of RHEL. Which is handy if you are a partner company developing products or drivers to run on RHEL, or you're a customer who wants to know what's going to come next. It's much less useful if you just want to run RHEL without paying. Or, of course, if you want to build your own copy of RHEL. We suspect that the wider RHEL user community doesn't care about Stream very much, and that may be a motivation behind the latest move.
In various forums online, there are outcries from users of downstream distros… just as there was when [6]the Hat cancelled CentOS Linux a few years ago. Once again, people are talking about betrayal of trust, violating the GPL, and so on. However, as far as we can see, the Hat is acting perfectly in accordance with the terms of the GPL, which only requires them to make source code available to people using the binaries built from them : in other words, to its paying customers. The key point being is that to obtain those binaries, customers – as well as developers on free accounts – must agree to a license agreement and are under the terms of a contract, which overrides the GPL license of the code itself.
[7]
In a way, this could be interpreted as a logical continuation of the move made when the company [8]brought CentOS in-house back in 2014 . That move legitimized this particular one of the several extant RHEL rebuilds, which resulted in the rest of them essentially shutting down their efforts — except of course for Oracle, which has deep pockets to fund Oracle Linux, complete with cheaper enterprise support contracts, an enhanced Btrfs-compatible kernel and so on.
Having given the move time to successfully eliminate most of the clones, Red Hat then killed off its own official free version of its paid-for flagship product. Instead, it switched to offering a free test version, an announcement which it made accompanied with lots of positive-sounding language about community involvement, and so on. In fact, what it was really doing was cutting off those that might be seen, from its perspective, as a bunch of freeloaders. The move was accompanied with [9]free production deployment of RHEL for developers – but only up to 16 machines.
Way forward
The door has not been completely closed. If we understand it correctly, in effect, Stream is periodically resynchronized with RHEL when there is a new major release. So, when RHEL 11.0 is released, Stream will briefly be in sync with it — which means that downstream distros could grab a copy of the code at that exact point in time, and build a new version compatible with that point-zero release of RHEL. The problem for the downstreams is that from that point on, they won't be able to get ahold of usable source code of each subsequent point release and the various ongoing updates.
Some commentators are pointing out that it's possible to sign up for a [10]free Red Hat Developer account, and obtain the source code legitimately that way. This is perfectly true, but the problem is that the license agreement that you have to sign to get that account prevents you from redistributing the software.
So although the downstream distros could still get hold of the software source code, they can't actually use it. In principle, if they make substantial modifications, they can share those, but the whole raison d'être of RHEL-compatible distros is to avoid major changes and so retain "bug-for-bug compatibility."
[11]
Of course, they could take a "publish and be damned" attitude and do it anyway. At best, the likely result is immediate cancellation of their subscription and account. That could work but will result in a cat-and-mouse game: downstream distributors continually opening new free developer accounts, and the Hat potentially retaliating by blueprinting downloads and stomping on violators' accounts. It would not be a sustainable model.
At worst, though, they could face potentially getting sued into oblivion.
In summary, it will still be possible to obtain the source code, via several different routes — although some of these come with very serious restrictions attached. For now, the official reactions of both [12]AlmaLinux and [13]Rocky Linux are guardedly optimistic, although there are signs of concern in the [14]discussion on the Rocky Linux forum.
Back in 2011, Red Hat [15]changed the way that it distributed its source code packages in a way that certainly looked as if it was specifically intended to make life difficult for the rebuilds. We don't know the company's motivations, and it's certainly not going to tell us, but perhaps that move was not successful enough and thus led to actually bringing CentOS in house.
As [16]FOSS desk said when CentOS Stream 9 came out , we feel that Red Hat's key mistake was adopting CentOS Linux in the first place. The move endorsed and legitimized a free competitor to the company's own paid-for, commercial product. (And one that was struggling at that point, which ought to have been no concern whatsoever to Red Hat.) If the plan was to inconvenience Oracle in some way, it failed, but it surely significantly cut into RHEL's sales.
Back then, the downstream distributors managed to find ways around that move, and it is perfectly possible that they will also be able to find ways around this one – but it will make continuing substantially more difficult. It's possible that the Hat is perturbed by the success of the new generation of rebuilds. While the bodies behind both Rocky and AlmaLinux are basically non-profits – Rocky is a public benefit corporation founded and owned by Greg Kurtzer – they are doing well.
For example, just last week, NASA [17]licensed Rocky Linux for its internal use. CERN and Fermilab meanwhile [18]use AlmaLinux.
The Big Purple Hat is presenting the move as not a particularly big deal – as if it were simply designed to boost takeup of Stream, while in fact it actually looks like a concerted attack upon the thriving new ecosystem of rebuilds that resulted from the cancellation of CentOS Linux.
TL;DR?
The timeline is long and involved, and if you are getting confused by now, we don't blame you in the least. The main events, and this jaded old vulture's interpretations of them, happened as follows.
1994: Red Hat Linux released.
2002: the first version of RHEL, version 2.1 – yes, [19]really – released, based on RHL 7.2.
2003: Red Hat Linux 9 released, after which development stopped.
2006: First Oracle Linux released, based on RHEL 4.5 and with the same version number.
2011: Red Hat restructures its source releases, making life more difficult for the RHEL rebuilds.
2014: Red Hat adopts CentOS Linux, making it official.
2020: Red Hat ends development of CentOS Linux, switching to CentOS Stream.
2021: First release of AlmaLinux follows a few months later, followed by Rocky Linux a few months after that.
2023: Red Hat stops making RHEL source code available to non-customers.
Meanwhile…
The job cuts which [20]struck both Kyndryl and [21]Red Hat earlier this year in the United States are now starting here in central Europe, and several former colleagues and some personal friends of the Reg FOSS Desk were laid off this week, from the Hat as well as from other arms of Big Blue.
[22]Microsoft has made Azure Linux generally available. Repeat, Azure Linux
[23]Red Hat releases RHEL 9.2 to customers, with buffet of rebuilds for the rest of us
[24]CentOS Stream 9: Understanding the new Red Hat OS release for non-Red-Hat-type people
[25]CentOS project changes focus, no more rebuild of Red Hat Enterprise Linux – you'll have to flow with the Stream
If this move does in fact result in the demise of Alma, Rocky et al , companies and communities which hundreds of people have just spent several years building, the end result may be a boost to IBM's bottom line, but it will also mean public opinion turning even further against the company. Since it was founded some 30 years ago, Red Hat permitted clones and rebuilds of its operating systems, back to the early days of Red Hat Linux. That is where [26]Mandrake Linux started out for example: as a rebuild of Red Hat Linux with the KDE desktop, at a point in time when the Hat felt that Qt's license precluded including it. Summarily killing them all off is not a move that is going to win the Big Purple Hat any friends… Except possibly among IBM shareholders.
What does this mean for Fedora?
Fedora users, and indeed contributors, need not fear – although some serious [27]discontentment is showing on the Fedora-Devel mailing list.
Fedora is upstream of RHEL: Software developed and tested in Fedora flows into CentOS Stream, whence it flows into RHEL. RHEL is in fact what pays for much of the work that goes into Fedora. If a cynical but not entirely unfair summary of CentOS Stream is that it's a rolling beta of the next RHEL point release, Fedora is a sort of rolling alpha of the next RHEL major release.
So while RHEL depends on Fedora technologically, the reverse is not true; principally, Fedora only depends on RHEL financially.
There are server versions of Fedora, and Red Hat users wanting a free RHEL relative can use those however they wish. The differences are that they're based on newer code, so they are far from identical to RHEL and never will be… and of course there are no stable long-term support releases of Fedora. ®
Get our [28]Tech Resources
[1] https://www.redhat.com/en/blog/furthering-evolution-centos-stream
[2] https://www.theregister.com/2023/05/19/rhel_92/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZJYV-9JgVU8mKE89EfaRnAAAANA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJYV-9JgVU8mKE89EfaRnAAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZJYV-9JgVU8mKE89EfaRnAAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2020/12/09/centos_red_hat/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJYV-9JgVU8mKE89EfaRnAAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2014/01/08/red_hat_to_team_up_with_communitybased_rhel_lookalike_centos/
[9] https://www.theregister.com/2021/01/20/red_hat_amends_developer_license/
[10] https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZJYV-9JgVU8mKE89EfaRnAAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://almalinux.org/blog/impact-of-rhel-changes/
[13] https://rockylinux.org/news/2023-06-22-press-release/
[14] https://forums.rockylinux.org/t/has-red-hat-just-killed-rocky-linux/10378/17
[15] https://www.theregister.com/2011/03/04/red_hat_twarts_oracle_and_novell_with_change_to_source_code_packaging/
[16] https://www.theregister.com/2021/12/10/centos_stream_9/
[17] https://sam.gov/opp/2e0365ce1e3c4c179b50fb15573d68e4/view
[18] https://www.theregister.com/2022/12/08/cern_fermilab_almalinux/
[19] https://www.redhat.com/en/blog/rhelvolution-brief-history-red-hat-enterprise-linux-releases-early-days-rhel-5
[20] https://www.theregister.com/2023/04/04/ibm_and_kyndryl_are_cutting/
[21] https://www.theregister.com/2023/04/28/red_hat_layoffs_union/
[22] https://www.theregister.com/2023/05/26/microsoft_azure_linux_container/
[23] https://www.theregister.com/2023/05/19/rhel_92/
[24] https://www.theregister.com/2021/12/10/centos_stream_9/
[25] https://www.theregister.com/2020/12/09/centos_red_hat/
[26] https://www.theregister.com/2022/02/15/comparing_the_descendants_of_mandrake/
[27] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/PPM7P7IZYJZRVHS6LCFHVVIJ6FGWUPD4/
[28] https://whitepapers.theregister.com/
Re: GPL violation
Hello Debian!
Re: GPL violation
Agree. The TL;DR is "get stuffed Redhat".
Re: GPL violation
I think mostly true however to stay true to the spirit of the license if a customer wanted to redistribute the source they got from Red Hat they'd have to strip it of any trademark type stuff first, if that is done then really nothing Red Hat/IBM can do(provided the underlying license for the source package is open source).
Re: GPL violation
The question comes from who is going to pay for those lawyers? IBM have very deep pockets and an in house legal team, so it would take someone with a lot of money and a vested interested to try and sue them for breach of the GPL. So from the existing RHEL compatible distros I only see Oracle as having deep enough pockets to try and take IBM to court over it.
Re: GPL violation
When the wording of the GPL is clear, plain, and obvious, and the violation of the contract is clear and direct, even the very limited amount of legal counsel the FSF can afford from donations should suffice.
Re: recipients of their binaries
What if I obtain the source, via whatever means (e.g. a legit free dev account) and compile the binaries myself? Given the right compiler (er, gcc probably) they will be indistinguishable from "the real thing".
-A.
Re: GPL violation
Precisely. This point was missing from the article. Red Hat will indeed not be in compliance with the GPL when they implement this policy.
If they want a free UNIX-like operating system that they can turn into a proprietary product, they will have to do what Apple did to make OS X. So long Red Hat Linux, hello Red Hat BSD.
Re: GPL violation
It's pretty sneaky.
Under the GPL, Red Hat doesn't have to give you the sources unless you get their binaries.
You don't get the Red Hat binaries unless you agree to their terms, which includes not giving out sources. Those terms *override* the GPL.
There is no GPL violation.
If you distribute the sources (which includes non-GPL code) you violate the Red Hat agreement. Red Hat can take legal action and/or refuse to sell you binaries.
If you think IBM won't take legal action, I refer you to "IBM vs SCO". IBM probably has more lawyers than the Reg has readers. There is not a hope in hell that FSF can out-lawyer IBM.
Fedora Dependency
I've never understood how people so easily gloss over Fedora's dependency on Red Hat and always boost the, "they are dependent on us!" narrative. When you're funding and resources come from a single place, and a large part of your human developers also come from and get paid by that same single place, you are dependent. Anything else smacks of naivety and/or adhering to a set of talking points handed down from corporate. 5-6 years ago, this chicken and egg dependency debate still might have been more of an academic argument or sniping between distro fans in a forum flame war, but boundaries once thought fixed sure are suddenly moving in an ever more restrictive manner.
I am surprised that IBM took this long
to find a way of effectively making Red Hat closed source.
One result is that I shall move servers to Debian both mine and those of my customers - some of whom have done things like: Red Hat for production machines and Rocky/something for development and standby machines.
It is tempting to say "just grab the source, it is GPL" but IBM is a law firm that happens to sell computers so even if I am in the right they will make it very expensive for me to do so ... much the same as a patent troll does "pay $40k for a license or risk $100k legal costs".
Shame: Red Hat was nice while it lasted.
Re: I am surprised that IBM took this long
" I am surprised that IBM took this long...to find a way of effectively making Red Hat closed source"
Indeed, and it is a very nasty thing to do so I hope that this new policy gets challenged in the courts.
While there is the synchronisation loophole that Liam pointed when a major release has taken place, I suspect this move will also result in a shift way from Red Hat and towards other distributions, e.g. SUSE Linux Enterprise Server and others.
From the GPL ( https://www.gnu.org/licenses/gpl-3.0.html ):
"If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term."
"Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License."
AFAICS that means that the only code on which RH can impose these new conditions is code entirely created by RH or a 3rd party which does not derive from GPL code. All other code receives GPL terms from upstream.
Also "if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code".
I wonder what effect this will have on projects receiving contributions from RH or even individual contributions RH employees.
From the GPL ( https://www.gnu.org/licenses/gpl-3.0.html ):
"If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term."
That was my understanding, but look at what it is stated to mean - as opposed to the presumed intent:
“The Program” refers to any copyrightable work licensed under this License.
The subscription agreement with Red Hat is not under the term of the GPL, indeed it is given before and as prerequisite to getting the licensed material, ergo it can't be construed as part of "The Program". External restrictions are not covered by that clause.
Sure, it's not in the spirit of the GPL but it looks to me as if they have a prima facie case. It's one of the risks of producing a licence so long and complex in the first place: that you may consider the FSF "one of the good guys" doesn't change that peril.
"... the only code on which RH can impose these new conditions is code entirely created by RH"
Does that include systemd?
If so that could be the spur to get distros to dump that POS.
Win/Win.
Doesn’t this mean Red Hat are potentially getting sued for GPL violation by Oracle? I mean, Oracle contribute to the Linux kernel so should have standing?
Oracle has a distro based on RHEL but I suppose the two have a financial understanding which covers this and, presumably this term will be imposed on Oracle's Linux customers. However RH have been kernel contributors. As I pointed out in another comment GPL ensures that those receiving GPLed source code get their permissions from upstream. What happens when RH themselves are the upstream for a particular snatch of source?
Well, who knows how that is now. At one point when OEL started up, it was found that there were typos in their SRPMs that had come directly from the CentOS version of the SRPM so it was obvious where they came from.
"Oracle has a distro based on RHEL but I suppose the two have a financial understanding which covers this" – and Oracle will welcome Red Hat's getting rid of the freeloader competition just as well.
Nothing about the GPL requires anyone to put the source code changes they make in a publicly accessible location, only that they cannot distribute binaries compiled from those changes without also giving the recipients of those binaries the potential to access the changes. RedHat are well within their rights to withhold the GPL-covered code they've written until their customers ask for it, when they have no choice but to provide it one way or another and they can't stop their customers doing with it whatever they like, except that they'll be restricted by the GPL as RedHat were.
What is there to sue for ? The kernel source is freely available from the community at kernel.org, always has been, always will be.
The GPL (v2) requires that the person who provides a binary based on a "derived work" of a GPL-licensed software must also provide the corresponding source code.
Such a shame they did not pull systemd from the world's other distros while closing the door
> And under the terms of their contracts with the Hat, that means that they can't publish it
This is allowed by the GPL. However the worst Red Hat can do is terminate their contract with the one who published it. They can't i.e sue them.
What this probably means though is that downstream projects are going to have to create bots to sign up to the free RHEL developer accounts for access. Which has a knock on of:
1) People like me who hates being tied to a server and simply mirror the entire repo for offline backup. RH will probably see me as a risk and will put download restrictions on me.
2) RH will simply terminate the free developer access programme.
Commercially: I can't see why companies would jump through these hoops. There are loads of companies who offer equal Linux support; just look around rather than sticking to that 90's style inertia (as charming and nostalgic as it is).
Community: Do people *really* want these guys dictating the direction of Linux via Systemd, Wayland, NetworkManager, etc, etc?
Welcome to IBMHEL.
It was only going to be a question of when, not if, the old Big Blue started squeezing revenue from anywhere it could find.
Ouch, Ubuntu here we come ….. So glad we held back jumping from C7 into the unknown!
Springdale Linux
Springdale Linux formerly PUIAS Linux does a rebuild of RHEL. Springdale is mainly for internal use at Princeton and the IAS, but the University IT staff who work on it chuck the binaries over the wall and have been doing for some decades.
https://springdale.math.ias.edu/
They have their thinking caps on...
https://groups.google.com/g/springdale-users/c/53hFsR7oLEQ
So three times the rug has been pulled part way through a planned support period without warning or time to adjust budgets. CERN, FermiLab and various University project staff will be most amused I'm sure.
Icon: for those who have been made redundant and for those having to make guesses...
The Software Freedom Conservancy's response
[1]A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model by Bradley M. Kuhn
[1] https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/
It's easy for end users to say "jump to Debian/Ubuntu/whatever". It's entirely different for commercial organizations to jump ship. In some cases, 3rd party vendors only support their products on specific releases of RHEL. Heck, I have seen one RHEL7-based product that doesn't even support RHEL8 yet. These are EXPENSIVE products and they may never support non-RHEL installations. And they may not even work even if you could install them. And sometimes there are no viable alternatives.
Welcome to the real world :(
So sort of like the mainframe was at one time? How very IBM. How's IB-Hat's mainframe business doing these days by the way?
Not very developer friendly
I have a few minor open source projects in several languages which I distribute via the usual avenues for such things. Before I release a new version I run an extensive automated test suit run on roughly a dozen different distros (including BSD) on x86 and ARM.
One of these test platforms has up until now been a Red Hat derivative. At first it was Centos, and later AlmaLinux.
The point of testing on Centos or AlmaLinux was to look for problems that people using RHEL may have and fix them before release. If AlmaLinux have to stop doing updates and get out of sync with RHEL then there's no point to running tests on it anymore. Fedora isn't a substitute since it is not the same thing as the current RHEL version.
I'm not going to either sign up for a RHEL license or a RHEL developers' license. All signing a developer agreement would do is give Red Hat greater ability to sue me for some incomprehensible reason in a foreign court. Why would I want to do that?
If AlmaLinux can't work around the current problems then I'll simply drop testing on RHEL derivatives. I don't imagine that Red Hat / IBM will even notice, but over time more developers may come to the same conclusion that I have and Red Hat may find that their platform gradually becomes the less reliable one on which to run actual applications.
For now though I'll wait to see what AlmaLinux can come up with.
GPL violation
Although Red Hat only need to supply source code to direct recipients of their binaries they are required to supply that source code with a GPL license - not GPL with restrictions. By tacking extra restrictions onto the license Red Hat are violating the GPL and lose their license to distribute GPL code for which they do not own the copyright or have some extra license from the copyright holders to distribute with a more restrictive licence.
Red Hat should expect letters from lawyers reminding them of their obligations.