'Uncertainty' drives LinkedIn to migrate from CentOS to Azure Linux
- Reference: 1724855288
- News link: https://www.theregister.co.uk/2024/08/28/linkedin_azure_linux/
- Source link:
Ievgen Priadka's post on the LinkedIn Engineering blog, titled [1]Navigating the transition: adopting Azure Linux as LinkedIn's operating system , is the visible sign of what we suspect has been a massive internal engineering effort. It describes some of the changes needed to migrate what the post calls "most of our fleet" from the end-of-life CentOS 7 to [2]Microsoft Azure Linux – the distro that grew out of and replaced its [3]previous internal distro, CBL-Mariner .
This is an important stage in a long process. [4]Microsoft acquired LinkedIn way back in 2016. Even so, as recently as the end of last year, we reported that [5]a move to Azure had been abandoned , which came a few months after it [6]laid off almost 700 LinkedIn staff – the majority in R&D.
[7]
The blog post is over 3,500 words long, so there's quite a lot to chew on – and we're certain that this has been passed through and approved by numerous marketing and management people and scoured of any potentially embarrassing admissions. Some interesting nuggets remain, though. We enjoyed the modest comment that:
However, with the shift to CentOS Stream, users felt uncertain about the project's direction and the timeline for updates. This uncertainty created some concerns about the reliability and support of CentOS as an operating system.
That strikes us as putting it mildly. The way that [8]Red Hat redefined its free distro has caused industry-wide consternation and even outrage, but few would have pegged Microsoft as one of the companies affected. We're also amused that Priadka cites "strong vendor support," defined as "having support from the OS vendor or a reliable support provider," as a motivation. This isn't a shot at Red Hat; it didn't support CentOS anyway. It is, however, praise for the Azure Linux team, and implies that they are on a par with any of the vendors that will [9]give you paid support on CentOS .
There are some interesting technical details in the post too. It seems LinkedIn is running on XFS – also the [10]RHEL default file system , of course — with the notable exception of [11]Hadoop , and so the Azure Linux team had to add XFS support. Some CentOS and actual RHEL is still used in there somewhere.
[12]
[13]
That fits perfectly with using any of the RHELatives. However, the post also mentions that the team developed [14]a tool to aid with deploying via MaaS, which it explicitly defines as Metal as a Service . MaaS is a [15]Canonical service , although it does support other distros – so as well as CentOS, there may have been some Ubuntu in the LinkedIn stack as well.
Some details hint at what we suspect were probably major deployment headaches. LinkedIn was using DKMS to build drivers into its kernels on the fly, which is a common use case and implies that Nvidia cards are involved. However, the Azure cloud insists on signed kernels, so Microsoft has built a repository of signed kernels incorporating all the hardware in use. Although the section about container images is garbled, Microsoft has also built an internal container image repository for Azure Linux to pull images from.
[16]Microsoft freshens up its in-house container Linux, CBL-Mariner
[17]Fujitsu gets ready to eat its own dogfood as company-wide digital transformation project kicks off
[18]Amazon’s Away Teams laid bare: How AWS's hivemind of engineers develop and maintain their internal tech
[19]MS Hotmail servers begin switch from FreeBSD to Win2k
Some of the other information covers things the teams did not do, which is equally informative. Development used to be done on "a full-fledged CentOS desktop VM with a window manager" – but Azure Linux doesn't have a GUI or any kind of desktop, and they haven't added one. They've found a way to connect IDEs running locally to remote Azure Linux VMs, complete with GPUs – over 1,500 of them, in four different geographical regions. As you would expect, it mentions Microsoft's own VS Code, but also [20]JetBrains' IntelliJ. LinkedIn was founded back in 2003, and any technology stack that's 21-plus years old is doubtless messy and contains a lot of different vendors' parts.
Microsoft has a long history of "eating its own dogfood," even when the exercise of moving acquisitions to its own stack has been painful. It [21]acquired Hotmail way back in 1997, but even two years later, as the still-young Register [22]was reporting in 1999 , it still ran on a mixture of Apache on FreeBSD, with Sun Solaris-powered database servers.
[23]
Back then, Windows NT 4 Server simply wasn't up to the task. It wasn't until Windows Server 2000 that it managed to [24]begin the transition , as The Reg reported in 2000. Microsoft even wrote up a detailed report, [25]Converting a UNIX .COM Site to Windows , describing the process. As late as 2003, the replacement [26]was still in progress .
The result of this major undertaking was significant improvements in the abilities, compatibility, and also stability of Windows Server. "Dogfooding" is an important effort. This vulture has in the past been a member of staff at two of the big three enterprise Linux vendors, and I speak from personal experience when I say that those companies relied heavily on external third-party vendors for their IT infrastructure – and, as far as I know, still do.
Microsoft is doing the right thing here, shouldering the burden, and considerable cost, of moving its own internal infrastructure to its own products. This can only be good for Azure Linux, and indeed, for Azure in general. Its fairly obscure and [27]limited little Linux distro is growing up fast, and we'd bet on major functional improvements coming in a future release.
[28]
And, indirectly, it's all thanks to Red Hat. ®
Get our [29]Tech Resources
[1] https://www.linkedin.com/blog/engineering/architecture/navigating-the-transition-adopting-azure-linux-as-linkedins-operatingsystem
[2] https://www.theregister.com/2023/05/26/microsoft_azure_linux_container/
[3] https://www.theregister.com/2021/08/18/cbl-mariner/
[4] https://www.theregister.com/2016/06/13/microsoft_buys_linkedin_26bn/
[5] https://www.theregister.com/2023/12/14/linkedin_abandons_migration_to_microsoft/
[6] https://www.theregister.com/2023/10/16/linkedin_layoffs/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zs9Jo-FDMakn8JOTZBNGyAAAABI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[8] https://www.theregister.com/2020/12/09/centos_red_hat/
[9] https://www.theregister.com/2024/06/06/lansweeper_centos/
[10] https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/managing_file_systems/overview-of-available-file-systems_managing-file-systems
[11] https://hadoop.apache.org/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zs9Jo-FDMakn8JOTZBNGyAAAABI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zs9Jo-FDMakn8JOTZBNGyAAAABI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://github.com/microsoft/azurelinux/blob/3.0-dev/toolkit/tools/imagecustomizer/README.md
[15] https://maas.io/
[16] https://www.theregister.com/2022/03/17/microsoft_updates_its_inhouse_linux/
[17] https://www.theregister.com/2020/10/05/fujitsu_project_fujitra_transformation/
[18] https://www.theregister.com/2019/05/14/amazons_away_teams/
[19] https://www.theregister.com/2000/08/01/ms_hotmail_servers_begin_switch/
[20] https://www.theregister.com/Tag/JetBrains/
[21] https://web.archive.org/web/20090803010737/http://www.indianexpress.com/res/web/pIe/ie/daily/19980107/00750984.html
[22] https://www.theregister.com/1999/04/19/microsoftowned_email_service_runs/
[23] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zs9Jo-FDMakn8JOTZBNGyAAAABI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[24] https://www.theregister.com/2000/08/01/ms_hotmail_servers_begin_switch/
[25] https://web.archive.org/web/20021021164226/http://www.securityoffice.net/mssecrets/hotmail.html
[26] https://jimbojones.livejournal.com/23143.html
[27] https://www.theregister.com/2022/03/17/microsoft_updates_its_inhouse_linux/
[28] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zs9Jo-FDMakn8JOTZBNGyAAAABI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[29] https://whitepapers.theregister.com/
Giving them some credit, MS has been doing that for decades. Once they built DOS using Xenix. Windows NT has grown and matured beyond recognition. I dare say that they’ve had to use and fix Office a lot. Their dev tools are pretty good. Under the current management they seem to have been very open to all sorts of things and played relatively nicely.
They could do pretty well if they really put effort into a Linux distribution.
When one considers what companies might put up some stiff competition against RedHat / IBM, MS is probably one of the likely candidates whom may even be acceptable to many in the OSS community.
I’ve sometimes wondered if an MS / Ubuntu merger could ever happen, and what that might produce.
"If you want to improve your company product make your company actually use it in real world conditions"
This makes me wonder about their patch teams. Do they really put up with their own product? Or maybe they do and have no experience of how much better it could be. Perhaps they should be made to run something like Debian as their daily drivers for a couple of calendar months and then switch back to be faced with the accumulated junk they'll have to catch up on.
Microsoft's in-house Linux.
Microsoft data centers now powered by Balmer turning in his grave (I know he's not dead but I always assumed he slept in a coffin anyway)
Re: Microsoft's in-house Linux.
Time travelling energy supply? Taking energy from a future spectre with an excess of angular momentum? Well given that many companies are all about money now, stuff the future, your description is highly plausible.
Better watch MS carefully. If they come up with both time travel and infinite energy, their stock is going up.
Re: Microsoft's in-house Linux.
Or have they already will be invented time travel?
(Reference to the much missed Douglas Adams for those not familiar with the grammatical challenges time travel presents).
Why so complicated?
I'd love to know more details. Surely it's a website running with a database behind it and some back-end apps. If they're not OS agnostic (even within the Linux sphere) I wonder why? And does that show poor planning on their part? Or is it just a case of something that grew organically and so future-proofing was never "baked in".
Re: Why so complicated?
The Linux world doesn’t make it especially easy. Even the field of containerisation is somewhat fragmented, with more being invented every day.
The whole hoohaa about CentOS is a good indicator that switching Linuxes is non trivial for a lot of people.
Re: Why so complicated?
It sounds like the major parts were the lack of GUI, requiring a remote development environment to be set up, and the custom(ish) filesystem.
Everything else appears to have been relatively trivial - at least, far easier than moving from one version of Exchange to another.
[Microsoft's in-house professional networking site is moving to Microsoft's in-house Linux.]
If you want to improve your company product make your company actually use it in real world conditions