Even Facebook struggles: Zuck's titanic database upgrade hits numerous legacy software bergs
- Reference: 1627023672
- News link: https://www.theregister.co.uk/2021/07/23/facebook_mysql_upgrade_legacy_issues/
- Source link:
A [1]post from the social network’s engineering team reveals that Facebook's last MySQL upgrade – to version 5.6 – took "more than a year".
Moving to version 8.0 has taken "a few years so far".
[2]
Some of the reasons for the slow rollout will sound familiar – such as continued work on legacy software, even as new infrastructure was being implemented.
[3]
[4]
Facebook had custom code – over 1700 patches in its in-house branch of MySQL 5.6. Even as it ported them to 8.0, the company was adding more custom features to version 5.6 that also needed to be dragged into the future.
[5]Facebook gardening group triumphs over slapdash Zuck censorbots
[6]Facebook pulls plug on mind-reading neural interface that restored a user's speech
[7]Pakistan bans TikTok, for the fourth time
Other parts of Facebook legacy code wouldn't work with MySQL 8.0, and therefore "required a deprecation and migration path forward".
"Upgrading from 5.6 to 8.0 skipped over 5.7 entirely. APIs that were active in 5.6 would have been deprecated in 5.7 and possibly removed in 8.0, requiring us to update any application using the now-removed APIs," wrote Facebook software engineer Herman Lee and production engineering manager Pradeep Nayak.
Complicating matters further was Facebook's use of the [8]MyRocks project, an effort that lets MySQL use RocksDB as its storage backend, and which Facebook was developing during its migration to MySQL 5.6.
[9]
Jumping from version 5.6 to 8,0 – and skipping version 5.7 – meant Facebook could not upgrade servers in place, so had to use logical dump and restore to build a new server.
"However, for very large mysqld instances, this can take many days on a live production server and this fragile process will likely be interrupted before it can complete. For these large instances, we had to modify our backup and restore systems to handle the rebuild," Lee and Nayak wrote.
Facebook tracked the porting process of its custom patches using … wait for it … spreadsheets – and recorded discrepancies as they went.
[10]
"Discrepancies on porting status would inevitably arise due to the large number of patches we needed to sift through and these notes helped us resolve them," the pair wrote.
For migration, Facebook hatched a plan to create secondary instances of MySQL databases, then copy data from version 5.6. Once that was done, the secondary version 8.0 instance would be copied to the primary.
To get that done at Facebook scale, "we needed to build new software infrastructure", the post states matter-of-factly.
We had to modify our backup and restore systems to handle the rebuild
Other issues that cropped up included increased memory usage, file format incompatibilities, and conflicts between newly adopted keywords and "popular table column names and aliases used in application queries".
The tone used by the post's authors suggests the project has made steady progress, with glitches to be overcome rather than setbacks that derailed progress or left project timelines in tatters.
The post ends by saying the apps Facebook has moved are benefiting from their new database environment.
"Overall, the new version greatly expands on what we can do with MySQL @ Facebook," Lee and Nayak conclude.
Whether that's good for the rest of us is, as ever, the question Facebook struggles to answer. ®
Get our [11]Tech Resources
[1] https://engineering.fb.com/2021/07/22/data-infrastructure/mysql/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YPqTPYS4iJ2ZVLZAlfAGHQAAAMw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPqTPYS4iJ2ZVLZAlfAGHQAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPqTPYS4iJ2ZVLZAlfAGHQAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/07/22/ny_gardeners_facebook_hoe_concerns/
[6] https://www.theregister.com/2021/07/15/facebook_dumps_mindreading_neural_interface/
[7] https://www.theregister.com/2021/07/21/pakistan_fourth_tiktok_ban/
[8] http://myrocks.io/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YPqTPYS4iJ2ZVLZAlfAGHQAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YPqTPYS4iJ2ZVLZAlfAGHQAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
Re: I would like to thank MySQL..
No doubt some Facebook dev years ago said, "it's only a small tweak, that won't cause any problems"...
Didn't know they were still using MySQL
When Facebook was originally built - early 2004 - it was a fairly basic PHP application which used MySQL for all storage (user profiles etc). I don't think it used anything else except maybe memcached or possibly Redis for caching.
I doubt MySQL would scale to hold all of the user data that it now needs to. What are they still storing in MySQL and what other storage technologies are they using? There must be an absolute myriad of data especially given that you can pretty much go back to when you opened your account and find posts.
People are quick to criticise Facebook but the way in which they've scaled that is nothing short of incredible.
Re: Didn't know they were still using MySQL
MySQL does scale pretty well in read-heavy environments. Unfortunately, it's shit™ for migrations due to the way tables are managed: "simple" changes to tables can take forever as they're migrated to temporary ones and back. And that forever can take even longer on an ever growing system.
If you have a system running on a DB you need to have a migration strategy in place. This might even include no migrations to new versions. But the real problem here seems to be running custom extensions, which obviously seemed a good idea at the time…
They could switch to Postgres and use FDW (Foreign Data Wrappers) to hook up the legacy databases while they work on a data migration strategy (inplace updates are not always possible) that minimises the dump and load problems.
Or they could ask Oracle for help…
Re: Didn't know they were still using MySQL
But the real problem here seems to be running custom extensions . Agreed.
Migrating even small or moderately sized DBs can be a PITA, but when you custom code "over 1700 patches" you really are creating a massive rod for your own back.
Probably it started small and snowballed (bazza^ "it's only a small tweak, that won't cause any problems"). Like firms that let all and sundry write MS Office VBA mods for convenience, and before they realise they're dependant on stuff they have no control over.
Facebook's key motto...
... is apparently now [1]move fast with stable infrastructure .
And yet their unstable infrastructure seems to be holding them back. It's almost as if it were all just empty sloganising.
[1] https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-break-things-isnt-how-we-operate-anymore/
Re: Facebook's key motto...
As I said in my first post people are quick to criticise Facebook. I bet none of those people would have the skills to create something that went from a bedroom project to being able to concurrently support billions of users and hold nearly 2 decades worth of data for all those people.
Generally speaking Facebook is very stable and that is _incredibly_ hard to achieve when you have over 2.8 billion users. Even if only 50% of them were active that's way more than your average application. The sort of people who criticise them generally don't even have 1 million users on their own platforms.
I'm yet to see an example of anybody who has taken something from an amateur project and scale it to what it has become. Granted it's somewhat annoying when the use-case is for the Karens of the world to share posts about how disappointed they are at their local school... but the tech and processes behind it is nonetheless very impressive.
Re: Facebook's key motto...
I'm criticising FB and I'm confident I can take a project from embryonic to mega-buster. I very much doubt I'm alone in that on these forums.
Of course, it's entirely the case that after two decades those that created a relatively future proof and scalable solution would have had it ballsed up by PHBs and their replacements that, without doubt, know better.
I'm not saying it's possible without glitches, it isn't, as you don't know what will develop in the tools used over that period. But there's best practice and the path of least resistance.
It's okay to "start in the bedroom" not knowing that growth will be epic, but there's no excuse for starting out on the wrong foot. It just seems that FB let it run away a bit too much.
Easy solution
DROP DATABASE facebook;
I would like to thank MySQL..
.. for making life difficult for Facebook.
Well done.