Poor Meta. Technical debt and user training made its exabyte-scale data migration tricky
(2023/01/27)
- Reference: 1674800892
- News link: https://www.theregister.co.uk/2023/01/27/meta_tulip_technical_debt/
- Source link:
Here’s one from the “welcome to the real world, kids, we have no sympathy for your plight” files: social media giant Meta’s engineering team has bemoaned the complexity of migrating from legacy technology.
In a Thursday [1]post detailing migration of exabyte-scale data stores to new schemas, a quartet of Meta software engineers offered the following insight into their work.
Migrations are hard. Moreover, they become much harder at Meta because of:
Technical debt: Systems have been built over years and have various levels of dependencies and deep integrations with other systems.
Nontechnical (soft) aspects: Walking users through the migration process with minimum friction is a fine art that needs to be honed over time and is unique to every migration.
Fellas, we’re going to let you in on a secret: everyone gets technical debt, and everyone has trouble educating users about new systems.
Meta is not special. It is not a beautiful and unique snowflake. It is the same sort of decaying collection of cobbled-together tech that every other organisation accrues over time.
In this case, the decrepit tech was “numerous heterogeneous services, such as warehouse data storage and various real-time systems, that make up Meta’s data platform — all exchanging large amounts of data among themselves as they communicate via service APIs.”
[2]
As Meta [3]detailed in December 2022, those systems struggled to scale as the data-harvesting giant built more AI workloads that needed to access data from diverse sources.
[4]
[5]
Improved data logging and serialization was the answer, so that data could describe itself more effectively and therefore be more easily ingested by diverse applications.
Meta built a system called “Tulip” to sort that out. And was chuffed that the formats it used required 40 percent to 85 percent fewer bytes and uses 50 percent to 90 percent fewer CPU cycles.
[6]Carmack quits Meta, brands it inefficient and unprepared for competition
[7]Unlucky for some: Meta chops 13% of global workforce
[8]Meta wants to sweat its servers for longer – at a cost of $60b
[9]Shareholders slam Zuckerberg's 'terrifying' $100b+ Metaverse experiment
As Meta’s Thursday post explains, Tulip may have been top tech but making it work was hard, not least because the social media giant employed over 30,000 logging schemas.
Across the four-year effort to adopt Tulip, Meta engineers found some data wasn’t able to be easily ingested or converted, or that doing so was computationally expensive. Some tools designed to ease migration created problems as they ran, so engineers created rate limiters so that issues didn’t snowball.
[10]
And then there were those pesky users, whose role planting Tulip in Meta’s tech garden necessitated the creation of a migration guide, an instructional video, plus a support team.
“Making huge bets such as the transformation of serialization formats across the entire data platform is challenging in the short term, but it offers long-term benefits and leads to evolution over time,” the post winds up.
“Designing and architecting solutions that are cognizant of both the technical as well as nontechnical aspects of performing a migration at this scale are important for success,” the post adds. “We hope that we have been able to provide a glimpse of the challenges we faced and solutions we used during this process.”
[11]
Meta’s four engineers probably have offered useful insights for those who face similar data-wrangling challenges. The rest of you who have lived through legacy migrations? Maybe less so.
And for everyone else, the insight here is that Met has become more efficient at wielding exabytes of data. Much of it gathered from, and about, you. ®
Get our [12]Tech Resources
[1] https://engineering.fb.com/2023/01/26/data-infrastructure/tulip-modernizing-metas-data-platform/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://engineering.fb.com/2022/11/09/developer-tools/tulip-schematizing-metas-data-platform/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2022/12/19/john_carmack_quits_meta/
[7] https://www.theregister.com/2022/11/09/meta_redundancies_mark_zuckerberg/
[8] https://www.theregister.com/2022/10/27/meta_q3_2022/
[9] https://www.theregister.com/2022/10/24/shareholders_to_meta_cut_staff/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
In a Thursday [1]post detailing migration of exabyte-scale data stores to new schemas, a quartet of Meta software engineers offered the following insight into their work.
Migrations are hard. Moreover, they become much harder at Meta because of:
Technical debt: Systems have been built over years and have various levels of dependencies and deep integrations with other systems.
Nontechnical (soft) aspects: Walking users through the migration process with minimum friction is a fine art that needs to be honed over time and is unique to every migration.
Fellas, we’re going to let you in on a secret: everyone gets technical debt, and everyone has trouble educating users about new systems.
Meta is not special. It is not a beautiful and unique snowflake. It is the same sort of decaying collection of cobbled-together tech that every other organisation accrues over time.
In this case, the decrepit tech was “numerous heterogeneous services, such as warehouse data storage and various real-time systems, that make up Meta’s data platform — all exchanging large amounts of data among themselves as they communicate via service APIs.”
[2]
As Meta [3]detailed in December 2022, those systems struggled to scale as the data-harvesting giant built more AI workloads that needed to access data from diverse sources.
[4]
[5]
Improved data logging and serialization was the answer, so that data could describe itself more effectively and therefore be more easily ingested by diverse applications.
Meta built a system called “Tulip” to sort that out. And was chuffed that the formats it used required 40 percent to 85 percent fewer bytes and uses 50 percent to 90 percent fewer CPU cycles.
[6]Carmack quits Meta, brands it inefficient and unprepared for competition
[7]Unlucky for some: Meta chops 13% of global workforce
[8]Meta wants to sweat its servers for longer – at a cost of $60b
[9]Shareholders slam Zuckerberg's 'terrifying' $100b+ Metaverse experiment
As Meta’s Thursday post explains, Tulip may have been top tech but making it work was hard, not least because the social media giant employed over 30,000 logging schemas.
Across the four-year effort to adopt Tulip, Meta engineers found some data wasn’t able to be easily ingested or converted, or that doing so was computationally expensive. Some tools designed to ease migration created problems as they ran, so engineers created rate limiters so that issues didn’t snowball.
[10]
And then there were those pesky users, whose role planting Tulip in Meta’s tech garden necessitated the creation of a migration guide, an instructional video, plus a support team.
“Making huge bets such as the transformation of serialization formats across the entire data platform is challenging in the short term, but it offers long-term benefits and leads to evolution over time,” the post winds up.
“Designing and architecting solutions that are cognizant of both the technical as well as nontechnical aspects of performing a migration at this scale are important for success,” the post adds. “We hope that we have been able to provide a glimpse of the challenges we faced and solutions we used during this process.”
[11]
Meta’s four engineers probably have offered useful insights for those who face similar data-wrangling challenges. The rest of you who have lived through legacy migrations? Maybe less so.
And for everyone else, the insight here is that Met has become more efficient at wielding exabytes of data. Much of it gathered from, and about, you. ®
Get our [12]Tech Resources
[1] https://engineering.fb.com/2023/01/26/data-infrastructure/tulip-modernizing-metas-data-platform/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://engineering.fb.com/2022/11/09/developer-tools/tulip-schematizing-metas-data-platform/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2022/12/19/john_carmack_quits_meta/
[7] https://www.theregister.com/2022/11/09/meta_redundancies_mark_zuckerberg/
[8] https://www.theregister.com/2022/10/27/meta_q3_2022/
[9] https://www.theregister.com/2022/10/24/shareholders_to_meta_cut_staff/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9Ou0fSAx2agOegUjX8DKAAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://whitepapers.theregister.com/
Meta found data migration difficult?
Harry Kiri
Strange - the interface to a skip(*) is usually quite easy to get the hang of...
(*dumpster to the cousins...)
Re: Meta found data migration difficult?
Pascal Monett
Nobody deletes data anymore. You never know when you might find some use for that obscure content in the Comments3 field.
In the past days, libero.it mail went down for days due to a storage migration
In the past days, libero.it, a mail system still used by millions of Italians (it was one of the commom mail providers before the gmail era) went down for days because a data migration to a new storage system went bad. Rumors say the issue was due to a bug in a new NetApp system. People could not access, send or receive email from 23 january until yesterday - and it looks the service is not available again for everyone.