So you want to replace workers with AI? Watch out for retraining fees, they're a killer
(2023/01/29)
- Reference: 1674993610
- News link: https://www.theregister.co.uk/2023/01/29/ai_retraining_costs/
- Source link:
Comment The lucid ramblings and art synthesized by ChatGPT or Stable Diffusion have captured imaginations and prompted no shortage of controversy over the role generative AI will play in our futures.
As we've seen with CNET and [1]Buzzfeed , executives are no less dazzled by AI's creative potential to [2]replace workers with profits. But one of things that's often missed in these conversations is the need to retrain these models regularly or risk them aging into irrelevance, particularly in rapidly evolving environments like the news.
ChatGPT, Stable Diffusion, Dall-E-2 and the majority of generative AI today are trained on large datasets and then made available as proof of concepts or exported as a pre-trained model.
[3]
Let's take Stable Diffusion as an example as it offers a glimpse at just how misleading the scope these models can be. Like Dall-E-2 Stable Diffusion is multi-modal. It's made up of a collection of models that work together to turn your works into a visual representation.
[4]
[5]
But where Stable Diffusion stands out is that its pre-trained model [6]can fit into just 4GB of Nvidia vRAM without sending the CPU into overdrive trying to churn data. This means you can run it at home on a decently powerful laptop or desktop so long as you've got a dedicated GPU with enough memory. The ability to run models at home has opened the eyes of many of the potential for generative AI, but while fun, pre-trained models also have a finite shelf life.
Imagine if you exposed a child to everything the world has to offer. For 18 years they absorb all the knowledge they can, but on the first day of their adult life they're locked away in a cave and isolated from the world. Now imagine you provided that person with art supplies and asked them to draw, paint, and render images based on your prompts.
[7]
At first the images would be relatively accurate, but with each passing day isolation puts them at a greater disadvantage. As the prompts increasingly venture into unfamiliar territory, the art steadily becomes less accurate.
A pre-trained AI model isn't much different. It's blind to the world from the point its training is complete. This is why for generative AI to be truly useful it's going to need to be retrained repeatedly. And herein lies the problem: while these AI models all seem magical, training them even once remains an exceptionally expensive proposition.
This makes private school look like a bargain
Calculating the cost of training is a tricky thing because there are so many variables at play. But for the purposes of this piece, we're going to take a look at floating point accuracy, model size, and training time to help put it all in perspective.
Most AI training today is done on GPUs each with a relatively small amount of fast memory onboard. Nvidia's A100 and H100 GPUs both sport 80GB of HBM memory, while AMD and Intel's GPUs are now pushing 128GB. While there are other architectures out there with different memory topologies, we're going to stick to Nvidia's A100 because the hardware is well supported, widely available in both on-prem and cloud environments and has been running AI workloads for years at this point.
Floating point accuracy is one of the biggest factors, as it plays both into training time and how much memory the model will need. The latter also dictates how much compute is required, as each accelerator only has so much memory. Training time itself is harder to quantify because it'll vary depending on compute density, the quantity of accelerators, the size of the dataset, the number of parameters at play and any number of other related variables.
[8]
Most models today are trained using FP32, FP16, or Bfloat16, though many industry players are now pushing FP8 calculations. As you drop down the scale, accuracy is traded for greater performance and the models tend to get smaller too. For this reason, it's not uncommon for models to use mixed precision, which essentially involves using lower accuracy calculations for some parameters and higher accuracy for others, usually to optimize performance.
So just how big are these models? Well, with ChatGPT generating no shortage of controversy as of late, let's take a look at GPT-3 on which the divisive AI model is based. At 175 billion parameters GPT-3, unveiled in [9]mid 2020 , was trained on a massive cluster of Nvidia V100 GPUs on a dataset of roughly 2TBs.
From what we understand, GPT-3 was trained using FP32 precision, which means four bytes per parameter. That works out to about 700GB of vRAM required just to fit the model. Today, that'd require about ten 80GB Nvidia A100s, but unless you want to wait years for it to train, you're gonna want a few more chunks of big iron.
Engineers at Nvidia, working alongside scientists at Stanford University and Microsoft Research, [10]estimated in a 2021 paper that it'd take 1,024 A100s 34 days to train GPT-3 on a 1.2TB dataset. To put that in perspective, that's the equivalent to 128 AWS p4de.24xlarge instances. At $40.96 per hour apiece, and with 816 hours required to train, that'd run you in the neighborhood of $4.28 million just to train it. Running inferencing on the trained model to ensure smarter performance is another issue entirely.
[11]AI cannot be credited as authors in papers, top academic journals rule
[12]Software glitch revokes copyright protection for AI-generated comic book
[13]Microsoft can't stop itself blowing billions on OpenAI
[14]Universities offered software to sniff out ChatGPT-written essays
And that's just GPT-3. Future models are expected to be an order of magnitude larger, with some speculating that GPT-4 could be as large as a trillion parameters in size. But, since we don't have any firm details on GPT-4 just yet, we'll look at another large language model from Nvidia.
Behold the Megatron
Nvidia's Megatron-Turing NLG language model has 530 billion parameters, making it more than three-times larger than GPT-3. According to Nvidia, it took 2,048 Nvidia A100s running in mixed precision [15]eight weeks to train the model. Going back to our AWS example, now we're talking about just over $14 million to train it once. It doesn't take much of an imagination to see why retraining every week on an incrementally larger dataset could get expensive in a hurry.
You might ask why not train on prem if the cloud is so expensive. This is a valid point, especially if you're going to be retraining your model constantly, but it still requires a big upfront investment.
Using Nvidia's Megatron-Turning NLG example from before, you'd need 256 8-GPU nodes. We'll use Nvidia's DGX A100 servers as an example. While the cost of these systems varies, we've seen pricing in the neighborhood of $175,000.
For 256 nodes the costs work out to $44.8 million and that doesn't consider the power and maintenance required to keep them up and running. Under full load, a 256 node cluster could chew through 1.7 megawatts an hour. Assuming constant retraining, you're looking at $2.2 million a year in power. Of course, in reality it should be a fair bit less than that.
Proliferation of faster accelerators and lower/mix precision calculations will certainly help, but that's assuming the models don't [16]continue to outpace our advances in silicon.
The point of diminishing returns
If we've learned anything about human nature, it's that we'll take whatever shortcuts we can if it means turning a buck. Massive natural language models like ChatGPT may be impressive, but the sheer cost to train and then retrain them will make them so impractical that only the largest companies can afford to use them to their full potential.
Businesses like Microsoft, which operate massive GPU clusters with tens of thousands of accelerators, are well positioned to do just that, so it's no surprise the company is making massive [17]investments in companies like OpenAI.
But as AI models and accelerators mature, the number of models tailored to specific applications are likely to proliferate.
We've already seen a slew of AI art generators emerge in the wake of Dall-E. But despite failing to deliver the same degree of polish as its rivals, Stable Diffusion's open source nature and ability to not only be deployed, but trained on consumer hardware, have made it a standout hit.
Stable Diffusion also demonstrates that AI isn't immune to the rule of diminishing returns. Luxury cars may captivate drivers, but if they can't afford them, they make do with their Ford or Honda. While it may lack the style or prestige of a luxury brand, it'll still get you to point A to B. There's no reason to think the same won't be true of AI adoption in the enterprise.
Ultimately, the goal isn't perfection, it's mediocrity. As long as the model is good enough - and costs less than having a person do it - the AI will have paid for itself. And as we've discussed, there are plenty of corners to cut. ®
Get our [18]Tech Resources
[1] https://pressgazette.co.uk/publishers/digital-journalism/buzzfeed-chatgpt/
[2] https://www.theregister.com/2023/01/19/cnet_reviewing_ai_authored_stories/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://stable-diffusion-ui.github.io/docs/installation/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2020/06/12/openai_ai_cloud_sale/
[10] https://arxiv.org/pdf/2104.04473.pdf
[11] https://www.theregister.com/2023/01/27/top_academic_publisher_science_bans/
[12] https://www.theregister.com/2023/01/25/glitch_us_copyright_office_ai/
[13] https://www.theregister.com/2023/01/23/microsoft_openai_applications/
[14] https://www.theregister.com/2023/01/23/turnitin_chatgpt_detector/
[15] https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
[16] https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
[17] https://www.theregister.com/2023/01/23/microsoft_openai_applications/
[18] https://whitepapers.theregister.com/
As we've seen with CNET and [1]Buzzfeed , executives are no less dazzled by AI's creative potential to [2]replace workers with profits. But one of things that's often missed in these conversations is the need to retrain these models regularly or risk them aging into irrelevance, particularly in rapidly evolving environments like the news.
ChatGPT, Stable Diffusion, Dall-E-2 and the majority of generative AI today are trained on large datasets and then made available as proof of concepts or exported as a pre-trained model.
[3]
Let's take Stable Diffusion as an example as it offers a glimpse at just how misleading the scope these models can be. Like Dall-E-2 Stable Diffusion is multi-modal. It's made up of a collection of models that work together to turn your works into a visual representation.
[4]
[5]
But where Stable Diffusion stands out is that its pre-trained model [6]can fit into just 4GB of Nvidia vRAM without sending the CPU into overdrive trying to churn data. This means you can run it at home on a decently powerful laptop or desktop so long as you've got a dedicated GPU with enough memory. The ability to run models at home has opened the eyes of many of the potential for generative AI, but while fun, pre-trained models also have a finite shelf life.
Imagine if you exposed a child to everything the world has to offer. For 18 years they absorb all the knowledge they can, but on the first day of their adult life they're locked away in a cave and isolated from the world. Now imagine you provided that person with art supplies and asked them to draw, paint, and render images based on your prompts.
[7]
At first the images would be relatively accurate, but with each passing day isolation puts them at a greater disadvantage. As the prompts increasingly venture into unfamiliar territory, the art steadily becomes less accurate.
A pre-trained AI model isn't much different. It's blind to the world from the point its training is complete. This is why for generative AI to be truly useful it's going to need to be retrained repeatedly. And herein lies the problem: while these AI models all seem magical, training them even once remains an exceptionally expensive proposition.
This makes private school look like a bargain
Calculating the cost of training is a tricky thing because there are so many variables at play. But for the purposes of this piece, we're going to take a look at floating point accuracy, model size, and training time to help put it all in perspective.
Most AI training today is done on GPUs each with a relatively small amount of fast memory onboard. Nvidia's A100 and H100 GPUs both sport 80GB of HBM memory, while AMD and Intel's GPUs are now pushing 128GB. While there are other architectures out there with different memory topologies, we're going to stick to Nvidia's A100 because the hardware is well supported, widely available in both on-prem and cloud environments and has been running AI workloads for years at this point.
Floating point accuracy is one of the biggest factors, as it plays both into training time and how much memory the model will need. The latter also dictates how much compute is required, as each accelerator only has so much memory. Training time itself is harder to quantify because it'll vary depending on compute density, the quantity of accelerators, the size of the dataset, the number of parameters at play and any number of other related variables.
[8]
Most models today are trained using FP32, FP16, or Bfloat16, though many industry players are now pushing FP8 calculations. As you drop down the scale, accuracy is traded for greater performance and the models tend to get smaller too. For this reason, it's not uncommon for models to use mixed precision, which essentially involves using lower accuracy calculations for some parameters and higher accuracy for others, usually to optimize performance.
So just how big are these models? Well, with ChatGPT generating no shortage of controversy as of late, let's take a look at GPT-3 on which the divisive AI model is based. At 175 billion parameters GPT-3, unveiled in [9]mid 2020 , was trained on a massive cluster of Nvidia V100 GPUs on a dataset of roughly 2TBs.
From what we understand, GPT-3 was trained using FP32 precision, which means four bytes per parameter. That works out to about 700GB of vRAM required just to fit the model. Today, that'd require about ten 80GB Nvidia A100s, but unless you want to wait years for it to train, you're gonna want a few more chunks of big iron.
Engineers at Nvidia, working alongside scientists at Stanford University and Microsoft Research, [10]estimated in a 2021 paper that it'd take 1,024 A100s 34 days to train GPT-3 on a 1.2TB dataset. To put that in perspective, that's the equivalent to 128 AWS p4de.24xlarge instances. At $40.96 per hour apiece, and with 816 hours required to train, that'd run you in the neighborhood of $4.28 million just to train it. Running inferencing on the trained model to ensure smarter performance is another issue entirely.
[11]AI cannot be credited as authors in papers, top academic journals rule
[12]Software glitch revokes copyright protection for AI-generated comic book
[13]Microsoft can't stop itself blowing billions on OpenAI
[14]Universities offered software to sniff out ChatGPT-written essays
And that's just GPT-3. Future models are expected to be an order of magnitude larger, with some speculating that GPT-4 could be as large as a trillion parameters in size. But, since we don't have any firm details on GPT-4 just yet, we'll look at another large language model from Nvidia.
Behold the Megatron
Nvidia's Megatron-Turing NLG language model has 530 billion parameters, making it more than three-times larger than GPT-3. According to Nvidia, it took 2,048 Nvidia A100s running in mixed precision [15]eight weeks to train the model. Going back to our AWS example, now we're talking about just over $14 million to train it once. It doesn't take much of an imagination to see why retraining every week on an incrementally larger dataset could get expensive in a hurry.
You might ask why not train on prem if the cloud is so expensive. This is a valid point, especially if you're going to be retraining your model constantly, but it still requires a big upfront investment.
Using Nvidia's Megatron-Turning NLG example from before, you'd need 256 8-GPU nodes. We'll use Nvidia's DGX A100 servers as an example. While the cost of these systems varies, we've seen pricing in the neighborhood of $175,000.
For 256 nodes the costs work out to $44.8 million and that doesn't consider the power and maintenance required to keep them up and running. Under full load, a 256 node cluster could chew through 1.7 megawatts an hour. Assuming constant retraining, you're looking at $2.2 million a year in power. Of course, in reality it should be a fair bit less than that.
Proliferation of faster accelerators and lower/mix precision calculations will certainly help, but that's assuming the models don't [16]continue to outpace our advances in silicon.
The point of diminishing returns
If we've learned anything about human nature, it's that we'll take whatever shortcuts we can if it means turning a buck. Massive natural language models like ChatGPT may be impressive, but the sheer cost to train and then retrain them will make them so impractical that only the largest companies can afford to use them to their full potential.
Businesses like Microsoft, which operate massive GPU clusters with tens of thousands of accelerators, are well positioned to do just that, so it's no surprise the company is making massive [17]investments in companies like OpenAI.
But as AI models and accelerators mature, the number of models tailored to specific applications are likely to proliferate.
We've already seen a slew of AI art generators emerge in the wake of Dall-E. But despite failing to deliver the same degree of polish as its rivals, Stable Diffusion's open source nature and ability to not only be deployed, but trained on consumer hardware, have made it a standout hit.
Stable Diffusion also demonstrates that AI isn't immune to the rule of diminishing returns. Luxury cars may captivate drivers, but if they can't afford them, they make do with their Ford or Honda. While it may lack the style or prestige of a luxury brand, it'll still get you to point A to B. There's no reason to think the same won't be true of AI adoption in the enterprise.
Ultimately, the goal isn't perfection, it's mediocrity. As long as the model is good enough - and costs less than having a person do it - the AI will have paid for itself. And as we've discussed, there are plenty of corners to cut. ®
Get our [18]Tech Resources
[1] https://pressgazette.co.uk/publishers/digital-journalism/buzzfeed-chatgpt/
[2] https://www.theregister.com/2023/01/19/cnet_reviewing_ai_authored_stories/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://stable-diffusion-ui.github.io/docs/installation/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9amLqQC0yvVZY61gjR1IwAAAFE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2020/06/12/openai_ai_cloud_sale/
[10] https://arxiv.org/pdf/2104.04473.pdf
[11] https://www.theregister.com/2023/01/27/top_academic_publisher_science_bans/
[12] https://www.theregister.com/2023/01/25/glitch_us_copyright_office_ai/
[13] https://www.theregister.com/2023/01/23/microsoft_openai_applications/
[14] https://www.theregister.com/2023/01/23/turnitin_chatgpt_detector/
[15] https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
[16] https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
[17] https://www.theregister.com/2023/01/23/microsoft_openai_applications/
[18] https://whitepapers.theregister.com/
Re: Bring on the Luddites!
Harry Kiri
The best description I've heard of today's 'AI' is just a jumped up autocomplete. It needs humans to curate all the data, humans to hand fettle, design the structure and build the transform in the network (for that is all it is).
The excitement over GPT3 is just the latest incarnation of people getting excited over implementing tech (and the notion that crude approximations of networked nodes is clever like a brain as it's structured like a brain), as opposed to why the tech solves a given problem. A while ago it was things like Alexa. Before that it was apps on phones, Before that it was Victorian clockwork automaton (as back then the world was deterministic).
The thing about GPT3 and its ilk is that as a thing it is clever, however its output isn't.
Re: Bring on the Luddites and the Crazies
amanfromMars 1
We've already seen what a 21st Century mill owner looks like, Mr Elon Musk buying up Twitter and pushing everyone out who refuses to do things his way. .... Plest
What a great load of nonsense that is, Plest, and haven’t you yet noticed the present is brutally suppressed by a system that will always prize profit over human diginity .... and now is being forced to be constantly fighting against the system, wasting and depleting its own blood and treasures, to try to ensure news and evidence of the situation does not escape and become general universal knowledge, although that rear guard action was discovered an impossible task to achieve and master some long time ago.
Who?
Derezed
Who is any of this wank for?
I heard ChatGPT attempting to present news and do journalism. Who asked for that? Who does it benefit? This irritating move towards self service check outs (who are they for? ) is the tip of the iceberg. More unions, more strikes,jobs for humans please.
Standing on the shoulders of giants
Pete 2
> that'd run you in the neighborhood of $4.28 million just to train it.
But that isn't how people learn. That one day someone decides that something you know is obsolete, so you have to return to primary school and start learning The cat sat on the mat and everything else you've ever know, all over again.
People learn incrementally: building their knowledge on what the already knew. Sure, there are times when a paradigm shift comes along and people have to relearn a skill or set of rules (such as when a country changes the side of the road its drivers drive on). But those a few and far between. And usually highly compartmentalised. When the UK when from pounds-shillings and pence to decimalised currency, that did not mean anyone had to relearn how to read or write.
So it should be with training AIs.
Bring on the Luddites!
Contrary to what most think, the Luddites were not anti-technology. When the weavers were put out of work by the new industrial processes it was the Luddites that demanded cross-training to the new technology, they wanted compensaiton schemes, they wanted fair wages for newly trained people on the new machines, they wanted health insurance, pensions for those injured and too old to work. They wanted to work with the new industrial barons but he owners of the "dark satanic mills" refused, spread the idea that the Luddites were anti-technology, that the Luddites were not willing to accept the new order of man working for machines, not the other way around. The mill owners had the power to get the government to hunt down and execute the Luddites for daring to ask for fair treatment for the disposessed.
We've already seen what a 21st Century mill owner looks like, Mr Elon Musk buying up Twitter and pushing everyone out who refuses to do things his way. As AI becomes more useful and starts put of we techies out of jobs, and it will to a greater or lesser extent, how long before we have to form the "Luddites V2" and how long until were brutally suppressed by a system that will always prize profit over human diginity.