AI caramba, those neural networks are power-hungry: Counting the environmental cost of artificial intelligence
(2021/09/13)
- Reference: 1631527391
- News link: https://www.theregister.co.uk/2021/09/13/ai_environmental_cost/
- Source link:
Feature The next time you ask Alexa to turn off your bedroom lights or [1]make a computer write dodgy code , spare a thought for the planet. The back-end mechanics that make it all possible take up a lot of power, and these systems are getting hungrier.
Artificial intelligence began to gain traction in mainstream computing just over a decade ago when we worked out how to make GPUs handle the underlying calculations at scale. Now there's a machine learning algorithm for everything, but while the world marvels at the applications, some researchers are worried about the environmental expense.
One of the most [2]frequently quoted papers on this topic, from the University of Massachusetts, analysed training costs on AI including Google's BERT natural language processing model. It found that the cost of training BERT on a GPU in carbon emissions was roughly the same as a trans-American jet flight.
[3]
Kate Saenko, associate professor of computer science at Boston University, worries that we're not doing enough to make AI more energy efficient. "The general trend in AI is going in the wrong direction for power consumption," she warns. "It's getting more expensive in terms of power to train the newer models."
[4]Imaginary numbers help AIs solve the very real problem of adversarial imagery
[5]DRAM-as-cache is too expensive for even Facebook – Zuck now blending it with NVM drives
[6]HPE bags $2bn HPC-as-a-service gig with the NSA
[7]US officials, experts fear China ransacked Exchange servers for data to train AI systems
The trend is exponential. Researchers associated with [8]OpenAI wrote that the computing used to train the average model increases by a factor of 10 each year.
Why is AI so power hungry?
Most AI these days is based on machine learning (ML). This uses a [9]neural network , which is a collection of nodes designed in layers. Each node has connections to nodes in the next. Each of these connections has a score known as a parameter or weight.
The neural network takes an input (such as a [10]picture of a hotdog ) and runs it through the layers of the neural network, each of which uses its parameters to produce an output. The final output is a judgement about the data (for example, was the original input a picture of a hotdog or not?)
[11]
[12]
Those weights don't come preconfigured. You have to calculate them. You do that by showing the network lots of labelled pictures of hot dogs and not hot dogs. You keep training it until the parameters are optimised, which means that they spit out the correct judgement for each piece of data as often as possible. The more accurate the model, the better it will be when making judgements about new data.
You don't just train an AI model once. You keep doing it, adjusting various aspects of the neural network each time to maximise the right answers. These aspects are called hyperparameters, and they include variables such as the number of neurons in each layer and the number of layers in each network. A lot of that tuning is trial and error, which can mean many training passes. Chewing through all that data is already expensive enough, but doing it repeatedly uses even more electrons.
[13]
The reason that the models are taking more power to train is that researchers are throwing more data at them to produce more accurate results, explains Lukas Biewald. He's the CEO of Weights and Biases, a company that helps AI researchers organise the training data for all these models while monitoring their compute usage.
"What's alarming about about it is that it seems like for every factor of 10 that you increase the scale of your model training, you get a better model," he says.
Yes, but the model's accuracy doesn't increase by a factor of 10. Jesse Dodge, postdoctoral researcher at the Allen Institute for AI and co-author of a paper called [14]Green AI , notes studies pointing to the diminishing returns of throwing more data at a neural network.
[15]
So why do it?
"There's a long tail of things to learn," he explains. ML algorithms can train on the most commonly-seen data, but the edge cases – the confusing examples that rarely come up – are harder to optimise for.
Our hotdog recognition system might be fine until some clown comes along in a hotdog costume, or it sees a picture of a hotdog-shaped van. A language processing model might be able to understand 95 per cent of what people say, but wouldn't it be great if it could handle exotic words that hardly anyone uses? More importantly, your autonomous vehicle must be able to stop in dangerous conditions that rarely ever arise.
"A common thing that we see in machine learning is that it takes exponentially more and more data to get out into that long tail," Dodge says.
Piling on all this data data doesn't just slurp power on the compute side, points out Saenko; it also burdens other parts of the computing infrastructure. "The larger the data, the more overhead," she says. "Even transferring the data from the hard drive to the GPU memory is power intensive."
Sharing is caring
There are various attempts to mitigate this problem. It starts at the data centre level, where hyperscalers are doing their best to switch to renewables so that they can at least hammer their servers responsibly.
Another approach involves taking a more calculated approach when tweaking your hyperparameters. Weights and Biases offers a "hyperparameter sweep" service that uses Bayesian algorithms to narrow the field of potential changes with each training pass. It also offers an "early stopping" algorithm which halts a training pass early on if the optimisation isn't panning out.
Not all approaches involve fancy hardware and software footwork. Some are just about sharing. Dodge points out that researchers could amortise the carbon cost of their model training by sharing the end result. Trained models released in the public domain can be used without retraining, but people don't take enough advantage of that.
"In the AI community, we often train models and then don't release them," he says. "Or the next people that want to build on our work just rerun the experiments that we did."
Those trained models can also be fine tuned with additional data, enabling people to tweak existing optimisations for new applications without retraining the entire model from scratch.
Training isn't the whole story
Making training more efficient only tackles one part of the problem, and it isn't the most important part. The other side of the AI story is inference. This is when a computer runs new data through a trained model to evaluate it, recognising hotdogs it has never seen before. It still takes power, and the rapid adoption of AI is making it more of a problem. Every time you ask Siri how to cook rice properly, it uses inference power in the cloud.
One way to reduce model size is to cut down the number of parameters. AI models often use vast numbers of weights in a neural network because data scientists aren't sure which ones will be most useful. Saenko and her colleagues have researched reducing the number of parameters using a concept that they call shape shifter networks that share some of the parameters in the final model.
"You might train a much bigger network and then distil it into a smaller one so that you can deploy a smaller network and save computation and deployment at inference time," she says.
Companies are also working on hardware innovations to cope with this increased inference load. Google's Tensor Processing Units (TPUs) are tailored to handle both training and inference more efficiently, for example.
Solving the inference problem is especially tricky because we don't know where a lot of it will happen in the long term. The move to edge computing could see more inference jobs happening in lower-footprint devices rather than in the cloud. The trick there is to make the models small enough and to introduce hardware advances that will help to make local AI computation more cost-effective.
"How much do companies care about running their inference on smaller devices rather than in the cloud on GPUs?" Saenko muses. "There is not yet that much AI running standalone on edge devices to really give us some clear impetus to figure out a good strategy for that."
Still, there is movement. Apple and Qualcomm have already produced tailored silicon for inference on smart phones, and startups are becoming increasingly innovative in anticipation of edge-based inference. For example, semiconductor startup Mythic launched an AI processor focused on edge-based AI that uses analogue circuitry and in-memory computing to save power. It's targeting applications including object detection and depth estimation, which could see the chips turn up in everything from factories to surveillance cameras.
As power consumption rises, so do the stakes
As companies grapple with whether to infer at the edge, the problem of making AI more energy efficient in the cloud remains. The key lies in resolving two opposing forces: on the one hand, everyone wants more energy efficient computing. On the other, researchers constantly strive for more accuracy.
Dodge notes that most academic AI papers today focus on the latter. Accuracy is winning out as companies strive to beat each other with better models, agrees Saenko. "It might take a lot of compute but it's worthwhile for people to claim that one or two percent improvement," she says.
She would like to see more researchers publish data on the power consumption of their models. This might inspire competition to drive efficiencies up and costs down.
The stakes may be more than just environmental, warns Biewald; they could be political too. What happens if computing consumption continues to go up by a factor of 10 each year?
"You have to buy the energy to train these models, and the only people that can realistically afford that will be Google and Microsoft and the 100 biggest corporations," he posits.
If we start seeing a growing inequality gap in AI research, with corporate interests out in front, carbon emissions could be the least of our worries. ®
Get our [16]Tech Resources
[1] https://www.theregister.com/2021/07/09/openai_codex_paper/
[2] https://arxiv.org/abs/1906.02243
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2021/09/02/imaginary_numbers_help_ais_solve/
[5] https://www.theregister.com/2021/09/03/facebook_cachelib_nvm_not_dram/
[6] https://www.theregister.com/2021/09/01/hpe_hpc_nsa/
[7] https://www.theregister.com/2021/08/31/in_brief_security/
[8] https://openai.com/blog/ai-and-compute/
[9] https://www.theregister.com/2019/02/26/ai_training_world_record/
[10] https://www.theregister.com/2017/07/10/skills_for_ai/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://cacm.acm.org/magazines/2020/12/248800-green-ai/fulltext
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[16] https://whitepapers.theregister.com/
Artificial intelligence began to gain traction in mainstream computing just over a decade ago when we worked out how to make GPUs handle the underlying calculations at scale. Now there's a machine learning algorithm for everything, but while the world marvels at the applications, some researchers are worried about the environmental expense.
One of the most [2]frequently quoted papers on this topic, from the University of Massachusetts, analysed training costs on AI including Google's BERT natural language processing model. It found that the cost of training BERT on a GPU in carbon emissions was roughly the same as a trans-American jet flight.
[3]
Kate Saenko, associate professor of computer science at Boston University, worries that we're not doing enough to make AI more energy efficient. "The general trend in AI is going in the wrong direction for power consumption," she warns. "It's getting more expensive in terms of power to train the newer models."
[4]Imaginary numbers help AIs solve the very real problem of adversarial imagery
[5]DRAM-as-cache is too expensive for even Facebook – Zuck now blending it with NVM drives
[6]HPE bags $2bn HPC-as-a-service gig with the NSA
[7]US officials, experts fear China ransacked Exchange servers for data to train AI systems
The trend is exponential. Researchers associated with [8]OpenAI wrote that the computing used to train the average model increases by a factor of 10 each year.
Why is AI so power hungry?
Most AI these days is based on machine learning (ML). This uses a [9]neural network , which is a collection of nodes designed in layers. Each node has connections to nodes in the next. Each of these connections has a score known as a parameter or weight.
The neural network takes an input (such as a [10]picture of a hotdog ) and runs it through the layers of the neural network, each of which uses its parameters to produce an output. The final output is a judgement about the data (for example, was the original input a picture of a hotdog or not?)
[11]
[12]
Those weights don't come preconfigured. You have to calculate them. You do that by showing the network lots of labelled pictures of hot dogs and not hot dogs. You keep training it until the parameters are optimised, which means that they spit out the correct judgement for each piece of data as often as possible. The more accurate the model, the better it will be when making judgements about new data.
You don't just train an AI model once. You keep doing it, adjusting various aspects of the neural network each time to maximise the right answers. These aspects are called hyperparameters, and they include variables such as the number of neurons in each layer and the number of layers in each network. A lot of that tuning is trial and error, which can mean many training passes. Chewing through all that data is already expensive enough, but doing it repeatedly uses even more electrons.
[13]
The reason that the models are taking more power to train is that researchers are throwing more data at them to produce more accurate results, explains Lukas Biewald. He's the CEO of Weights and Biases, a company that helps AI researchers organise the training data for all these models while monitoring their compute usage.
"What's alarming about about it is that it seems like for every factor of 10 that you increase the scale of your model training, you get a better model," he says.
Yes, but the model's accuracy doesn't increase by a factor of 10. Jesse Dodge, postdoctoral researcher at the Allen Institute for AI and co-author of a paper called [14]Green AI , notes studies pointing to the diminishing returns of throwing more data at a neural network.
[15]
So why do it?
"There's a long tail of things to learn," he explains. ML algorithms can train on the most commonly-seen data, but the edge cases – the confusing examples that rarely come up – are harder to optimise for.
Our hotdog recognition system might be fine until some clown comes along in a hotdog costume, or it sees a picture of a hotdog-shaped van. A language processing model might be able to understand 95 per cent of what people say, but wouldn't it be great if it could handle exotic words that hardly anyone uses? More importantly, your autonomous vehicle must be able to stop in dangerous conditions that rarely ever arise.
"A common thing that we see in machine learning is that it takes exponentially more and more data to get out into that long tail," Dodge says.
Piling on all this data data doesn't just slurp power on the compute side, points out Saenko; it also burdens other parts of the computing infrastructure. "The larger the data, the more overhead," she says. "Even transferring the data from the hard drive to the GPU memory is power intensive."
Sharing is caring
There are various attempts to mitigate this problem. It starts at the data centre level, where hyperscalers are doing their best to switch to renewables so that they can at least hammer their servers responsibly.
Another approach involves taking a more calculated approach when tweaking your hyperparameters. Weights and Biases offers a "hyperparameter sweep" service that uses Bayesian algorithms to narrow the field of potential changes with each training pass. It also offers an "early stopping" algorithm which halts a training pass early on if the optimisation isn't panning out.
Not all approaches involve fancy hardware and software footwork. Some are just about sharing. Dodge points out that researchers could amortise the carbon cost of their model training by sharing the end result. Trained models released in the public domain can be used without retraining, but people don't take enough advantage of that.
"In the AI community, we often train models and then don't release them," he says. "Or the next people that want to build on our work just rerun the experiments that we did."
Those trained models can also be fine tuned with additional data, enabling people to tweak existing optimisations for new applications without retraining the entire model from scratch.
Training isn't the whole story
Making training more efficient only tackles one part of the problem, and it isn't the most important part. The other side of the AI story is inference. This is when a computer runs new data through a trained model to evaluate it, recognising hotdogs it has never seen before. It still takes power, and the rapid adoption of AI is making it more of a problem. Every time you ask Siri how to cook rice properly, it uses inference power in the cloud.
One way to reduce model size is to cut down the number of parameters. AI models often use vast numbers of weights in a neural network because data scientists aren't sure which ones will be most useful. Saenko and her colleagues have researched reducing the number of parameters using a concept that they call shape shifter networks that share some of the parameters in the final model.
"You might train a much bigger network and then distil it into a smaller one so that you can deploy a smaller network and save computation and deployment at inference time," she says.
Companies are also working on hardware innovations to cope with this increased inference load. Google's Tensor Processing Units (TPUs) are tailored to handle both training and inference more efficiently, for example.
Solving the inference problem is especially tricky because we don't know where a lot of it will happen in the long term. The move to edge computing could see more inference jobs happening in lower-footprint devices rather than in the cloud. The trick there is to make the models small enough and to introduce hardware advances that will help to make local AI computation more cost-effective.
"How much do companies care about running their inference on smaller devices rather than in the cloud on GPUs?" Saenko muses. "There is not yet that much AI running standalone on edge devices to really give us some clear impetus to figure out a good strategy for that."
Still, there is movement. Apple and Qualcomm have already produced tailored silicon for inference on smart phones, and startups are becoming increasingly innovative in anticipation of edge-based inference. For example, semiconductor startup Mythic launched an AI processor focused on edge-based AI that uses analogue circuitry and in-memory computing to save power. It's targeting applications including object detection and depth estimation, which could see the chips turn up in everything from factories to surveillance cameras.
As power consumption rises, so do the stakes
As companies grapple with whether to infer at the edge, the problem of making AI more energy efficient in the cloud remains. The key lies in resolving two opposing forces: on the one hand, everyone wants more energy efficient computing. On the other, researchers constantly strive for more accuracy.
Dodge notes that most academic AI papers today focus on the latter. Accuracy is winning out as companies strive to beat each other with better models, agrees Saenko. "It might take a lot of compute but it's worthwhile for people to claim that one or two percent improvement," she says.
She would like to see more researchers publish data on the power consumption of their models. This might inspire competition to drive efficiencies up and costs down.
The stakes may be more than just environmental, warns Biewald; they could be political too. What happens if computing consumption continues to go up by a factor of 10 each year?
"You have to buy the energy to train these models, and the only people that can realistically afford that will be Google and Microsoft and the 100 biggest corporations," he posits.
If we start seeing a growing inequality gap in AI research, with corporate interests out in front, carbon emissions could be the least of our worries. ®
Get our [16]Tech Resources
[1] https://www.theregister.com/2021/07/09/openai_codex_paper/
[2] https://arxiv.org/abs/1906.02243
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2021/09/02/imaginary_numbers_help_ais_solve/
[5] https://www.theregister.com/2021/09/03/facebook_cachelib_nvm_not_dram/
[6] https://www.theregister.com/2021/09/01/hpe_hpc_nsa/
[7] https://www.theregister.com/2021/08/31/in_brief_security/
[8] https://openai.com/blog/ai-and-compute/
[9] https://www.theregister.com/2019/02/26/ai_training_world_record/
[10] https://www.theregister.com/2017/07/10/skills_for_ai/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://cacm.acm.org/magazines/2020/12/248800-green-ai/fulltext
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YT91vSLQ5ELAQ1IgWB4B3gAAAAc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[16] https://whitepapers.theregister.com/
The fallacy of optimization
Mike 137
@ b0llchit
Well said! This has even happened with LED lighting. In many places we now have increased energy use and more light pollution as people install many more room lights and swathes of domestic "low energy" outdoor lighing where there was none before
Pascal Monett
"Counting the environmental cost of artificial intelligence statistical analysis machines"
FTFY
We do not have AI. I will not stop bringing that point up.
machines and entropy
Anonymous Coward
I'd guess we can regard training anything as a process of reducing its entropy, so if we knew something about its number of states and the energy required to switch them, it might be possible to estimate the energy required to reach a given level of functionality, at least in rather general terms.
I suppose then it is a question of whether Beast-machines or Silicon-machines are more economical for the planet, and who gets to decide which should be propagated...
jmch
Yep, no real intelligence there, just glorified pattern matching that doesn't know what it's doing. Throwing more power at it won't change the fact that, fundamentally, there is no intelligence there. That's before even beginning to consider that even the most cutting edge brain biology and biochemistry has still eff-all idea of how human intelligence actually works.
Apropos of nothing, a low-power laptop processor consumes about 40W. A human brain consumes about 20W.
Martin Gregorie
Yes.
If the device can't explain how it arrived at the answer it just gave you, then it obviously isn't intelligent.
Human alternative
Primus Secundus Tertius
The human being runs on about 75 Watts. The brain - that with which we think we think - uses a fair chunk of that.
Re: Human alternative
jmch
About 20W I believe.
Would be interesting to match for example a chess or go human grand master vs program that not only runs on 20W, but only includes models that have been trained with 20W, instead of vs beasts like deep blue
Re: Human alternative
cdegroot
Just what I wanted to say - the consensus seems to be that the brain uses around 20 watts. I'll call this applied mathematics stuff "artificial intelligence" as soon as they approach the intelligence of, say, a dog using, say, 20 kilowatts.
A language processing model might be able to understand
Neil Barnes
Understand? Or just transcribe? I know which way I'm voting; with Pascal.
a hotdog shaped van
Anonymous Coward
... perhaps teach to it recognise both vans and hotdogs? Then it could decide which is most likely.
Of course the number of "other things with an appearance somewhat like a hotdog" might be quite large, leading to an equally large, albeit slightly different, training problem :-)
"we're not doing enough to make AI more energy efficient"
Mike 137
Has anyone examined the quality of the machine code? AI is probably written in an abstracted high level language - very likely using external libraries - and almost certainly this leads to bloat as in all other software these days.
The bigger processing power waste
Edwin
is in cryptocurrency, where the processing load for mining is pure waste.
Wouldn't mind seeing that addressed, but I suspect most crypto miners don't care and it would presumably be impossible to regulate.
...have already produced tailored silicon for inference on smart phones, and startups are becoming increasingly innovative in anticipation of edge-based inference.
The fallacy of optimization: you reduce the energy consumption for the individual and simultaneously increase the number of individuals.
And it looks like the increase in use is a significant factor larger than the optimizations. Thus, the energy consumption is still increasing. If optimization really would be making a dent (on a larger front), then we'd see a global decline of "carbon usage". Unfortunately, at the large scale, we see an increase or at best a flat line.