Microsoft and OpenAI method could make training large neural networks cheaper
(2022/03/14)
- Reference: 1647259332
- News link: https://www.theregister.co.uk/2022/03/14/microsoft_openai_mutransfer/
- Source link:
Companies scaling up their neural network models could cut expensive training costs by employing a new technique developed by researchers at Microsoft and OpenAI.
Machine-learning systems are often compared to black boxes. Data is fed into an algorithm and out pops some more data. It can be a label classifying an object in an image, a string of text, or even a snippet of code. The computation that happens in the middle involves manipulating countless matrices, and is a mystifying, hand-wavy process experts don't quite fully understand.
There are several properties developers tinker with to boost a model's performance during the training stage. These so-called hyperparameters are separate from the data, and are often manually fine-tuned based on intuition alone. Finding the optimum hyperparameters requires training and adjusting the model numerous times; all that computation is costly and time-consuming. As systems grow larger and larger in size – with billions and trillions of parameters – it becomes too expensive to fine-tune.
[1]
"In practice, people rely on many rules of thumb to come up with 'educated guesses' of hyperparameters to use for a large model run without much confidence of their optimality," Greg Yang, a senior researcher at Microsoft, and Edward Hu, a PhD Student at Mila, a research institute based in Montreal, told The Register .
[2]
[3]
Together with researchers from OpenAI, they devised a method described in a [4]paper , named [5]μTransfer (pronounced mu-transfer) to make it easier to fine-tune large neural networks. First, they find the optimal hyperparameters by tinkering with a smaller model, and then transfer them to a larger, scaled-up system. The team experimented with μTransfer on the text-generating GPT-3 architecture, reusing hyperparameters from a 40-million-parameter model to a 6.7-billion-parameter sized one.
[6]SPEC mulls benchmarks for ML processing performance
[7]Driverless car first: Chinese biz recalls faulty AI
[8]DeepMind AI tool helps historians restore ancient texts
[9]Rate of autonomous vehicle safety improvement slowing – research
By getting rid of the need to repeatedly adjust the larger GPT-3's hyperparameters, the team estimated their fine-tuning costs using μTransfer was only 7 percent of what it would be to pre-train the model. Without μTransfer, the cost of fine-tuning the bigger system would have costed 167.5 times more, we're told. Models containing billions of parameters can rack up millions of dollars in compute costs.
"We are able to keep the optimal hyperparameters stable across model size thanks to a new parametrization suggested by the theory of neural network infinite-width limits," Yang and Hu told us. Neural networks, loosely modeled on the structure of brains, are made up of layers of neurons. The width of a network is described by the number of neurons contained in each layer, a wider network has more neurons. The depth of a network is described by the number of layers, a deeper network has more layers.
The pair explained the theory is an abstract concept that allows researchers to study the limits of a model as it increases in size. They found that some hyperparameters, like the learning rate, should be adjusted for the different width sizes of each layer. Other hyperparameters, however, aren't so easily transferable. Developers will still need to fine-tune their models if they can.
[10]
μTransfer is most effective for scaling existing architectures to larger sizes, where some hyperparameters can be reused. "Rather than being applied to fine-tuning, we are more likely to see our technique being used to find better hyperparameters and 'supercharge' model pretraining in the near future. We believe that the biggest payoff will come from pretraining enormous models with billions or even trillions of parameters," they said.
If you want to use μTransfer in scaling up your own models, the code can be found [11]here . ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.microsoft.com/en-us/research/publication/tuning-large-neural-networks-via-zero-shot-hyperparameter-transfer/
[5] https://www.microsoft.com/en-us/research/blog/%c2%b5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/
[6] https://www.theregister.com/2022/03/11/spec_developing_benchmarks_for_ml/
[7] https://www.theregister.com/2022/03/10/driverless_car_first_chinese_company/
[8] https://www.theregister.com/2022/03/09/ai_ancient_greek/
[9] https://www.theregister.com/2022/03/08/rate_of_av_safety_improvement/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://github.com/microsoft/mup
[12] https://whitepapers.theregister.com/
Machine-learning systems are often compared to black boxes. Data is fed into an algorithm and out pops some more data. It can be a label classifying an object in an image, a string of text, or even a snippet of code. The computation that happens in the middle involves manipulating countless matrices, and is a mystifying, hand-wavy process experts don't quite fully understand.
There are several properties developers tinker with to boost a model's performance during the training stage. These so-called hyperparameters are separate from the data, and are often manually fine-tuned based on intuition alone. Finding the optimum hyperparameters requires training and adjusting the model numerous times; all that computation is costly and time-consuming. As systems grow larger and larger in size – with billions and trillions of parameters – it becomes too expensive to fine-tune.
[1]
"In practice, people rely on many rules of thumb to come up with 'educated guesses' of hyperparameters to use for a large model run without much confidence of their optimality," Greg Yang, a senior researcher at Microsoft, and Edward Hu, a PhD Student at Mila, a research institute based in Montreal, told The Register .
[2]
[3]
Together with researchers from OpenAI, they devised a method described in a [4]paper , named [5]μTransfer (pronounced mu-transfer) to make it easier to fine-tune large neural networks. First, they find the optimal hyperparameters by tinkering with a smaller model, and then transfer them to a larger, scaled-up system. The team experimented with μTransfer on the text-generating GPT-3 architecture, reusing hyperparameters from a 40-million-parameter model to a 6.7-billion-parameter sized one.
[6]SPEC mulls benchmarks for ML processing performance
[7]Driverless car first: Chinese biz recalls faulty AI
[8]DeepMind AI tool helps historians restore ancient texts
[9]Rate of autonomous vehicle safety improvement slowing – research
By getting rid of the need to repeatedly adjust the larger GPT-3's hyperparameters, the team estimated their fine-tuning costs using μTransfer was only 7 percent of what it would be to pre-train the model. Without μTransfer, the cost of fine-tuning the bigger system would have costed 167.5 times more, we're told. Models containing billions of parameters can rack up millions of dollars in compute costs.
"We are able to keep the optimal hyperparameters stable across model size thanks to a new parametrization suggested by the theory of neural network infinite-width limits," Yang and Hu told us. Neural networks, loosely modeled on the structure of brains, are made up of layers of neurons. The width of a network is described by the number of neurons contained in each layer, a wider network has more neurons. The depth of a network is described by the number of layers, a deeper network has more layers.
The pair explained the theory is an abstract concept that allows researchers to study the limits of a model as it increases in size. They found that some hyperparameters, like the learning rate, should be adjusted for the different width sizes of each layer. Other hyperparameters, however, aren't so easily transferable. Developers will still need to fine-tune their models if they can.
[10]
μTransfer is most effective for scaling existing architectures to larger sizes, where some hyperparameters can be reused. "Rather than being applied to fine-tuning, we are more likely to see our technique being used to find better hyperparameters and 'supercharge' model pretraining in the near future. We believe that the biggest payoff will come from pretraining enormous models with billions or even trillions of parameters," they said.
If you want to use μTransfer in scaling up your own models, the code can be found [11]here . ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.microsoft.com/en-us/research/publication/tuning-large-neural-networks-via-zero-shot-hyperparameter-transfer/
[5] https://www.microsoft.com/en-us/research/blog/%c2%b5transfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/
[6] https://www.theregister.com/2022/03/11/spec_developing_benchmarks_for_ml/
[7] https://www.theregister.com/2022/03/10/driverless_car_first_chinese_company/
[8] https://www.theregister.com/2022/03/09/ai_ancient_greek/
[9] https://www.theregister.com/2022/03/08/rate_of_av_safety_improvement/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yi90uUMDqW5eJAruFN-JOAAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://github.com/microsoft/mup
[12] https://whitepapers.theregister.com/