Want to save the planet from AI? Chuck in an FPGA and ditch the matrix
- Reference: 1719386774
- News link: https://www.theregister.co.uk/2024/06/26/ai_model_fpga/
- Source link:
In [1]a paper titled, "Scalable MatMul-free Language Modeling," authors Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason Eshraghian describe how the energy appetite of artificial intelligence can be moderated by getting rid of [2]matrix multiplication and adding a custom field-programmable gate array (FPGA).
AI – by which we mean predictive, hallucinating machine learning models – has been terrible for keeping Earth habitable because it uses so much energy, [3]much of which comes from fossil fuel use. The operation of datacenters to provide AI services has increased Microsoft's CO 2 emissions [4]by 29.1 percent since 2020, and AI-powered Google searches each [5]use 3.0 Wh , ten times more than traditional Google queries.
[6]
Earlier this year, a report from the [7]International Energy Agency [PDF] projected that global data center power consumption will nearly double by 2026, rising from 460TWh in 2022 to just over 800TWh in two years. The hunger for energy to power AI has even reinvigorated [8]interest in nuclear power , because accelerating fossil fuel consumption for the sake of chatbots, bland marketing copy, and on-demand image generation has become politically fraught, if not a potential crime against humanity.
[9]
[10]
Jason Eshraghian, an assistant professor of electrical and computer engineering at the UC Santa Cruz Baskin School of Engineering and the paper’s lead author, told The Register that the research findings could provide a 50x energy savings with the help of custom FPGA hardware.
"I should note that our FPGA hardware was very unoptimized, too," said Eshraghian. "So there's still a lot of space for improvement."
[11]
The prototype is already impressive. A billion-parameter LLM can be run on the custom FPGA with just 13 watts, compared to 700 watts that would have been required using a GPU.
To achieve this, the US-based researchers had to do away with matrix multiplication, a linear algebra technique that is widely used in machine learning and is costly from a computational perspective. Instead of multiplying weights (parameters assigned to link neural network layers) consisting of floating point numbers between 0 and 1, the computer scientists added and subtracted binary {0, 1} or ternary representations {-1, 0, 1}, thus demanding less of their hardware.
Other researchers over the past few years have explored alternative architectures for neural networks. One of these, [12]BitNet , has shown promise as a way to reduce energy consumption through simpler math. As described in a [13]paper released in February, representing neural network parameters (weights) as {-1, 0, 1} instead of using 16-bit floating point precision can provide high performance with much less computation.
[14]To solve AI's energy crisis, 'rethink the entire stack from electrons to algorithms,' says Stanford prof
[15]Using AI in science can add to reproducibility woes, say boffins
[16]Big brains divided over training AI with more AI: Is model collapse inevitable?
[17]What's up with AI lately? Let's start with soaring costs, public anger, regulations...
The work of Eshraghian and his co-authors demonstrates what can be done with this architecture. Sample code has been [18]published to GitHub.
Eshraghian said, the use of "ternary weights replaces multiplication with addition and subtraction, which is computationally much cheaper in terms of memory usage and the energy of actual operations undertaken."
[19]
That's combined, he said, with the replacement of "self-attention," the backbone of transformer models, with an "overlay" approach.
"In self attention, every element of a matrix interacts with every single other element," he said. "In our approach, one element only interacts with one other element. By default, less computation leads to worse performance. We compensate for this by having a model that evolves over time."
Eshraghian explained that transformer-based LLMs take all text in one hit. "Our model takes each bit of text piece by piece, so our model is tracking where a particular word is situated in a broader context by accounting for time," he said.
Reliance on ternary representation of data does hinder performance, Eshraghian acknowledged, but he and his co-authors found ways to offset that effect.
"Given the same number of computations, we're performing on par with Meta's open source LLM," he said. "However, our computations are ternary operations, and therefore, much cheaper (in terms of energy/power/latency). For a given amount of memory, we do far better."
Even without the custom FPGA hardware, this approach looks promising. The paper claims that by [20]fused kernels in the GPU implementation of ternary dense layers, training can be accelerated by 25.6 percent while memory consumption can be reduced by 61 percent compared to a GPU baseline.
"Furthermore, by employing lower-bit optimized CUDA kernels, inference speed is increased by 4.57 times, and memory usage is reduced by a factor of 10 when the model is scaled up to 13B parameters," the paper claims.
"This work goes beyond software-only implementations of lightweight models and shows how scalable, yet lightweight, language models can both reduce computational demands and energy use in the real-world." ®
Get our [21]Tech Resources
[1] https://arxiv.org/abs/2406.02528
[2] http://matrixmultiplication.xyz/
[3] https://www.washingtonpost.com/business/2024/06/21/artificial-intelligence-nuclear-fusion-climate/
[4] https://www.theregister.com/2024/05/16/microsoft_co2_emissions/
[5] https://www.theregister.com/2024/01/30/ai_is_changing_search/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZnvmwIy6-U8o14XHm6zcigAAAQE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[7] https://iea.blob.core.windows.net/assets/6b2fd954-2017-408e-bf08-952fdd62118a/Electricity2024-Analysisandforecastto2026.pdf
[8] https://www.theregister.com/2024/05/01/ai_nuclear_dc_uranium/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnvmwIy6-U8o14XHm6zcigAAAQE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnvmwIy6-U8o14XHm6zcigAAAQE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnvmwIy6-U8o14XHm6zcigAAAQE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://arxiv.org/abs/1708.04788
[13] https://arxiv.org/asb/2402.17764
[14] https://www.theregister.com/2024/06/05/a_high_five_for_stanford/
[15] https://www.theregister.com/2024/05/29/using_ai_in_science_can/
[16] https://www.theregister.com/2024/05/09/ai_model_collapse/
[17] https://www.theregister.com/2024/04/15/stanford_report_ai/
[18] https://github.com/ridgerchu/matmulfreellm
[19] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnvmwIy6-U8o14XHm6zcigAAAQE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[20] https://github.com/dmlc/nnvm-fusion
[21] https://whitepapers.theregister.com/
Re: Old news?
No, I don’t think it is old news. There’s a nice subtlety here, which gets obscured by them making more than one change at a time.
The key point is that they have replaced the “self attention” mechanism. The self-attention is/was there because it’s the only way for the algorithm to look at the whole set of text at once. But what probably hasn’t occurred to people before, is that the self-attention algorithm is designed to handle the situation on a normal CPU where only one register-bank is available to the CPU execution units at one time - it’s costly to shove things in and out of memory, all the way through cache.
But FPGAs don’t have that constraint: the execution units can see tens of thousands of tokens simultaneously - so you can design a different self-attention mechanism, the one you would have designed if the CPU constraints hadn’t been there. We didn’t know, because software people make software assumptions.
And then, on top of that, he’s refactored the overall matrix multiplication algorithm to use what FPGAs do well: bit wise operations. That has a performance hit, but one you can mitigate by simply having more parameters, and that precision trade off comes out differently on an FPGA than a CPU or GPU.
But no; this is new stuff, because we didn’t know about self-attention until a couple of years back. And presumably his improved self-attention algorithm tuned for FPGAs is an all-new concept we haven’t seen before. It’s nice work, I think.
It still might not be “successful” because GPUs are so dominant, that it’s easier to make progress there than on limited-supply restricted skill-set FPGAs. And just throw more GPUs at the problem. That I don’t know, but $100M of GPU time per training run, focuses a lot of minds, so we’ll see.
Perhaps work on the ...
... "hallucinations'" issue first, THEN make it go faster and/or with less power.
If you can't get trustworthy results out of the kludge, it's a dead-end anyway. No point in making it more efficient.
Re: Perhaps work on the ...
Ah but, never mind the quality, feel the width!
Re: Perhaps work on the ...
Not necessarily true. He’s also changed the self-attention, which is a major thing. This isn’t just an algorithmic refactoring for efficiency. The quality and type of issues will be “different”. Can’t say whether they will be better or worse; but different.
Re: Perhaps work on the ...
> Perhaps work on the .. ... "hallucinations'" issue first, THEN make it go faster and/or with less power.
Is it not possible that by having multiple systems can reduce hallucinations? In an operation is of lower power and faster, then it can get run several times from 'different angles'.
I as a human see a distant white spot against a grass background. My life experience tells me it might be a mushroom, a rock, a plastic bucket or a seagull. However, I don't act on any one line of reasoning, perception or memory. I remember that the month is May, so I rule out mushroom. I remember that I'm not in a chalk area, so a rock seems less likely. I watch it for a while and it doesn't move, so not a seagull. I walk closer to it, and confirm that it is a plastic bucket.
I suspect we humans are hallucinating all the time, but we don't act on any one hallucination.
13W instead of 700
That's almost 54 times less.
FPGA FTW !
Monogamy?
"In self attention, every element of a matrix interacts with every single other element," he said. "In our approach, one element only interacts with one other element."
The first ("self attention") by analogy is narcissistic promiscuity and the second ("interacts with one") monogamy. ;)
The former is both delusional and unhealthy continuing the analogy with hallucinatory AI.
At 3Wh per AI query (3 J per sec for 1 hour = 10800 J* or 10.8 kJ !) or a 3.0V chip drawing 60.0A for 1 minute. So we are cooking the planet in order to replace the human lack of intelligence with an artificial version of that same deficiency? ;)
* For unrecoverable faredge reformists and the recalcitrant left pondial: ~10.2 BTU, 7966 ft-lb(f), 2.58 kcal
So with more efficient techniques
They'll just throw more data and more complex models to their FPGA and still need the same amount of power...
Re: So with more efficient techniques
Yes. Why is that a problem? Power usage is only a problem if we aren’t getting the desired output results achieved.
The *wrong* way to look at it is to say that a human brain requires 100W, so until LLM achieves that it is “inefficient”.
The *right* way is to observe that we don’t own slaves any more, and human brains don’t come in jars. We’re prepared to pay humans at least $10per hour, for really low-skill tasks. That’s about 100kW in electricity usage.
As long as LLM is using less than 100kW, ie less than one full rack in data centre, I don’t care as long as it gets the job done. We can debate *if* and *when* it will get the job done, but that’s what we should be focusing on.
Old news?
FPGA's were the only way to perform neural nets fast and efficiently enough last century.
A new generation make the 'startling discovery' again.