Raspberry Pi's New Add-on Board Has 8GB of RAM For Running Gen AI Models (theverge.com)
- Reference: 0180588630
- News link: https://it.slashdot.org/story/26/01/15/1849235/raspberry-pis-new-add-on-board-has-8gb-of-ram-for-running-gen-ai-models
- Source link: https://www.theverge.com/news/862748/raspberry-pi-ai-hat-2-gen-ai-ram
> Raspberry Pi is launching a new add-on board capable of running generative AI models locally on the Raspberry Pi 5. Announced on Thursday, the $130 AI HAT+ 2 is an upgraded -- and more expensive -- version of the module launched last year, [1]now offering 8GB of RAM and a Hailo 10H chip with 40 TOPS of AI performance.
>
> Once connected, the Raspberry Pi 5 will use the AI HAT+ 2 to handle AI-related workloads while leaving the main board's Arm CPU available to complete other tasks. Unlike the previous AI HAT+, which is focused on image-based AI processing, the AI HAT+ 2 comes with onboard RAM and can run small gen AI models like Llama 3.2 and DeepSeek-R1-Distill, along with a series of Qwen models. You can train and fine-tune AI models using the device as well.
[1] https://www.theverge.com/news/862748/raspberry-pi-ai-hat-2-gen-ai-ram
Terribly disappointed in the name (Score:3)
Given the general view of AI and LLMs (especially on /.), they should have called it the AI Supplementary Storage HAT.
Or, ASSHAT for short.
Re: (Score:1)
People on slashdot who are luddites are hilarious to me. AI is the next stage of human evolution (as soon as we can integrate it into our brains), and yet they resist.
Reminds me of how VR/AR is a logical step to cybernetics, and yet they resist. The real beta tests for that, ghost in the shell cybernetic utopia future, were google glass etc, but the beta testers were called glassholes, when all they were, were visionaries who were a few decades too early to a future that is coming.
Re: (Score:3)
I'd be using AR glasses right now if they were not created by the big platforms as just another way to make you, (me) the consumer, the product. To surf the net like the Major using her cyberbrain and a few virtual and physical agents we'd need a much larger leap in understanding the mammalian brain. I don't trust Elmo to develop a safe brain/computer interface.
Re: (Score:2)
> People on slashdot who are luddites are hilarious to me. AI is the next stage of human evolution (as soon as we can integrate it into our brains), and yet they resist. Reminds me of how VR/AR is a logical step to cybernetics, and yet they resist. The real beta tests for that, ghost in the shell cybernetic utopia future, were google glass etc, but the beta testers were called glassholes, when all they were, were visionaries who were a few decades too early to a future that is coming.
No. These people were glassholes because the only thing they enabled was recording people for the purpose of large companies somehow monetizing it.
One is not a luddite for shunning shit tech.
Re: (Score:2)
Not sure there are that many Luddites here, we just want it on our terms. And in many instances that means being unwilling to move forward with a new technology if it means surrendering our privacy.
Product in search of a market (Score:2)
Is there a demand for these at all? Seems like their making the product before there is a market..
Re: (Score:2)
They're chasing a fad. Their hope is that there are enough of their customers chasing this same fad that they can make a profit off of them buying what sounds to be an essentially worthless product. (And that's even if you're willing to grant that "full-size" LLMs are worthwhile.)
Maybe they do; maybe they don't. But as one of their customers, I personally resent this diversion of resources from more worthwhile projects in any case.
Who cares? (Score:2)
Fuck them for focusing on commercial customers during COVID. I hope every hobbyist repays the Raspberry Pi Foundation's (lack of) loyalty.
Re: (Score:2)
Exactly.
---
Hey from SATX ex-ATX. The snowpocaylpse and paving over of Rainey St for condo douchebags and treatlerites, I noped out of there.
Just because you can... (Score:3)
Seeing as this consumes the only PCIe port on the device, you can't use NVMe storage in conjunction with it, making the entire thing far less useful since all of the other storage options for the Pi are dogshit
I have yet to see a use case for small LLMs (Score:2)
Since all shortcomings of the very large language models popular these days are much more pronounced and abundant in the not-so-large-language-models that fit in 8GB, I wonder what use cases this is meant for. I could understand why somebody would want to use some small "AI-upscaler" or "image recognition" in a Raspberry PI... but LLMs?
Re: I have yet to see a use case for small LLMs (Score:2)
Does seem a bit small for any gen AI I know of. 16GB seems to be the minimum. Can you use multiple hats perhaps to expand the RAM? Perhaps useful for computer vision/audio.
Re: (Score:3)
I run deepseek on my 4060 8gb and it's been great.
Re: (Score:2)
been great at what?
Re: (Score:3)
nah you just hallucinated it was great.
Re: (Score:2)
> I could understand why somebody would want to use some small "AI-upscaler" or "image recognition" in a Raspberry PI... but LLMs?
I'm sure there are a few use cases, but the thing that comes to mind for me right now is something like Home Assistant.
Re: (Score:2)
I have a dream of running my own personal "google home" from my basement - I want it to turn on and off lights, maybe adjust the thermostat using voice commands. I also want it access a few web pages and be able to answer questions (via voice) regarding their contents: local weather, stock prices, maybe Wikipedia. No reporting back to the motheship because I am the mothership. This might be of a size to be able to accomplish this.
Re: I have yet to see a use case for small LLMs (Score:2)
I was going to ask this very question.
Re: (Score:1)
Z-Wave controllers already do all of that without the non-privacy of Google or the shit of AI.
Re: (Score:2)
LLMs don't need to be large to be useful. Large LLMs are great for generative AI where you insist it creates a story, but small scale LLMs find their niche in contextual search, translation, OCR, and many cases at the *input* side of whatever it is you are trying to achieve.
You can also get very small models if you restrict the application. E.g. if you need basic inference the model can be small. If you need reasoning the model can also be small if your source space is small.
AI is more than LLMs, and LLMs a
Re: (Score:2)
As if people are training models for the job, and will do so specifically for tasks running on a Pi.
Re: (Score:2)
There are lots of people training smaller language models, including ones for specific tasks, and using the ones that already exist.
You seem to be big on assumptions but for readers who are actually interested Hugging Face has a couple of articles:
[1]Small Language Models (SLM): A Comprehensive Overview [huggingface.co]
[2]A Survey of Small Language Models in the Era of LLMs: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness [huggingface.co]
[1] https://huggingface.co/blog/jjokah/small-language-model
[2] https://huggingface.co/blog/FairyFali/slm-survey
Re: (Score:2)
For voice assistants it's helpful for it to be local. It turns out that 98% of commands fall into about 10-12 commands (Set a timer for 5 min, turn on/off the lights, what time is it, whats todays date, turn on/off tv, turn on/off the lights in another room). The device catalogs all these requests and then makes a list of the top ~30 requests and if the request matches something on the list with ~0.85 confidence it doesn't even go to the LLM it just runs the command. That's how you get the instant response