Your PC can probably run inferencing just fine – so it's already an AI PC
- Reference: 1710315067
- News link: https://www.theregister.co.uk/2024/03/13/age_of_ai_pc/
- Source link:
While Intel suggests that application developers will soon infuse all software with AI, and PCs must be ready for them, the workload that currently matters most is inferencing to power large language models.
Any AI PC worthy of the name must therefore perform inferencing quickly and well.
[3]
Inferencing is the process that transforms a submitted prompt into its response. Doing so needs a machine that can grind through multiple gigabytes of data – and even more matrix multiplications.
[4]
[5]
While such a compute-intensive task sounds like it could well stretch the capacities of a desktop PC (ChatGPT and its ilk run in colossal datacenters), I've discovered that a lot of the hardware we already have can do an adequate job of inferencing. Anyone can run a "good enough" chatbot on a PC – provided the PC has enough RAM and a mid-range GPU.
That means a GPU with at least 8GB of VRAM (and not one from Intel – at least at the moment, for lack of driver support). That's not too pricey – which is good, because it's table stakes. For RAM, you'd be happier with 32GB than 16GB, while 8GB simply won't cut it. And that's about it. That's all you're going to need.
[6]
Apple has finessed this exceptionally well – possibly entirely accidentally – with its M-series SoCs. Treating memory as "unified" – it's all VRAM if it needs to be – all modern M-series Macs with at least 16GB of RAM can serve as an AI PC.
As it happens, four of the PCs I've purchased over the last decade meet these specs – including an eight year-old top-of-the-line monster purchased as a "VR PC" with an Intel 6700K CPU and then-top-of-the-line Nvidia GTX 980 Ti GPU. More or less in line with Moore's Law, it's somewhat slower than the newer machines in my collection, but it runs a chatbot just fine – locally.
While not purchased as AI PCs, my more recent acquisitions each inference at a speed comparable to the completions generated by OpenAI's GPT-3.5-Turbo – virtually indistinguishable from their bigger cousin both in speed, and in the quality of the completions they generate.
[7]We asked Intel to define 'AI PC'. Its reply: 'Anything with our latest CPUs'
[8]Desktop GPU shipments jumped by a third – no thanks to AI PCs
[9]Apple gets in on the AI PC hype, claims fanless M3 MacBook Air is fab for LLMs
[10]Dell promises 'every PC is going to be an AI PC' whether you like it or not
To test these systems, I feed them a [11]fairly sophisticated prompt , instructing them to act as an "agent" – solving a problem by generating JSON files that get fed into another program. ("Small pieces loosely joined" still applies in the Age of AI.) And it all … just works.
So it turns out I already have AI PCs all over the place. Who knew? What's more, there are a lot of AI PCs out there. Tens of millions. Which is good – because we're going to need them.
[12]
While Microsoft would have us pasting every sensitive document into Copilot for analysis – thereby giving Redmond datacenters a sniff of our most confidential workflows – plenty of documents are simply too sensitive to be shared at all.
What about the documents so hot they shouldn't even live on a networked machine? What about medical information? Or classified documents? So much of what we work with comes with strings attached, preventing it from being shared freely.
Legal documents frequently fall into this category – the kinds of tasks where the recently released [13]SaulML legal language model could be enormously helpful. If a lawyer or paralegal had that model running on their local PC, they could freely pump documents into it, ask it to generate a close reading, even new contract terms – all without worrying about any confidential information leaks. Many law firms would run toward that sort of tool, screaming " Shut up and take my client's money! "
I prototyped that tool last weekend – cutting the full-fat open source SaulLM model down to size, so it could work on my fleet of AI PCs. I'm not a lawyer, so I can't fully judge its quality. I did, however, paste into it vast sections of a contract from a prospective business partner, then ask it some hard questions – without any fears of leaks.
Even if the lawyers don't love it, the rest of us – confronted with endless, abstruse terms and conditions that we [14]ignore at our peril – will find it very useful.
Likewise the hardware vendors – who rely on you junking the PC you bought to work from home three years ago for their revenue cycle – won't like to admit it, but you probably already have a perfectly serviceable AI PC. Just not the AI PC that vendorland wants you to rush out and buy. ®
Get our [15]Tech Resources
[1] https://www.theregister.com/2024/02/09/ai_pc_gartner/
[2] https://www.theregister.com/2024/03/12/what_is_an_ai_pc/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZfGHSl7pPAZMXQUlFYWwvQAAAcU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZfGHSl7pPAZMXQUlFYWwvQAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZfGHSl7pPAZMXQUlFYWwvQAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZfGHSl7pPAZMXQUlFYWwvQAAAcU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/03/12/what_is_an_ai_pc/
[8] https://www.theregister.com/2024/03/06/desktop_gpu_shipments/
[9] https://www.theregister.com/2024/03/04/apples_fanless_macbook_air_gets/
[10] https://www.theregister.com/2024/02/26/dell_ai_pc/
[11] https://www.safelyandwisely.ai/testing-prompt
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZfGHSl7pPAZMXQUlFYWwvQAAAcU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://www.theregister.com/2024/03/09/better_call_saul_llm/
[14] https://www.theregister.com/2022/10/10/paypal_fine_disinformation/
[15] https://whitepapers.theregister.com/
Re: Extra mile
Haven't you read [1]this ?
[1] https://www.theregister.com/2024/02/24/chatgpt_cuddy_legal_fees/
Re: Extra mile
"I don't hallucinate at work"
So I can run a local chatbot
With my i9 10980XE, my 64GB of DDR4, my 8TB of HDDs and my GeForce RTX 4080 Panther, it would seem, from your experience, that I have a machine that can run a chatbot.
Now if you could just convince me why I would need one. I already have a wife and a cat if I wish to talk with someone, and even the cat has more brains than a chatbot.
Re: So I can run a local chatbot
Of course your cat has more brains than a chatbot. You are demonstrating a common misunderstanding fostered by marketing scum. The I in AI does not stand for "INTELLIGENCE". It stands for "INCOMPETENCE". Are you really submitting the argument that your cat (or wife for that matter) is more incompetent than a sophisticated modern technology that has been lovingly tuned by mankind's best minds to produce random, unreliable, and possibly wildly incorrect answers to just about any conceivable question? Really now. How likely is that?
Re: So I can run a local chatbot
The authors use of the word Chatbot in the original article probably was not the best term to use to get their point across, as there are lot of the work that AI tools running locally can do are not chatbots. The author did give examples of why some people might want to run then on their own PC rather than just using Chat GPT or other online services, such as for when privacy or legal concerns means you don't want to or cannot let the information leave your system.
Re: So I can run a local chatbot
"Chatbot" doesn't mean something you discuss your day with, it refers to the fact that you can have a back and forth conversation with it about something - in this example some document you want it to review.
The example being if someone gives you a contract to sign, you might have a "chat" with your lawyer about its contents. (Although if you using a chatbot for legal questions, ensure the initial prompt tells it to remind you "this is not legal advice, I do not have liability insurance" with every answer!)
Re: So I can run a local chatbot
My system has 32GB RAM, an RTX4070 and 8GB of VRAM. It can run all of the freely-licensed chatbots from GPT4All - but maybe that's not saying so much. There is a definite difference in quality of responses between those models and the ones you get online from Google or OpenAI. It can run Stable Diffusion's image generation models ... just ... if there's not too much else running and you use the version that's been optimised to need less memory.
As to why you'd want it, they are genuinely useful. I'm using them for language learning at the moment. They're perfectly capable of setting you exercises, correcting them, explaining what you got wrong, chatting informally, assessing formal writing, introducing new grammar and so on and so on, all stuff you'd ordinarily pay a language teacher real money to do for you. Does it get the odd thing wrong? Probably. But it gives you about 98% of what a paid language tutor gives you, is free and is available 24 hours a day, for a few minutes or a few hours whenever you want it.
Tiny Jetson Orin SBCs can inference at a reasonable rate using pre-trained models. Even a basic Jetson Nano can do it usefully to ID objects.
Extra mile
> Many law firms would run toward that sort of tool, screaming "Shut up and take my client's money!"
And with luck, many clients would say to their lawyers"I can now do that without your assistance. What value do you add?"