> I know that the main Apple motivation is to massage the investors, but is people really using their Macs to run local LLMs? (Genuine question, I have no idea). Another question: are software that have real AI capabilities using those NPUs or just exchanging data with company servers?

Yes, limited local AI. The have built a Privacy-Focused, three-tiered AI architecture, unique (AFAICT) in the industry.

Tom's guide explains it fairly well

[1]https://www.tomsguide.com/ai/a... [tomsguide.com]

[1] https://www.tomsguide.com/ai/apple-intelligence-everything-to-know-about-apples-ai

Apple ones - yes. The unified memory architecture makes them the fastest things you can buy for running AI outside of $5000 nVidia AI GPUs. Every desktop GPU is going to run out of memory on any serious model, and nothing else is as fast as these.

I mean, sure, you're likely wanting the M5 Ultra or M5 Max for doing that, rather than the base M5, but given that they're the same architecture, just with bits chopped off or doubled up, yeh... they're gonna need that.

I do for my personal dev projects. It's getting to the point where I can see it being the new norm at my day job in the next 18 months.

Yeah, they absolutely are. There are some Local LLM reddit groups where people are doing some neat stuff.

The M* hardware is very impressive.

I am. M4 Max with 128GB RAM. It's one of the cheapest options to run large models (basically anything that won't fit on a 5090's 32GB).

128GB of VRAM- ya. I'm using mine to do it. Love it.

The only thing that really uses the NPU is the OS.

You can build stable diffusion checkpoints to use it, but the only advantage is power efficiency- the GPU is faster... at many times the power draw.

There was some validity to this sort of argument when their base models came with 8 GB of memory and 128/256 GB of storage. That was always pretty borderline and you needed to factor in ~$400 on top of the base price to get it to a reasonable spot.

These have 16 GB memory and 512 GB storage. That's plenty for a large portion of the market.

What comparable laptop can you get for cheaper?

Apple is shockingly a strong name in local LLM use. Just not at the base model level..... The max and pro chips have extremely fat memory bus speeds and up to 192gb unified ram, but even the 96gb or 64gb options can run hefty LLMs

Wait... Apple doesn't come to mind for AI? You clearly no absolutely nothing about buying hardware for running AI.

In order of speed, your options are:

A server with $5000 nVidia GPUs with lots of VRAM on board.

A Mac, preferably with their highest end chip in it

A desktop PC with a high end desktop graphics card

Thanks to their unified memory architecture, Apple's neural engines are the fastest thing that doesn't cost $5000 for the GPU alone that doesn't instantly run out of memory running any vaguely serious

> For one thing, it's got questionable usability. For another, if I'm interested in AI, Apple isn't the name that immediately comes to mind..

Yeah, Siri sucks and Apple's models are behind, but Apple is doing some interesting research and the M chip architecture is very, very good for running local models.

Even if you ignore Apple's own AI software, it's popping up in 3rd party software all over the place, including graphics and video editing.

I have an M1 MacBook Pro Max I've run gpt-oss-20b on it. It wouldn't support a dozen users, but it worked fine for me for experimental and dev testing.

Considering I didn't buy this machine with AI as an intended use, that's pretty amazing.

There's definitely some room for Apple in this space, while everybody thinks the only player is Nvidia.

> If you keep the lid closed and don't do anything with it.

It's easy to tell you haven't used Apple hardware in a long, long time.