It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...

All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.

Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.

Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!

It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

I wish qwen3.8 had a MoE variant, but the skinny is it won't be coming.

If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s.

It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP.

But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens.

I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.

Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun.

That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.

I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.

    EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+

Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)

Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it.

How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?

Regarding the MTP, if you haven't found out why yet, the `--spec-draft-p-min 0.8`. Something to play with and of course it's non-deterministic and depends on both the model itself and whatever you're actually throwing at it.

This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198

Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.

I'm having the same constraint, 8GB VRAM, and was trying the past months to get a local model running to drive my Home Assistant setup via voice, so many tool calls, little room for errors. I've tried multiple models, different temperature and top p values, different context windows... turns out that a quantized gemma4 is the best pick so far but qwen3.5:9b isn't far off. Surprisingly newer models are not necessarily performing better.

I've had great success using Gemma4 even on old hardware (4GB VRAM). I haven't found a Qwen model that can match it on very low end hardware.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

I had heard of this and actually did set the reasoning to medium ahead of time…

Is there an easy way for a n00b with LMStudio to switch it to medium? Asking for a friend… XD

xhigh gives better results

Not necessarily.

I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.

Simon Willison’s blog has another example (SVG of a circle).

It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.

With qwen 27b, setting the right reasoning effort for the specific task is important. With xhigh it has a chance at hard problems that bigger models may even fail. But for many everyday tasks, I have found that no reasoning and a system prompt instructing it to be brief is good enough. Note that even with thinking disabled, it may still get into long "chain of thought" reasoning state (out of thinking blocks) if the task is hard and you do not give further instructions, esp with access to tools etc.

Not if it fills up its entire context with "But wait..."

xhigh tends not to do that. Uses caveman-ish language. But the reasoning trace does tend to obsess about stuff that it should just ask you about.

We don’t know what quantization level was used for the weights or the kv cache for you or for parent poster, so this is probably an apples to oranges comparison.

I've recently learned and then observed that oMLX serves local models much, much faster than LM Studio.

Set its thinking lower. This is a known issue. It still thinks A LOT with lower reasoning levels

Check out this: https://news.ycombinator.com/item?id=49402232 both article and comments. There are a lot of knobs to tweak, and some are pretty impactful.

Maybe giving pi more output by setting higher value to maxTokens will resolve his issue

I’ve been using the mlx version with orb studio an opencode

its all still somewhat of a dice roll

I got the qwen 3.8 abliterated model running on my MacBook Pro M5 48GB and it's pretty nice having a local model that can do a lot of experimentation without rails.

orcarouter or obliteratus?

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

It’s hot and also LOUD and runs the battery down quick.

But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in.

It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever?

I have an m5 max 64gb and I think anything slower would be quite painful.

>It’s hot and also LOUD and runs the battery down quick.

Who'd run this on battery?

Or do you mean kills the battery faster even when used while charging?

There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge.

That window is waning as more and more memory and graphics processing power is being used locally.

I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.

Running your dev environment on your laptop is so 2025.

"So 20XX" is so 2012.

No idea what you are talking about. My battery lasts longer than ever while running vim and make and GCC. It’s amazing. Not sure why your windows are closing.

Because the local LLM, which you are not running, is running for much longer than gcc and is eating the battery. Different choices, different outcomes.

Bur the implication here in the parent comment is that for ‘real dev work’ to be done you must be running an LLM, and therefore you can’t do it on a MacBook anymore.

Turns out you can still accomplish stuff with a text editor and compiler.

Of course we can code without a LLM, but if we want to, as the sibling comment says, don't run a local LLM on a laptop. I'm offloading the LLM to the LLM vendor, like most of us IMO. The CPU is still busier then it used to be, because of the local harness.

> There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge.

I was responding to this. I am appalled that anyone thinks (and is willing to say out loud in public) that they cannot do "real dev work" without an LLM.

So don’t run a local LLM?

Running the laptop hot ages the battery even when you’re plugged in.

Merely existing noticeably ages most battery chemistries.

You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.

I was trying to do some work on the train with no signal.

Couldn't you just enjoy the ride?

I have an m3 max 64GB and it runs ok, I was playing around with it last night. Its reasoning is better, but the token/s still makes an MoE model (35B A3B) preferable so I switched back.

Try using Goose instead of Claude's harness? Goose doesn't load as much context in at the start, so it might be more usable. I've definitely been able to get it past 128K, although I typically don't go beyond 70-80k. MoE might also be better at this.

If you're running it while idle and don't need the quickest results, reducing the clock speed improves energy efficiency (and in your case avoids overheating the battery). There will be some optimal speed that maximizes computations per joule that depends on the specific load and can only be found by measurement.

On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?

The Mac unfortunately just has two performance modes ‘all out power and melting’ or ‘cold and really really slow’

My M5 Max has 3 modes, "low", "automatic", and "high". Automatic doesn't seem to simply switch between low and high -- it seems to sit in the middle and vary dynamically based on workload and temperature. With automatic I get more than half as many tokens per second as on high, but with much lower fan speed and temperature.

I remember doing this with crypto mining way back in the day. That machine will die within a year.

Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom.

I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.

Yikes isn’t 95 well above the danger zone for most components or have things changed? I remember crypto mining in the early 2010’s and making sure to stay under 85C on my GPU.

Stuff runs so hot these days. I think one of the parameters pushed to keep Moores law from completely flattening was heat. My desktop regularly hits 95 just compiling code.

I guess thermal throttling will inevitably kick in.

Just curious, what is your ram and how many tokens per second do you get with that?

RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.

I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL

If you're going to try to use a mac laptop for long term inference you'll probably want to buy one of those $30 laptop fan cooling pads (sold for absurd sized 17 inch x86-64 gaming laptops) on Amazon. Otherwise you'll roast the battery over time and significantly hurt its cycle life and capacity. If you can get significant airflow under the laptop that'll help a lot.

I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.

Imagine spending all that money on Apple hardware only to throttle it to a fraction of its performance lol

Steve Jobs would be proud. People really are holding their Apple hardware wrong.

I tested Gemma 4 a couple months ago and yes it ramps the fans to max. Definitely a thing that should only be done on wall power.

Having a server in the basement helps a lot :-D Then tailscale from everywhere.

Mineral oil bath?

just decent air cooling and you'll be okay. it will get up to 75/80C though for my M5 MBP.

Which exact model are you running? With only 48GB of RAM, by the time I got a model small enough, it was pretty bad in performance both in speed and reasoning.

How many tok/s are you getting? What gen mbp?

My M4 Pro 48GB gets about 13tok/s, in both 3.6 and 3.8 27b Qwens. Qwen A3B and Gemma get closer to 100tok/s from memory but the results are pretty poor for coding tasks.

Edited to add: for agentic workflow I’m running omlx which tells me it has about a 90% cache hit rate (tradeoff is some disk and mem space) - that noticeably changes the felt speed.

I get around 20 tok/s, 4 bit quant, MTP, 4 bit KV cache quantisation. On an M4 Pro 48Gb.

I feel like it's 50/50 between people doing that, and people that have spent a lot of time tuning a system they are pointing at focused and well specified problems.

Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB.

That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.

Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?

Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.

Qwen3.8-27B runs at 59.5 tok/s on my M4 Max, 40-core GPU, 128 GB

I use it occasionally for classification and other tasks but I wouldn't trust those smaller models with the real work and for larger data processing it's too slow, e.g. a dataset I wanted to classify would've taken 56 days on my laptop vs just paying the cheap Luna prices to openai and getting it done in a few hours.

59.5 t/s is really good. Which engine/quant are you using?

Not sure I would trust Luna with that. Deepseek Pro Max and Code Mode I would be more inclined to trust.

I've been using Qwen3.6-37B-A3B on an M1 Max w/ llama.cpp and for my practical uses I prefer it to qwen3.8. When 3.8 does answer its slower and, qualitatively, marginally better than qwen3.6, but 3.8 often ends up in unresolved thought loops and runs slower. The Moe 3.6 on my setup is much faster, 500t/s peaks, 30t/s typical, vs 3.8 150 peak, 4-9 t/s typical.

While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.

Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.

It’s not because you didn’t find use cases for local LLMs that there are none.

I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc

Gemma 4 26B A4B gives me steady 20 tps.

Regarding this analogy, fps don't matter as much for Photoshop, since it's not an immediate mode GUI. 8 fps would be quite ok for comfortably getting feedback on live image filters and such.

M3 Pro 36GB. I am getting 17 tps with MTPLX.

We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.

GPT-OSS 20B didn’t really merit the fanfare even when it was released; it’s definitely not competitive now. Even the 120B version has been well eclipsed by smaller LLMs at this point. The last version of Qwen 27B/35B was better, and now the new one is even better than that!

Was there a more recent refresh or is this the model from a year ago? The frontier models were barely functional and almost useless a year ago (gpt oss was pre opus 4.5!) - I would be very surprised if the original drop is anything more than totally obsolete/irrelevant at this point

Yes, the old entirely stupid old gpt-oss. But Sonnet and GPT were very useful then already, qwen also.

I find that I remember models being a lot better than they were, even when I remember them being not very good - because of a novelty factor ("whoa it can do that now?") mostly. And then I go back and look at them and its like, what how did I find this impressive.

A funny example - I remember thinking "yeah sonnet 3.5 is a really good coding model"

https://stack.convex.dev/using-cursor-claude-and-convex-to-b...

>Prompting Cursor to Scaffold my App: FAIL This was my first hurdle.

>It became immediately apparent that I would not be able to prompt my way through the entire process.

>While the tooling we have is undeniably powerful, it's not yet capable of completing most nontrivial tasks

It couldn't run pnpm install lmao. Opus 4.5 was a crazy jump

gpt-oss is about a year older than qwen 3.8 27b

How much RAM? And what do you use it for if I might ask?

how quick does it respond? what are specs of your laptop?

I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.

Had the same reaction so had Grok create a script to:

- find a free GPU droplet on digital ocean

- fire it up

- pull in a snapshot of the model + extra files/packages etc

- set up a ssh tunnel so that the localhost:8000 routes to the above

Then I just configured OpenCode to use the above and was off to the races.

Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)

> Works out to be about ~$2/hr all said

What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.

It's actually a bit less than $2 but I rounded up just to make the math easier.

And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.

> It's actually a bit less than $2 but I rounded up just to make the math easier.

That's cool, what actual GPU though? I'm still curious :P

gpu-6000adax1-48gb RTX 6000 Ada GPU Droplet - 1X

Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model.

I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure.

(persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)

> the only difference is I setup a persistent drive,

Going to try this out vs the snapshot!

I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.

Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?

It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that

if you are so sure about what the final shape of your output is then its prbly not a common use of ai

If you are sure about the final shape of your output it's a great use case for AI, as you can define what you want in your prompt and refine towards it!

It's where you don't know the end state you're looking for that you'll end up generating slop on top of slop and creating a whole Gastown just to power your Gastown.

Reread what he said maybe?