People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.

Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.

For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

It seems, and that seems is entirely my unvalidated impression, that Ollama lags in features, as they're integrating after the fact those changes. But (seriously) an LLM told me that, when some aspects of MoE models were better supported with the latest llama.

And it did in that case make a significant difference.

It's very far behind llama.cpp, vLLM and SGLang in features yes. In part because of that but also due to some poor default settings it generally performs a lot worse as well.

I know nothing about "vLLM" -> anything better compared to "llama.cpp"?

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

> is there something fundamentally wrong with Ollama ?

Yes https://news.ycombinator.com/item?id=47788385

Maybe some were fixed, but:

1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length

2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`

3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.

4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.

5) Often meaningfully slower than llama.cpp

6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.

> (and also because VLLM is not super Windows friendly)

llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run

  llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL

will pull the model you expect from hugging face (or -m for manually downloaded .gguf) and start web server with API and HTML chat app.

7) can't use reranker with it, asked by people for one year or two... 8) ...

Thank you for your responses. They've been very helpful. I'm going to test the llama.cpp binaries directly. That seems to me to be the closest to what Ollama is trying to achieve, after all.

Are you confusing llama (the model from meta that popularised a lot of open AI stuff but is quite dated now) and ollama (the set of tools for running models, originally llama but no longer limited to it, not maintained by meta)