# Local LLMs in 2026: Which Models Run on Your Hardware

> How to run an LLM locally: size models by memory with the weights plus KV cache formula, compare Qwen, Gemma 4 and gpt-oss, licenses, M4 Max tok/s.
> Author: Roman Belov · Published: 2026-10-10 · Source: https://futurecraft.pro/blog/local-llm-guide/

A local LLM is a language model with open weights that runs on your computer without a cloud API. The weights are downloaded and computed in your own memory, so three things decide which model to pick: memory size (weights plus the context KV cache), the license, and whether speed or quality matters more for the task.

People usually start with a "best models" ranking. They end up with a model that doesn't fit in memory, or fits with a 4k-token context. The Qwen3.8-27B model card says 262,144 tokens of context, and the Ollama tag page says the file is 18 GB. Neither says how much memory the model will take on your machine. On a Mac there's also a GPU memory limit with no official formula, and the same Ollama tag gives you different files on a Mac and on Linux.

This guide shows how to pick a local model by calculation instead of by ranking: the "weights + KV cache" memory formula checked against Ollama measurements, a "how much memory, which models" table with context and licenses, an overview of the Qwen, Gemma 4, gpt-oss, DeepSeek, Mistral and Muse families, runners, and measurements on a Mac with an Apple M4 Max and 36 GB of memory. Installing Ollama, its API and the Modelfile are covered separately in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/).

## What a local LLM is and when it makes sense

A runner launches the local model: Ollama, llama.cpp, LM Studio, vLLM or MLX. It loads the weights into VRAM (unified memory on a Mac), holds the context KV cache and serves an API, usually OpenAI-compatible. Apps and agents connect to it the same way they connect to a cloud provider. Only the `base_url` points at localhost.

| | Local model | Cloud API |
|---|---|---|
| Data | Stays on your machine | Goes to the provider |
| Cost | Hardware and electricity | Pay per token |
| Quality | Below frontier models, grows with size | Frontier models: Opus 5.5, Sonnet 5.5, Fable 5.1 |
| Context | Limited by memory, not by the model card | Hundreds of thousands of tokens and more |
| Offline | Yes, once the weights are downloaded | No |

When a local model makes sense:

- the data can't leave the machine: personal data, internal logs, code under NDA. The [Ollama FAQ](https://docs.ollama.com/faq) says Ollama doesn't see your prompts or data when you run locally;
- the task is high-volume and simple: classification, field extraction, draft summaries, embeddings for search;
- you need to work offline or experiment without a token bill;
- you want a predictable backup for when a cloud provider is down.

When it doesn't: multi-step agent tasks and complex refactoring. In an OpenCode test, local 7-8B models handled short edits, but none of them reliably solved a multi-step task. Details are in the [OpenCode guide](https://futurecraft.pro/blog/opencode-guide/). In production, a local model usually becomes one provider next to the cloud ones. How to split tasks between them and build fallback is covered in [Multi-Provider LLM Routing](https://futurecraft.pro/blog/multi-provider-llm-architecture/) and [Production LLM Stack](https://futurecraft.pro/blog/production-llm-stack/).

## How much memory a local LLM needs: the formula

Model memory is the sum of the weights, the context KV cache and the runner's working buffers. You can calculate the first two. The third only shows up in a measurement.

### Weights and quantization

Weights = parameters × bits per weight / 8. Quantization lowers the bits per weight. Names like "Q4" and "Q8" are rounded: llama.cpp quantizations keep some tensors at higher precision, so the real figure is higher. Here's the table from the [`llama-quantize` docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), using Llama 3.1 8B as the example:

| Quantization | Bits per weight | Size for 8B |
|---|---|---|
| Q4_K_M | 4.89 | 4.58 GiB |
| Q5_K_M | 5.70 | 5.33 GiB |
| Q6_K | 6.56 | 6.14 GiB |
| Q8_0 | 8.50 | 7.95 GiB |
| F16 | 16.0 | 14.96 GiB |

That gives a quick rule for Q4_K_M: about 0.6 GB per billion parameters. For 27B that's ~17 GB, and the `qwen3.8:27b-q4_K_M` file on ollama.com is 18 GB because it also contains a vision encoder. Q8_0 is almost twice as large, 30 GB for the same model. BF16 is 56 GB.

Besides llama.cpp's GGUF quantizations, Apple Silicon has MLX formats. In the Ollama library these are tags with the suffixes `nvfp4`, `mxfp8` and `mlx`. NVFP4 is a 4-bit NVIDIA format, and the file size is close to Q4_K_M: `qwen3.8:27b-nvfp4` is also 18 GB. According to the [Ollama blog](https://ollama.com/blog/mlx-performance), on Gemma 4 12B the quality loss relative to BF16 (by perplexity) is about half as large with NVFP4 as with q4_K_M. That's a vendor measurement on one model.

### Context KV cache

For each context token, the model stores keys and values (K and V) in every full-attention layer:

```text
KV per token = 2 × full_attention_layers × KV_heads × head_dim × bytes_per_element
KV total     = KV per token × context tokens
```

The parameters come from the model's `config.json` on Hugging Face. By default Ollama stores the cache in f16, which is 2 bytes per element. Here's the calculation checked against the measurements from the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/) (Ollama 0.40.14, 32768 context, M4 Max):

| Model | Layers / KV heads / head_dim | KV per token | KV at 32k | Memory minus file (`ollama ps`) |
|---|---|---|---|---|
| qwen2.5-coder:7b | 28 / 4 / 128 | 56 KiB | 1.75 GiB | 6.6 − 4.7 = 1.9 GB |
| qwen3:8b | 36 / 8 / 128 | 144 KiB | 4.5 GiB | 9.9 − 5.2 = 4.7 GB |

For a standard transformer with grouped-query attention, the calculation matches the measurement within 0.2 GB. For qwen3:8b the KV cache at 32k is almost as large as the weights. qwen2.5-coder:7b has half as many KV heads and 28 layers instead of 36, so its cache is 2.5 times smaller, even though the two models are nearly the same size.

Checking the calculation against `ollama ps` doesn't work for every model. In the test below, for models on the llamacpp runner, the SIZE column turned out not to be comparable with the file: 1.6 GB for `gemma4:26b-a4b` with an 18 GB file. Before you check, look at RUNNER in `ollama ps`. For llama.cpp models, count usage as the file plus the runner process footprint.

### 256K on the model card isn't 256K in your memory

Newer architectures keep full attention in only some layers, so their context is cheaper. The table is calculated from the models' `config.json` on Hugging Face and isn't confirmed by measurement:

| Model | Attention layout | KV per token (estimate) | At 32k | At maximum |
|---|---|---|---|---|
| Qwen3.8-27B | 64 layers: 16 full-attention (4 KV heads, head_dim 256), 48 Gated DeltaNet with a constant state | 64 KiB | ~2 GiB | ~16 GiB at 256K |
| Gemma 4 31B | 10 global layers (4 KV heads, head_dim 512) + 50 with a 1024-token sliding window | 40-80 KiB + ~0.8 GiB for the windows | ~2-3.3 GiB | ~11-21 GiB at 256K |
| Gemma 4 26B-A4B | 5 global layers (2 KV heads, head_dim 512) + 25 with a sliding window | 10-20 KiB + ~0.2 GiB | ~0.5-0.8 GiB | ~2.7-5.2 GiB at 256K |
| gpt-oss-20b | 12 full-attention layers (8 KV heads, head_dim 64) + 12 with a 128-token sliding window | 24 KiB | ~0.75 GiB | ~3 GiB at 128K |

The Gemma 4 range comes from the `attention_k_eq_v` flag in the config. Keys and values are shared in the global layers, and if the runner uses that, the cache is half the size.

On a 36 GB Mac, where the GPU sees 28.1 GiB, this difference gets practical. Qwen3.8-27B in 4 bits (18 GB) with the full 256K window would need another ~16 GiB for the cache and won't fit in the limit. With a 32k context it fits, but barely.

In the November 7 test, `qwen3.8:27b-nvfp4` with a 32768 context ran at 100% GPU and took 18 GB in `ollama ps` right after loading, then 25 GB after runs with an 8.9K-token prompt. By calculation, the KV cache for the full 32k is about 2 GiB, so most of the growth comes from the MLX runner's working buffers. Ollama doesn't show the breakdown. That's why the headroom can't be set as a single number: you check it on your own machine with your own long prompts.

### KV cache quantization and Flash Attention

You can store the cache at lower precision. Per the [Ollama FAQ](https://docs.ollama.com/faq):

- `OLLAMA_KV_CACHE_TYPE=q8_0` takes about half the memory of f16, `q4_0` about a quarter;
- cache quantization only works with Flash Attention on (Ollama turns it on by itself if the backend supports it);
- the setting is global and applies to every model on the server;
- memory grows in proportion to `OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH`, so each parallel stream gets its own cache.

The FAQ describes Ollama's llama.cpp-based engine. Whether the variable affects the MLX runner wasn't tested for this guide.

## Hardware for local LLMs: Mac, NVIDIA, CPU

Generation speed for a local model is bound by memory: for each new token, the runner reads all the active weights from memory. That gives an upper estimate: memory bandwidth divided by the size of the active weights in bytes. It's a ceiling, and real speed is usually lower. Speculative decoding can push past it, when the model accepts more than one token per read of the weights.

The test M4 Max with 36 GB has 410 GB/s of bandwidth. For qwen2.5-coder:7b with a 4.7 GB file the ceiling is about 87 tokens per second, and the measurement on Ollama 0.40.14 gave 69.5, about 80% of the ceiling. Prompt processing (prefill) computes all tokens in parallel and depends mostly on GPU compute.

### Apple Silicon

A Mac shares memory between the CPU and GPU (unified memory), so it fits models that would need 24-48 GB of VRAM on a discrete card. The M4 Max comes with up to 128 GB and up to 546 GB/s of bandwidth, the M4 Pro with up to 64 GB and 273 GB/s ([Apple Newsroom](https://www.apple.com/newsroom/2024/10/apple-introduces-m4-pro-and-m4-max/)). The 546 GB/s figure is for the top M4 Max with the 16-core CPU. The 36 GB configuration used for the measurements in this guide only comes with the 14-core CPU and 32-core GPU, and it has [410 GB/s](https://support.apple.com/en-us/121553).

The GPU doesn't get all the memory. On an M4 Max with 36 GB, the Ollama server writes `total_vram=28.1 GiB` to its log, about 78% of the total. Neither Apple nor Ollama publishes a formula for this share, so on machines with a different amount of memory, check the server log. The system limit `iogpu.wired_limit_mb` isn't overridden on the test machine (value 0). Changing it wasn't tried for this guide, and it can't be recommended without testing.

`total_vram` also sets Ollama's default context ([docs.ollama.com/context-length](https://docs.ollama.com/context-length)): below 24 GiB it's 4k tokens, 24-48 GiB gives 32k, and 48 GiB and up gives 256k. A 36 GB Mac lands in the middle range and gets 32768.

The second path on a Mac is MLX, Apple's machine learning framework. Ollama introduced an MLX-based engine for Apple Silicon in [version 0.19](https://ollama.com/blog/mlx) (March 30, 2026, preview) and required a Mac with more than 32 GB of unified memory for it. The vendor's measurement on Qwen3.5-35B-A3B: generation 58 → 112 tok/s and prompt processing 1154 → 1810 tok/s going from Ollama 0.18 (Q4_K_M) to 0.19 (NVFP4). The post doesn't name the chip, so these numbers don't transfer to your machine. LM Studio on Apple Silicon runs models through llama.cpp or MLX ([LM Studio docs](https://lmstudio.ai/docs/app)).

The practical consequence was checked on the test machine. At the time of the November 7 test (Ollama 0.40.14), the `qwen3.8:27b` tag on a Mac downloads the MLX build. `ollama show` reports nvfp4 quantization, the safetensors format and a minimum Ollama version of 0.32.12, and `ollama ps` shows the `mlx` runner. To get GGUF in Q4_K_M on the same machine, you have to request the explicit tag `qwen3.8:27b-q4_K_M`.

The manifest for the same `qwen3.8:27b` tag, requested from something other than a Mac, points to GGUF. The same tag gives different files on different platforms, so compare speed across machines by explicit tags that include the quantization.

### NVIDIA and AMD

On NVIDIA the constraint is stricter: the model and the cache have to fit in the card's VRAM. Per the [Ollama docs](https://docs.ollama.com/gpu), you need compute capability 5.0+ and driver 550 or newer (570+ for cards with compute capability 5.0-6.2). The RTX 5090 has 32 GB of GDDR7 on a 512-bit bus ([NVIDIA](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/)). For other cards, think in VRAM classes of 8, 12, 16, 24 and 32 GB and check the amount in your card's spec sheet.

AMD is supported through ROCm v7 on Linux and Windows. Windows and Linux also have a Vulkan backend, which is on by default if it's installed.

If a model doesn't fit in VRAM entirely, llama.cpp and Ollama split the layers between the GPU and system RAM. The PROCESSOR column in `ollama ps` then shows a split like `48%/52% CPU/GPU` instead of `100% GPU`. Speed drops, because part of the weights is read from slower memory. For MoE the split usually costs less, since each token reads only a small share of the weights. How much speed drops in either case wasn't measured for this guide.

### CPU-only

llama.cpp is a "Plain C/C++ implementation without any dependencies", per its [README](https://github.com/ggml-org/llama.cpp). It runs the model on the CPU when there's no GPU and has a hybrid CPU+GPU mode for models larger than VRAM. On a CPU, generation is limited by RAM bandwidth, and on a typical PC that's lower than VRAM or a Mac's unified memory. That's enough for chat on 1-4B models, but not for agents or long context. This guide has no CPU measurements of its own, and the numbers in online reviews come without a methodology.

## How much memory, which models

The table is built from the tag pages in [ollama.com/library](https://ollama.com/library) as of November 7, 2026. Sizes are 4-bit file sizes (Q4_K_M, QAT or MXFP4); the KV cache and runner buffers come on top. The "GPU memory" column is what the GPU sees: on a Mac that's roughly three quarters of unified memory (observed on one machine), on NVIDIA it's the VRAM.

| GPU memory | Models (tag, file size) | Type | Context | License |
|---|---|---|---|---|
| 8 GB | `gemma4:e2b-it-qat` (4.3 GB), `qwen3.5:4b` (3.3-4.0 GB) | dense | 128K / 256K | Apache 2.0 |
| 12-16 GB | `gemma4:e4b-it-qat` (6.1 GB), `qwen3.5:9b` (6.6-7.6 GB), `gemma4:12b-it-qat` (7.2 GB) | dense | 128K / 256K | Apache 2.0 |
| 16 GB, at the limit | `gpt-oss:20b` (14 GB) | MoE, 3.6B active | 128K | Apache 2.0 |
| 24 GB | `qwen3.8:27b-q4_K_M` (18 GB), `gemma4:26b-a4b-it-q4_K_M` (18 GB), `muse-glimmer:30b` (18 GB), `gemma4:31b-it-q4_K_M` (20 GB) | dense; Gemma 26B is MoE, 3.8B active | 128K-256K | Apache 2.0 |
| 28-32 GB (36 GB Mac, RTX 5090) | the same with a 32-64k context, `qwen3.6:35b-a3b` (23 GB) | Qwen3.6 35B is MoE, 3B active | 256K | Apache 2.0 |
| 48-64 GB | `qwen3.8:27b-q8_0` (30 GB), `gemma4:31b-it-q8_0` (34 GB), `qwen3.6:35b-a3b-q8_0` (39 GB) | dense and MoE in 8 bits | 256K | Apache 2.0 |
| 80-128 GB | `gpt-oss:120b` (65 GB), `llama4:scout` (67 GB), `qwen3.5:122b` (81 GB), BF16 versions of 27-31B (56-64 GB) | MoE and dense BF16 | 128K-10M | Apache 2.0; Llama 4 has its own |
| GPU cluster | DeepSeek V4.1 Flash, Mistral Large 4, Muse Spark Open | MoE, hundreds of billions of parameters and up | up to 1M | MIT; Mistral and Muse have their own |

How to read the table:

- Tags with a size range (`qwen3.5:9b`, `gemma4:e4b`) have different files for different platforms. A range doesn't mean a Mac gets MLX: on the test M4 Max, `gemma4:e4b` downloaded a 9.6 GB GGUF with the llamacpp runner. `ollama show` tells you which file you got.
- OpenAI's model card says `gpt-oss:20b` runs "within 16GB of memory". On a 16 GB Mac the GPU sees less than 16 GB, so on that machine the model is at the limit.
- On a 36 GB Mac, `qwen3.8:27b-q8_0` (30 GB) doesn't fit in the 28.1 GiB GPU limit even without context. That's a conclusion from file sizes, not a measurement.
- A dense model and a MoE model with the same file size take the same memory, but the MoE reads several times fewer weights per token and so generates faster. In the test: 49.8 tok/s for `gemma4:26b-a4b` versus 10.6 for `qwen3.8:27b-q4_K_M`.

Separate Qwen tables and a short "memory → models" guide for Ollama are in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/).

## Open model families as of November 2026

Dates below are when the weights were published on Hugging Face or when the vendor announced them. Version numbers change every few months, so open the model card and the tag page before you choose.

### Qwen

The widest range of sizes. All the local models listed here accept images and support tool calls and reasoning mode (the vision, tools and thinking labels in Ollama).

- **Qwen3.8-27B** (August 2026, Apache 2.0) is a dense 27B model with hybrid attention: 48 Gated DeltaNet layers and 16 standard attention layers. Native context is 262,144 tokens, extendable to 1M. In Ollama: `qwen3.8:27b` (18 GB; on a Mac this is the MLX build), `q4_K_M` (18 GB), `q8_0` (30 GB), `bf16` (56 GB), and the MLX variants `nvfp4` (18 GB) and `mxfp8` (32 GB).
- **Qwen3.8-Flash-Next** (August 2026) is a MoE with 125B parameters and 6B active. The license is `qwen-community-1.0`, not Apache 2.0. The Ollama library had no tag for it as of November 7.
- **Qwen3.6** (April 2026, Apache 2.0) has a dense 27B and a MoE 35B-A3B (3B active), with 256K context. Ollama has tags with a `coding` suffix; the model page doesn't say what's changed in those builds.
- **Qwen3.5** (February 2026, Apache 2.0) is a lineup from 0.8B to 122B-A10B, including sizes for 8-16 GB machines.

### Gemma 4

Google announced Gemma 4 on [April 2, 2026](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) under Apache 2.0: the "effective" E2B and E4B for phones and edge devices (128K context), the MoE 26B-A4B (3.8B active) and the dense 31B (256K). A 12B showed up on Hugging Face in May. According to Google, 26B and 31B in BF16 fit on a single 80 GB H100, and the quantized versions run on consumer GPUs. In Ollama, Gemma 4 has QAT tags (quantization-aware training, files smaller than regular Q4) and MLX variants.

### gpt-oss

OpenAI's open weights from August 2025, under Apache 2.0. `gpt-oss-20b` is a MoE with 21B parameters and 3.6B active, and `gpt-oss-120b` has 117B with 5.1B active ([model card](https://huggingface.co/openai/gpt-oss-20b)). Expert weights are stored in MXFP4, context is 128K, and you set the reasoning level in the system prompt (low, medium, high). Per the card, 20b runs in 16 GB of memory and 120b on a single 80 GB GPU. In Ollama these are `gpt-oss:20b` (14 GB) and `gpt-oss:120b` (65 GB), with no MLX tags.

### DeepSeek

DeepSeek V4.1 Flash (September 2026, MIT, 1M context) is a MoE with hundreds of billions of parameters. The weights are open, but Ollama has the model only as a cloud model, with no file sizes. The same goes for DeepSeek-V4-Pro. The DeepSeek models people run locally are R1 and its distills (`deepseek-r1` from 1.5b to 70b, 94.8M downloads in Ollama). The distills are built on Qwen and Llama base models, so check the license on the card for each size.

### Mistral and Devstral

- **Mistral Small 4** (January 2026, `2603` update in March, Apache 2.0) is a MoE with 119B parameters and 6.5B active, 256K context. The Ollama library had nothing under that name as of November 7.
- **Devstral Small 2** (November 2025, Apache 2.0) is a 24B model for code with 256K context. On the card, the vendor says it runs on a single RTX 4090 or a Mac with 32 GB.
- **Mistral Large 4** (September 30, 2026) is a MoE with about 1T parameters and ~52B active, 1M context. The launch examples on the card assume 8 GPUs, and in Ollama the model is cloud-only. The license is the Mistral Ecosystem Open-Weight License; read its text before use.

### Llama and Muse

The last lineup under the Llama brand in the meta-llama organization on Hugging Face is Llama 4 (April 2025): Scout (17B active, 16 experts, 109B total) and Maverick (128 experts). In Ollama, `llama4:scout` is 67 GB and `llama4:maverick` is 245 GB. The license is the Llama 4 Community License, and access to the weights on HF is by request. `llama3.1` and `llama3.2` still lead Ollama downloads, but they're 2024 models.

Meta's newer open weights come out under the Muse brand. **Muse Glimmer 30B** (August 2026, Apache 2.0) has 29.6B parameters with a vision encoder and a context of 131,072+. The card lists hardware targets of 24, 32 and 64 GB of VRAM, and Ollama has the model as `muse-glimmer:30b` (18 GB). **Muse Spark Open** (816B-A42B, September 2026) is released under its own license with manual access approval, and in Ollama it's cloud-only.

### Open, but not local

Some models have open weights but won't run on a single machine: DeepSeek V4.1 Flash and V4-Pro, Mistral Large 4, Muse Spark Open and the 2.4T-parameter Qwen3.8 flagship. If you have one Mac or one graphics card, these are cloud models for you: through Ollama's cloud, the vendor's API or an aggregator. Running them yourself takes a multi-GPU server and a server runner. The Mistral Large 4 card, for example, recommends vLLM and SGLang and gives examples on 8 GPUs.

## Dense vs MoE: memory and speed

A name like `35B-A3B` means 35 billion parameters in total and 3 billion active per token. A MoE (mixture of experts) model keeps all its weights in memory, but for each token a router picks a few experts, and only they are computed.

Two consequences for choosing a model:

1. **Memory decides what fits.** `gemma4:26b-a4b-it-q4_K_M` and `qwen3.8:27b-q4_K_M` are both 18 GB and need the same space in memory.
2. **Active parameters decide generation speed.** Per token, the MoE reads about 3.8B parameters from memory instead of 27B. By the "bandwidth / active bytes" estimate, its ceiling is several times higher than a dense model of the same size: on the test M4 Max (410 GB/s), about 23 tok/s for a 27B dense model in Q4 versus about 180 for 26B-A4B. These are theoretical ceilings. In the test, the GGUF builds gave 10.6 and 49.8 tok/s: a gap of 4.7 times, smaller than in theory but of the same order.

A MoE and a dense model with the same file size differ in quality, and vendor self-reports with different harnesses don't allow a fair comparison. Compare models on your own prompts for your own tasks. Speed for five models on one machine is in the "Benchmark on an M4 Max with 36 GB" section.

## Tools to run an LLM locally

| Tool | What it is | License | When to pick it |
|---|---|---|---|
| Ollama | CLI and server, REST and OpenAI-compatible API, model library, MLX on Apple Silicon | MIT | A local API for your own code and agents |
| LM Studio | Desktop app and the headless `llmster` server, OpenAI-compatible endpoints, llama.cpp and MLX | see the terms on the website | Working without a CLI, browsing models in a GUI |
| llama.cpp | The base of many runners, GGUF, `llama-server` with an OpenAI-compatible API, 17 backends | MIT | Full control over quants and the build |
| vLLM | A GPU server runner with request batching | Apache 2.0 | An NVIDIA server for several users |
| Jan | Desktop app | see the `janhq/jan` repository | A GUI alternative to LM Studio |
| MLX / mlx-lm | Apple's framework and a package for running LLMs | MIT | Apple Silicon, when you want Python and your own experiments |

Version numbers aren't in the table: runners update faster than a table can keep up, and llama.cpp builds are numbered with `bNNNN` tags anyway. Ollama installation, API and settings are in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/), and the two most popular desktop options are compared in [LM Studio vs Ollama](https://futurecraft.pro/blog/lm-studio-vs-ollama/).

## Benchmark on an M4 Max with 36 GB

The goal of the test is to check two conclusions of this guide on one machine: how much faster a MoE is than a dense model with the same file size, and what Ollama's MLX path gives over GGUF on the same model.

**Verified:** November 7, 2026, Ollama 0.40.14, Mac with Apple M4 Max (14-core CPU, 32-core GPU, 410 GB/s) and 36 GB of unified memory, GPU sees 28.1 GiB, macOS 27.2, on AC power. Heavy apps were closed before the test. Claude and system processes stayed open, about 17 GB of memory without a model. Swap was at 7 GB of 8 at the start after earlier runs and never fully cleared.

Methodology:

- 32768 context (`num_ctx`) for all models, temperature 0, reasoning off (`think: false`), up to 400 response tokens;
- two prompts: short (~55 tokens, "explain the KV cache in about 200 words") and long (~8.9K tokens of text with a request to summarize it in 5 bullet points);
- 5 runs per prompt; the table shows the median, with the minimum and maximum in parentheses;
- a unique prefix before each run, so the prompt cache doesn't inflate prompt processing speed;
- speeds come from the server's response fields: generation as `eval_count / eval_duration`, prompt processing as `prompt_eval_count / prompt_eval_duration`; TTFT is the time to the first token in the stream;
- all models were unloaded between models, and for each one the increase in `Pageouts + Swapouts` from `vm_stat` during the test (16 KB pages) was recorded, to see whether the system went into swap.

| Model (tag) | Type | Format / RUNNER | Generation, short prompt, tok/s | Generation, long prompt, tok/s | Prompt processing (~8.9K), tok/s | TTFT, long prompt, s | Swap during test |
|---|---|---|---|---|---|---|---|
| `qwen3.8:27b-nvfp4` | dense 27B | safetensors nvfp4 / mlx | 32.4 (30.4-35.5) | 19.4 (14.2-19.7) | 104 | 85.7 | ≈ 11 GB |
| `qwen3.8:27b-q4_K_M` | dense 27B | GGUF Q4_K_M / llamacpp | 10.6 (9.6-10.7) | 10.1 (9.3-10.2) | 115 | 77.2 | ≈ 0.8 GB |
| `gemma4:26b-a4b-it-q4_K_M` | MoE 26B, 3.8B active | GGUF Q4_K_M / llamacpp | 49.8 (48.7-52.0) | 49.3 (48.7-50.5) | 706 | 13.0 | ~0 |
| `gpt-oss:20b` | MoE 21B, 3.6B active | GGUF MXFP4 / llamacpp | 45.3 (44.7-47.9) | 47.2 (46.2-47.6) | 862 | 11.7 | ~0 |
| `gemma4:e4b` | dense, 4.5B effective | GGUF / llamacpp | 75.3 (49.5-76.1) | 46.9 (45.5-50.0) | 679 | 13.4 | ~0 |

The long prompt has a different length for each model because of the tokenizers: 8,887 tokens for Qwen, 9,044 for Gemma 4 26B, 8,344 for gpt-oss, 9,040 for gemma4:e4b. Why the same text yields a different token count per model is covered in [what tokens are and how to count them](https://futurecraft.pro/blog/tokens-explained/). The wide spread for `gemma4:e4b` on the short prompt comes from the first run (49.5 tok/s). The rest were around 75.

On this machine the `qwen3.8:27b` tag without a suffix points to the same MLX build as `qwen3.8:27b-nvfp4`, so the GGUF for comparison was requested with the explicit `qwen3.8:27b-q4_K_M` tag.

### Memory: why `ollama ps` doesn't work for estimates

In Ollama 0.40.14, for models on the llamacpp runner, the SIZE column in `ollama ps` isn't comparable with the file: 1.6 GB for `gemma4:26b-a4b` with an 18 GB file, 1.1 GB for `gemma4:e4b` with 9.6 GB. The reason is how the runners hold weights. llama.cpp maps the weights file into memory (mmap), and those pages don't count toward the process footprint. MLX keeps the weights in the process's own memory, so for the MLX build SIZE is closer to reality, but it grows as the model runs.

Runner process footprint (`phys_footprint` from the macOS `footprint` utility) after a short request with a 32768 context:

| Model | Model file | Runner footprint | Estimated usage | `ollama ps` |
|---|---|---|---|---|
| `qwen3.8:27b-nvfp4` (mlx) | ~18 GB | 17 GB | 17 GB, weights inside the process | 18 GB, 25 GB after long runs |
| `qwen3.8:27b-q4_K_M` | 16.8 GB + 0.9 GB vision projector | 3.5 GB | ≈ 21.2 GB | 17 GB |
| `gemma4:26b-a4b` | 18.0 GB | 2.4 GB | ≈ 20.4 GB | 1.6 GB |
| `gpt-oss:20b` | 13.8 GB | 0.9 GB | ≈ 14.7 GB | 12 GB |
| `gemma4:e4b` | 9.6 GB | 1.7 GB | ≈ 11.3 GB | 1.1 GB |

For llama.cpp models, real usage is roughly the file size plus the runner footprint, which holds the KV cache and buffers. For MLX, look at the runner footprint or Activity Monitor. On this version, SIZE in `ollama ps` can't tell you whether a model will fit.

For the MLX Qwen build, memory grew from 18 GB after loading to 25 GB after long runs with the same context. Together with 17 GB of open apps that's more than 36 GB, and during the test the system pushed about 11 GB into swap (676,000 pages). The GGUF build of the same model barely touched swap (0.8 GB), even though by estimate it takes no less memory. One explanation fits how the runners work: the system can drop pages of an mmap'd file and reread them from disk later without writing to swap, while MLX process memory has to be swapped out.

### What the test showed

MoE versus dense held up with room to spare. With the same file size of about 18 GB, `gemma4:26b-a4b` generates 49.8 tok/s and `qwen3.8:27b-q4_K_M`, in the same format on the same runner, 10.6 tok/s: a difference of 4.7 times. On the long prompt the gap is even bigger: prompt processing at 706 versus 115 tok/s, TTFT at 13.0 versus 77.2 seconds. `gpt-oss:20b` behaves the same way. Memory decides whether a model fits, and active parameters decide how fast it answers.

MLX versus GGUF on the same model gave a gain, but only with memory to spare. On the short prompt, `nvfp4` is three times faster than `q4_K_M`: 32.4 versus 10.6 tok/s. On the long one the gain shrank to 1.9 times (19.4 versus 10.1), and the spread between runs widened to 14.2-19.7. The MLX build ran out of memory, and the system went into swap. The MLX path didn't speed up long-prompt processing on this model: 104 versus 115 tok/s.

32.4 tok/s is above the upper estimate for an 18 GB file at 410 GB/s (about 23 tok/s), and the Ollama server log shows why. The MLX runner turns on MTP (multi-token prediction) speculative decoding for `qwen3.8:27b` by itself: over the run, the median acceptance was 0.67 and `avg_accepted` was 1.29 tokens per iteration on top of the main one. The llama.cpp runner has no MTP. So nvfp4's gain over q4_K_M combines the format, the engine and MTP, and this test can't separate their contributions.

On a 36 GB Mac, a dense 27B in 4 bits fits in the GPU limit, but with apps open it runs on the edge, and on long prompts the MLX build loses most of its advantage. A MoE with the same file size is several times faster here and doesn't touch swap. On a Mac with more than 32 GB, it makes sense to start with MLX tags if the model fits with a few gigabytes to spare.

These numbers apply to one 36 GB machine with open apps and swap in use, to Ollama 0.40.14 and a 32K context. On a Mac with 64 GB or more, on other chips and on NVIDIA the picture may differ; that wasn't tested. Also untested: parallel requests, speed at a 64K context, agents running on these models, and thermals under sustained load.

## Which local LLM to pick for the task

There's no "best model" ranking here. Vendors don't publish comparisons you can verify, and the benchmarks on model cards were run with different harnesses. Each type of task has its own memory, context and capability requirements.

**Chat and text work.** Take the largest model that fits in memory with the context you need. On 16 GB that's 4-12B (Qwen3.5, Gemma 4); on 24-36 GB it's 26-31B in 4 bits.

**Code in chat.** You need a long context, since files and diffs eat tens of thousands of tokens. Ollama has separate coding tags (`qwen3.6:*-coding`, `qwen3-coder`), and Mistral has Devstral Small 2. Check the quality on your own tasks.

**Agents.** A model for code and a model for an agent have different requirements. An agent needs stable tool calls in the right format and a context of 64000 tokens or more: the [Ollama docs](https://docs.ollama.com/context-length) recommend that threshold for agents and coding tools. In the OpenCode test, qwen2.5-coder:7b with the `tools` label printed tool calls as text, and 8B models passed the task inconsistently ([OpenCode guide](https://futurecraft.pro/blog/opencode-guide/)). The [Ollama guide](https://futurecraft.pro/blog/ollama-guide/) shows how to connect Claude Code to a local model through Ollama, though Anthropic doesn't officially support that setup.

**Embeddings and RAG.** You don't need a chat model for this: Ollama has separate models with the `embedding` label, such as `nomic-embed-text` and `qwen3-embedding`. The [RAG guide](https://futurecraft.pro/blog/rag-system-guide/) explains how to choose embeddings and build retrieval, and [Context Engineering vs RAG](https://futurecraft.pro/blog/context-engineering-vs-rag/) covers when a long context is enough without RAG.

**Images.** Qwen3.5-3.8, Gemma 4 and Muse Glimmer 30B accept images as input. Look for the `vision` label on the model's library page.

## Licenses: what to check before commercial use

"Open weights" and "open source" are different things. You can download the weights, but the license of the specific model sets the terms of use. The licenses below come from the `license` field and the card text on Hugging Face. This isn't legal advice.

| License | Models |
|---|---|
| Apache 2.0 | Qwen3.8-27B, Qwen3.6, Qwen3.5, Gemma 4, gpt-oss, Mistral Small 4, Devstral Small 2, Muse Glimmer 30B |
| MIT | DeepSeek V4.1 Flash, DeepSeek-V4-Pro, DeepSeek R1 (for distills, check the base model) |
| Vendor's own license | Qwen3.8-Flash-Next (`qwen-community-1.0`), the 2.4T Qwen3.8 flagship, Llama 4 (Llama 4 Community License), Mistral Large 4 (Mistral Ecosystem Open-Weight License), Mistral Medium 3.5, Muse Spark Open |

Before you build a product on a local model:

1. Open the card for the specific model and size: Qwen3.8 27B and Flash-Next have different licenses.
2. For models in the third row, read the full license text and check it against your scenario: an internal tool, a product for customers, or the model as a service.
3. Check what a derived model is built on: a distill or fine-tune inherits the base model's terms.
4. Record the model version and a link to the license in the project docs.

## Local and cloud models together

A common setup: simple and sensitive tasks go to the local model, hard ones go to the cloud, and if the local server is down, the request goes to the cloud. You can build this at three levels:

- **in the application code**: routing by task type, retries and fallback, covered in [Multi-Provider LLM Routing](https://futurecraft.pro/blog/multi-provider-llm-architecture/);
- **through a self-hosted gateway**: [LiteLLM](https://futurecraft.pro/blog/litellm-guide/) describes Ollama and cloud providers in one `config.yaml` with fallback between them;
- **through a cloud aggregator**: [OpenRouter](https://futurecraft.pro/blog/openrouter-guide/) gives you one key for cloud models, including open models you can't run locally.

How to tie this to observability, budgets and evals is described in [Production LLM Stack](https://futurecraft.pro/blog/production-llm-stack/).

## Limitations and gotchas

### The default context depends on VRAM

On machines where the GPU sees less than 24 GiB, Ollama sets a 4k default context. An agent's system prompt doesn't fit, and Ollama silently truncates the input. Before you connect an agent, check the CONTEXT column in `ollama ps`.

### Context can't go above the native window

A `num_ctx` larger than the window the model was trained on doesn't work: for qwen3:8b, a request for 65536 got 40960 from Ollama. There's no error, and the real value only shows up in `ollama ps`. Details are in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/).

### One tag, different files

`qwen3.8:27b` on a Mac and on Linux are different builds (MLX nvfp4 and GGUF Q4_K_M). Speed and memory numbers from other people's tests are comparable only with the same explicit tag, runner version and context.

### `ollama ps` understates memory for llama.cpp models

In Ollama 0.40.14, for models on the llamacpp runner, the SIZE column can be several times smaller than the file: 1.6 GB for `gemma4:26b-a4b` with an 18 GB file. The weights are mapped from the file through mmap and don't count toward the process. Estimate usage as the file plus the runner footprint; for MLX models, look at the footprint or Activity Monitor.

### The Mac GPU memory limit isn't documented

28.1 GiB out of 36 GB is an observation on one machine. On a Mac with 16, 64 or 128 GB the share may differ, so check `total_vram` in the Ollama server log.

### Other people's speed numbers don't transfer

Vendor MLX measurements were taken on other chips and models. Even on one machine, the numbers change with the runner version: in the Ollama guide, the same qwen2.5-coder:7b took 6.6 GB instead of 8.2 GB after moving from Ollama 0.24.0 to 0.40.14.

### The table goes stale within months

Qwen released 3.5, 3.6 and 3.8 within six months, and some tags in the Ollama library were updated within the past week. The numbers in this guide are tied to the check date (November 7, 2026) and Ollama 0.40.14. For a real small model shipped inside a Mac product, see the [Ambient AI for Mac postmortem](https://futurecraft.pro/blog/ambient-ai-mac-tauri-swift-ollama-postmortem/).

## Where to start

1. Find out how much memory the GPU sees: on NVIDIA it's the VRAM, on a Mac it's `total_vram` in the Ollama server log.
2. Subtract memory for context: an agent needs 64k tokens or more, and you can calculate the KV cache from the model's `config.json`.
3. Pick a model from the "how much memory, which models" table, and prefer MoE if generation speed matters.
4. Check the license on the model card on Hugging Face.
5. Download the model with an explicit tag that includes the quantization and run `ollama run <tag> --verbose`.
6. Check `ollama ps`: CONTEXT, RUNNER and `100% GPU` in PROCESSOR. For models on llama.cpp, count memory as the file plus the runner footprint, because SIZE in `ollama ps` 0.40.14 isn't comparable with the file for them.
7. If the model can't handle the task, add a cloud model through fallback. A model larger than your available memory will partly spill to the CPU and lose speed.