# Ollama Guide: Install, Models, REST API and Modelfile

> Install Ollama on macOS, Windows and Linux, pick a model that fits your memory, use the CLI, REST and OpenAI-compatible APIs, and build a Modelfile.
> Author: Roman Belov · Published: 2026-10-05 · Source: https://futurecraft.pro/blog/ollama-guide/

Ollama is an open-source, MIT-licensed program for running language models on your own computer. It downloads models from the ollama.com library, manages them through a CLI and runs a local server on port 11434: a native REST API, an OpenAI-compatible `/v1` and an Anthropic-compatible `/v1/messages`. The [ollama/ollama](https://github.com/ollama/ollama) repository has more than 186,000 stars, and the latest release as of November 6, 2026 is v0.40.15.

Installing takes one command. A second one starts your first model. The trouble comes later. A model doesn't fit in memory. An agent loses the start of its prompt, because on a machine with less than 24 GiB of VRAM the default context is only 4k. `num_ctx` in a Modelfile gets silently capped. A "coding" model prints tool calls as text instead of running them.

This guide covers installing Ollama on macOS, Windows, Linux and in Docker, picking a model for your memory, working with it through the CLI, `/api` and `/v1`, tuning the context and building your own Modelfile. Speed and memory numbers come from a Mac with an Apple M4 Max and 36 GB of memory running Ollama 0.40.14. Where nothing was tested (Windows, Linux, NVIDIA), the text says so, and the commands come from the documentation.

## What Ollama is and when you need it

Ollama does three jobs. It downloads model weights from its library (or takes your own GGUF file). It runs the model on the GPU, and partly on the CPU if VRAM runs short. And it keeps an HTTP server running that the CLI, the Ollama app, your code and third-party tools talk to. If you're choosing between Ollama and LM Studio, see the [LM Studio vs Ollama comparison](https://futurecraft.pro/blog/lm-studio-vs-ollama/) with benchmarks on the same model.

How a local model compares with a cloud API:

| | Ollama (local) | Cloud API (Claude, GPT) |
|---|---|---|
| Data | Stays on your machine | Goes to the provider |
| Cost | Hardware and electricity | Pay per token |
| Quality | Below frontier models, depends on size | Frontier models |
| Speed | Depends on GPU and model size | Depends on provider and load |
| Availability | Works offline once the model is downloaded | Needs network, account, payment |

The [FAQ](https://docs.ollama.com/faq) is explicit about privacy: when you run locally, Ollama doesn't see your prompts or data. That's a common reason to pick a local model: classifying personal data, parsing logs, experimenting without a token bill. One example is the [Ambient AI for Mac postmortem](https://futurecraft.pro/blog/ambient-ai-mac-tauri-swift-ollama-postmortem/), where llama3.2:3b running through Ollama analyzes work patterns and no data goes to the cloud.

In a production architecture, a local model usually becomes one provider among several rather than a replacement for the cloud. How to route tasks between models and keep a fallback is covered in [Production LLM Stack](https://futurecraft.pro/blog/production-llm-stack/).

## How to install Ollama

The commands below come from the [README](https://github.com/ollama/ollama) and the [documentation](https://docs.ollama.com) as of November 2026. The macOS install was tested; the other platforms are described from the documentation.

### macOS

Requirements: macOS Sonoma (14) or newer. On Apple Silicon both CPU and GPU are used; Intel Macs run on CPU only ([docs.ollama.com/macos](https://docs.ollama.com/macos)).

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

The second option is to download Ollama.dmg and drag the app into Applications. On first launch the app offers to create a CLI link in `/usr/local/bin`.

Homebrew has an `ollama` formula in homebrew-core. The official documentation doesn't cover this method. The formula installs the CLI and server without the GUI app, and `brew services start ollama` runs the background service:

```bash
brew install ollama
brew services start ollama
```

### Windows

Per the [documentation](https://docs.ollama.com/windows): Windows 10 22H2 or newer, or Windows 11. NVIDIA needs driver 551.61+, AMD needs a stack with ROCm v7 support or the Radeon Vulkan driver. No admin rights are required; the installer puts Ollama in your home folder.

```powershell
irm https://ollama.com/install.ps1 | iex
```

Or download OllamaSetup.exe. To run Ollama as a service, there is the `ollama-windows-amd64.zip` archive.

### Linux

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

The script creates an `ollama` user and a systemd service. For a manual install, the [Linux page](https://docs.ollama.com/linux) lists `.tar.zst` archives for amd64 and arm64 plus a separate ROCm package for AMD. Managing the service:

```bash
sudo systemctl status ollama
journalctl -e -u ollama          # server logs
sudo systemctl edit ollama       # environment variables
```

To update on Linux, run the same script again.

### Docker

```bash
# CPU only
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU (requires NVIDIA Container Toolkit)
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker exec -it ollama ollama run llama3.2
```

For AMD there is the `ollama/ollama:rocm` image ([docs.ollama.com/docker](https://docs.ollama.com/docker)). On macOS the Docker route does nothing for speed: the [FAQ](https://docs.ollama.com/faq) states that Docker Desktop on macOS has no GPU acceleration. On a Mac, install Ollama natively.

### Checking the install

```bash
ollama --version
curl http://localhost:11434/api/version
# {"version":"0.40.14"}
```

If the CLI prints `could not connect to a running Ollama instance`, the server isn't running. Start it with the app, the systemd service, `brew services` or `ollama serve` in a separate terminal.

## Ollama models: how to pick one for your hardware

Memory is the main constraint. The model has to fit entirely in VRAM (on Apple Silicon, in the part of unified memory the GPU can see). Otherwise some layers move to the CPU and speed drops. Memory goes to both the weights and the context KV cache. The memory math, model families, and 27B benchmarks are in the [local LLM guide](https://futurecraft.pro/blog/local-llm-guide/).

A test on Ollama 0.40.14 shows the gap:

| Model | File | In memory (`ollama ps`, context 32768) |
|---|---|---|
| qwen2.5-coder:7b | 4.7 GB | 6.6 GB |
| qwen3:8b | 5.2 GB | 9.9 GB |

On macOS the GPU doesn't get all of the memory. On a 36 GB machine, the Ollama server reported `total_vram=28.1 GiB` in its log.

The size guide below is based on the tag pages at [ollama.com/library](https://ollama.com/library) as of November 6, 2026. These are file sizes; context memory comes on top. Treat the table as a starting point; `ollama ps` on your own machine has the final word.

| GPU memory | What is reasonable to run | Examples (file size) |
|---|---|---|
| 8 GB | 1-4B | qwen3:4b (2.5 GB), qwen3.5:4b (3.3-4.0 GB) |
| 16 GB | 7-9B | qwen2.5-coder:7b (4.7 GB), qwen3:8b (5.2 GB), qwen3.5:9b (6.6-7.6 GB) |
| 24-36 GB | 14-20B, MoE around 30B | qwen3:14b (9.3 GB), gpt-oss:20b (14 GB), qwen3:30b (19 GB) |
| 48 GB and up | 27-35B | qwen3.8:27b (18 GB), qwen3.6:35b (23 GB) |
| Servers, 80+ GB | 100B+ | gpt-oss:120b (65 GB), qwen3.5:122b (81 GB) |

### Tags and quantization

A model name has a family and a tag: `qwen3:8b`, `qwen3.6:27b-q4_K_M`. A tag without a suffix is the default variant. Suffixes such as `q4_K_M`, `q8_0` and `bf16` mark the quantization: fewer bits per weight mean a smaller file, less memory and a more noticeable quality loss. For qwen3.8:27b that is 18 GB at `q4_K_M`, 30 GB at `q8_0` and 56 GB at `bf16`. Some recent models also have `-mlx` tags for Apple Silicon.

Each model page shows capability labels:

- `tools`: the model is trained to call tools (function calling);
- `thinking`: the model can produce reasoning before the answer, switched on and off with the `think` parameter;
- `vision`: accepts images;
- `embedding`: a model for vectors, not for chat (nomic-embed-text, bge-m3, qwen3-embedding);
- `cloud`: the model has a cloud variant that runs on Ollama's servers.

The `tools` label doesn't guarantee the model can handle an agent: qwen2.5-coder:7b has it, yet in OpenCode it printed tool calls as text.

### Ollama and Qwen

Qwen models fill a good share of the top rows by downloads in the library. As of November 6, 2026 it includes:

| Model | Sizes | Context | Labels |
|---|---|---|---|
| qwen2.5-coder | 0.5b-32b | 32K | tools |
| qwen3 | 0.6b-235b | 40K or 256K, depending on size | tools, thinking |
| qwen3-coder | 30b, 480b | 256K (30b) | tools |
| qwen3.5 | 0.8b-122b | 256K | vision, tools, thinking |
| qwen3.6 | 27b, 35b (35b-a3b) | 256K | vision, tools, thinking |
| qwen3.8 | 27b | 256K | vision, tools, thinking |

For a 16 GB machine, qwen3:8b and qwen3.5:9b fit. qwen2.5-coder:7b handles short coding tasks in chat, but its page was last updated more than a year ago, and it does not work as an agent. The native context of qwen3:8b is 40960 tokens; qwen3.5 has 256K. For an agent this matters: the OpenCode system prompt grew to about 53,000 tokens in testing.

Family numbering and the library lineup change every few weeks: at the time of checking, qwen3.8 had been updated six days earlier and qwen3.6 a week earlier. Before you choose, open the model page and check the update date and the tags.

## Ollama CLI commands

| Command | What it does |
|---|---|
| `ollama run qwen3:8b` | Interactive chat; with a string argument, runs a single prompt |
| `ollama pull qwen3:8b` | Downloads a model; an interrupted download resumes on rerun |
| `ollama ls` | Lists downloaded models with sizes |
| `ollama ps` | Models loaded in memory: memory, PROCESSOR, CONTEXT, time until unload |
| `ollama stop qwen3:8b` | Unloads the model from memory right away |
| `ollama rm qwen3:8b` | Deletes the model from disk |
| `ollama show qwen3:8b` | Parameters, native context, capabilities; `--modelfile` prints the Modelfile |
| `ollama create name -f Modelfile` | Builds a model from a Modelfile |
| `ollama cp src dst` | Copies a model under another name |
| `ollama serve` | Runs the server in the current terminal |
| `ollama launch [integration]` | Configures and launches an external tool with Ollama |

The `--verbose` flag on `ollama run` prints stats after the answer:

```bash
ollama run qwen2.5-coder:7b --verbose "Write a Python function is_prime(n)."
# total duration:  ...
# load duration:   ...   # loading the model into memory
# prompt eval rate: ...  # prompt processing speed, tok/s
# eval count:       ...  # tokens generated
# eval rate:        ...  # generation speed, tok/s
```

To compare models, look at `eval rate`, the answer generation speed. `load duration` shows how long loading the model took: seconds on the first request, a fraction of a millisecond for a model that is already loaded. For models with the `thinking` label, turn reasoning off for short answers: `ollama run qwen3:8b --think=false`.

## Benchmark: qwen2.5-coder:7b and qwen3:8b on M4 Max

Setup: Mac with Apple M4 Max, 36 GB of memory, macOS 27.2, Ollama 0.40.14 from Homebrew (release 0.40.15 had not reached the formula at test time). Server on default settings.

| Model | Load time (load / total) | Generation, first request | Generation, warm model | `ollama ps` |
|---|---|---|---|---|
| qwen2.5-coder:7b | 2.11 s / 4.0 s | 69.5 tok/s | 68.9 tok/s | 6.6 GB, CONTEXT 32768, 100% GPU |
| qwen3:8b (`--think=false`) | 3.78 s / 6.5 s | 59.5 tok/s | 59.4 tok/s | 9.9 GB, CONTEXT 32768, 100% GPU |

How to read the numbers:

- Load time was measured while the model files were already in the macOS file system cache. This test doesn't show loading from a cold disk, which takes longer.
- Generation speed for 7-8B models on this hardware stays around 60-70 tokens per second and barely depends on whether it is the first request.
- Downloading qwen2.5-coder:7b (4.7 GB) over a stable connection took about a minute and a half. That time depends entirely on the network.

The Ollama version affects memory. On Ollama 0.24.0 the same qwen2.5-coder:7b with the same 32768 context took 8.2 GB; on 0.40.14 it took 6.6 GB, while generation speed barely changed (72 and 70 tok/s versus 69.5 and 68.9). Numbers from older guides are worth rechecking on your version.

## Ollama REST API

The native API lives at `http://localhost:11434/api`. No key is needed locally ([docs.ollama.com/api](https://docs.ollama.com/api/introduction)). The full endpoint list is in the reference:

| Endpoint | Purpose |
|---|---|
| `POST /api/generate` | Answer to a single prompt |
| `POST /api/chat` | Answer to a message history, with tools |
| `POST /api/embed` | Embeddings |
| `GET /api/tags` | Downloaded models |
| `GET /api/ps` | Loaded models, `context_length`, `size_vram` |
| `POST /api/show` | Model parameters and capabilities |
| `POST /api/pull`, `/api/create`, `/api/copy`, `DELETE /api/delete` | Model management |
| `GET /api/version` | Server version |

A request to `/api/generate`:

```bash
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "Write a one-line Python lambda that squares a number. Code only.",
  "stream": false,
  "options": {"num_predict": 60}
}' | jq -r .response
# ```python
# square = lambda x: x ** 2
# ```
```

On a warm model this request took 0.27 s. `/api/chat` takes a `messages` array instead of `prompt`:

```bash
curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Explain a Python decorator in one sentence."}],
  "think": false,
  "stream": false
}' | jq -r .message.content
```

The parameters you need most often:

- `stream` defaults to `true`: the answer arrives as a stream of chunks. For scripts and curl, set it to `false`.
- `options` holds generation parameters: `temperature`, `top_k`, `top_p`, `seed`, `stop`, `num_predict`, `num_ctx`.
- `keep_alive` sets how long the model stays in memory after the request (`"10m"`; `0` unloads it right away).
- `think: false` turns off reasoning for models with `thinking`. Without it, qwen3 spends tokens on reasoning even for a one-line answer.
- `format` takes `"json"` or a JSON schema for structured output.

The response includes `total_duration`, `load_duration`, `prompt_eval_count`, `eval_count` and `eval_duration`. Durations are in nanoseconds, so generation speed is `eval_count / eval_duration * 1e9`.

Python and JavaScript have official libraries, `pip install ollama` and `npm i ollama`:

```python
from ollama import chat

response = chat(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Name three uses of embeddings."}],
    think=False,
)
print(response.message.content)
```

## Ollama OpenAI-compatible API

The address is `http://localhost:11434/v1`. An existing OpenAI client connects by changing `base_url`. Any key works: the SDK requires a value, and Ollama doesn't check it ([docs.ollama.com/openai](https://docs.ollama.com/openai)).

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")

resp = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Write a one-line Python lambda that squares a number."}],
    max_tokens=60,
)
print(resp.choices[0].message.content)
```

Supported endpoints are `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings`, `/v1/responses` (stateless only) and the audio endpoints. Chat completions support streaming, JSON mode, `seed`, base64 images, `tools` and `reasoning_effort`. For qwen3, `"reasoning_effort": "none"` gives a short answer with no reasoning.

What `/v1` lacks:

- `tool_choice`, `logit_bias`, `user`, `n`, logprobs;
- images by URL (base64 only);
- a context size parameter. If you need a different `num_ctx`, create a model with a Modelfile and call it by name.

For a tool that expects a specific model name such as `gpt-3.5-turbo`, give it a copy: `ollama cp qwen3:8b gpt-3.5-turbo`.

Both APIs are equally fast: on a warm qwen2.5-coder:7b, `/v1/chat/completions` answered in 0.23 s versus 0.27 s for `/api/generate`. The choice between them is about format. If your code is already written for the OpenAI SDK, or the tool only supports OpenAI-compatible providers, use `/v1`. If you need `num_ctx`, `keep_alive` and model management, use `/api`.

## Context and memory in Ollama

Context size sets how many tokens the model sees at once and how much memory the KV cache takes. The default depends on VRAM ([docs.ollama.com/context-length](https://docs.ollama.com/context-length)):

| VRAM | Default context |
|---|---|
| under 24 GiB | 4k |
| 24-48 GiB | 32k |
| 48 GiB and up | 256k |

On the test machine the server saw 28.1 GiB and set 32768, as confirmed by the log line `vram-based default context ... default_num_ctx=32768`. Yet the Modelfile page lists a default `num_ctx` of 2048. The documentation disagrees with itself, so check the real value in the CONTEXT column of `ollama ps`.

For agents, web search and coding tools, the documentation recommends at least 64000 tokens. On a 16 GB machine the default is 4k, and an agent's system prompt will not fit: Ollama truncates the input, and the model loses its instructions.

How to change the context:

```bash
# for the whole server
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# for a single /api request
curl -s http://localhost:11434/api/generate -d '{"model":"qwen3:8b","prompt":"hi","options":{"num_ctx":32768}}'
```

In the Ollama app, a slider in Settings controls the context. The third way is `PARAMETER num_ctx` in a Modelfile.

Raising the context has two limits:

1. **The model's native window.** `num_ctx` above it has no effect. qwen3:8b has a 40960-token window, and when 65536 was requested by any of the three methods, `ollama ps` showed `CONTEXT 40960`. gemma4:e4b has a native window of 131072, and 65536 was applied with no capping. `ollama show <model>` shows the native window.
2. **Memory.** As the context grows, so does the KV cache. qwen3:8b took 11 GB at a 40960 context versus 9.9 GB at 32768. If memory runs out, part of the model moves to the CPU, and PROCESSOR shows a CPU/GPU split instead of `100% GPU`.

Two variables from the [FAQ](https://docs.ollama.com/faq) save memory on the cache. `OLLAMA_KV_CACHE_TYPE=q8_0` halves the KV cache compared with `f16` at a very small quality loss; `q4_0` cuts it to a quarter with a small to medium loss. A quantized cache only works with Flash Attention: `OLLAMA_FLASH_ATTENTION=1`. The Homebrew formula suggests exactly this pair of variables in its post-install notes.

## Modelfile: building your own Ollama model

A Modelfile is a text file that describes a derived model: base weights, parameters and a system prompt ([docs.ollama.com/modelfile](https://docs.ollama.com/modelfile)). A tested example:

```dockerfile
FROM qwen3:8b
PARAMETER num_ctx 65536
PARAMETER temperature 0.2
SYSTEM You are a concise Python code assistant. Answer with code first.
```

```bash
ollama create fc-qwen3-64k -f Modelfile
ollama show fc-qwen3-64k         # num_ctx 65536, temperature 0.2, System, Capabilities
ollama run fc-qwen3-64k --think=false "Reverse a string in Python."
ollama ps                        # CONTEXT 40960: capped at the native window
```

`ollama show` reports `num_ctx 65536`, but after the run `ollama ps` shows 40960. The model's native window set the ceiling.

Modelfile instructions:

| Instruction | Purpose |
|---|---|
| `FROM` | Required: a library model, a safetensors folder or a GGUF file |
| `PARAMETER` | Generation and context parameters |
| `SYSTEM` | System prompt |
| `TEMPLATE` | Prompt template (Go template syntax) |
| `MESSAGE` | Example dialogue in the history (system, user, assistant) |
| `LICENSE` | License text |
| `REQUIRES` | Minimum Ollama version for the model |
| `CAPABILITY` | Adds a capability to a GGUF or MLX model |

Common parameters and their documented defaults: `temperature` 0.8, `top_k` 40, `top_p` 0.9, `min_p` 0.0, `repeat_penalty` 1.0, `num_predict` -1 (no limit), `seed` 0. `stop` can be set several times. To see the Modelfile of any model, including its library template, run `ollama show --modelfile qwen3:8b`.

Besides setting the context for the OpenAI API, a Modelfile is useful for a task-specific system prompt (code reviewer, ticket classifier) and a fixed temperature. Another use case is packaging your own GGUF under a clear name.

## Ollama server settings and security

The server is configured through environment variables:

| Variable | Default | What it changes |
|---|---|---|
| `OLLAMA_HOST` | `127.0.0.1:11434` | Server address and port |
| `OLLAMA_MODELS` | `~/.ollama/models` (macOS) | Model folder |
| `OLLAMA_KEEP_ALIVE` | 5 minutes | How long a model stays in memory after a request |
| `OLLAMA_CONTEXT_LENGTH` | VRAM-based | Default context |
| `OLLAMA_NUM_PARALLEL` | 1 | Parallel requests per model |
| `OLLAMA_MAX_LOADED_MODELS` | 3 × number of GPUs (3 on CPU) | How many models stay in memory at once |
| `OLLAMA_MAX_QUEUE` | 512 | Request queue; beyond it the server returns 503 |
| `OLLAMA_ORIGINS` | 127.0.0.1, 0.0.0.0 | Allowed CORS origins |

Parallel requests cost memory: per the documentation, it grows in proportion to `OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH`.

Where to set the variables depends on how Ollama was installed. For the macOS app, use `launchctl setenv` and restart the app. On Linux, use `systemctl edit ollama` and `Environment=` lines in the `[Service]` section. On Windows, set user environment variables and restart Ollama. If you started the server by hand with `ollama serve`, set the variables in the same terminal.

By default Ollama listens only on localhost. `OLLAMA_HOST=0.0.0.0:11434` opens it to the local network, but the local API has no key. Anyone who can reach the port can run models, download new ones and delete existing ones. Expose Ollama only through a reverse proxy with authentication, a VPN or an SSH tunnel (`ssh -L 11434:localhost:11434 host`).

Cloud features (cloud models and web search) are turned off with `OLLAMA_NO_CLOUD=1` or with `"disable_ollama_cloud": true` in `~/.ollama/server.json`.

## Integrations: Ollama for agents and IDEs

The `ollama launch` command configures an external tool to work with a local model and launches it. In version 0.40.14, `ollama launch --help` lists 18 integrations, among them claude, codex, opencode, copilot, droid, cline, vscode, openclaw, hermes and qwen. The Integrations section of the [documentation](https://docs.ollama.com/integrations) also has pages for JetBrains, Zed, Xcode, Goose, n8n and other tools. The `--model` flag picks the model, `--config` sets up the integration without launching it, and `--restore` returns the tool to its original profile. `launch` itself was not run while preparing this article, because it changes the configs of installed tools.

**Claude Code.** Ollama has supported the Anthropic Messages API (`/v1/messages`) [since version 0.14.0](https://github.com/ollama/ollama/releases/tag/v0.14.0), so Claude Code can be pointed at a local model:

```bash
ollama launch claude --model qwen3.5
# or manually
export ANTHROPIC_AUTH_TOKEN=ollama      # required, but not checked
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
claude --model qwen3.5
```

The [integration docs](https://docs.ollama.com/integrations/claude-code) require a context of 64k or more for local models. Anthropic [does not support](https://code.claude.com/docs/en/llm-gateway) running Claude Code on non-Claude models, so this setup is at your own risk.

**OpenCode.** This open-source terminal agent connects with `ollama launch opencode`, or manually through the `@ai-sdk/openai-compatible` provider with the address `http://localhost:11434/v1`. The config and results of testing local models are in the [OpenCode guide](https://futurecraft.pro/blog/opencode-guide/).

**IDE extensions.** Cline, Roo Code, VS Code, JetBrains and Zed each have their own integration page in the Ollama docs. This is a common option for a free stack; where it sits among paid tools is shown in the overview of [AI coding tools](https://futurecraft.pro/blog/best-ai-coding-tools/).

**Your own code and RAG.** For embeddings there are the nomic-embed-text, bge-m3 and qwen3-embedding models and the `/api/embed` and `/v1/embeddings` endpoints. Embeddings were not tested for this article. How retrieval on top of them works is covered in the [RAG guide](https://futurecraft.pro/blog/rag-system-guide/).

## Ollama limitations and pitfalls

### Good coding model ≠ good agent

An agent needs a model that reliably calls tools in the right format and holds a long system prompt. Writing code in chat is a different task. In an OpenCode test on the task "add a function and a test for it", local models produced these results:

| Model | Context | Result |
|---|---|---|
| qwen2.5-coder:7b | 32768 | Printed tool calls as JSON text; no files changed. Repeated on two Ollama versions |
| qwen3:8b | 32768 and 40960 | Native tool calls, but unstable results: one clean run on the older Ollama version, none clean out of four runs on the newer one (a neighboring function deleted, duplicate `return`, timeouts) |
| gemma4:e4b | 65536 | Task done in 79 s, but the agent deleted an existing function and its test |

The `tools` label in the library doesn't guarantee working tool calls in a given agent. Local 8B models can work in agent mode, but check their edits and keep the project under git. The full method and test logs are in the [OpenCode guide](https://futurecraft.pro/blog/opencode-guide/).

### The default context is too small for agents

Machines with less than 24 GiB of VRAM get 4k by default, and 24-48 GiB gets 32k. An agent's system prompt with tool descriptions easily exceeds both. In the OpenCode test, where OpenCode picked up other agents' skills from the home folder, the prompt grew to about 53,000 tokens, and Ollama truncated it to 32768 (`truncating input prompt` in the server log). Before connecting an agent, check CONTEXT in `ollama ps` and the model's native window.

### `num_ctx` hits the native window

A Modelfile, an environment variable and `options.num_ctx` cannot raise the context beyond what the model was trained on. There is no error: `ollama show` displays the value you set, and the real one is visible only in `ollama ps`.

### Versions change fast

Between 0.24.0 and 0.40.14, memory use for the same model dropped by 1.6 GB. In the library, qwen3.6 and qwen3.8 were updated within the week before checking. Numbers in any guide, this one included, are tied to an Ollama version and a date.

### Quality and speed are not cloud-level

In the checks for this article, 7-8B models answered short tasks such as a one-line lambda correctly, but none of them reliably solved a multi-step agent task that edited two files. 60-70 tokens per second on an M4 Max is single-user speed. Several parallel clients need `OLLAMA_NUM_PARALLEL` and memory for each stream.

### Docker on Mac and cold starts

Docker Desktop on macOS has no GPU acceleration, so models there run on the CPU. The first request after idle time waits for the model to load (by default a model is unloaded after 5 minutes without requests). If latency matters, raise `OLLAMA_KEEP_ALIVE` or pass `keep_alive` in the request.

## Where to start

1. Install Ollama for your OS and check `curl http://localhost:11434/api/version`.
2. Pick a model from the memory table: for 16 GB, qwen3:8b or qwen3.5:9b; for 8 GB, 4B models.
3. Run `ollama run <model> --verbose` and check `eval rate`, then `ollama ps`: the model should be at `100% GPU`.
4. Connect your code through `/v1` with the OpenAI SDK or through the `ollama` library.
5. For an agent, raise the context to 64k if the model allows it (`ollama show`), and check the result in `ollama ps`.
6. Before setting `OLLAMA_HOST=0.0.0.0`, put a proxy with authentication in front of the server.