Ollama Guide: Install, Models, REST API and Modelfile

Ollama is an open-source, MIT-licensed program for running language models on your own computer. It downloads models from the ollama.com library, manages them through a CLI and runs a local server on port 11434: a native REST API, an OpenAI-compatible /v1 and an Anthropic-compatible /v1/messages. The ollama/ollama repository has more than 186,000 stars, and the latest release as of November 6, 2026 is v0.40.15.

Installing takes one command. A second one starts your first model. The trouble comes later. A model doesn’t fit in memory. An agent loses the start of its prompt, because on a machine with less than 24 GiB of VRAM the default context is only 4k. num_ctx in a Modelfile gets silently capped. A “coding” model prints tool calls as text instead of running them.

This guide covers installing Ollama on macOS, Windows, Linux and in Docker, picking a model for your memory, working with it through the CLI, /api and /v1, tuning the context and building your own Modelfile. Speed and memory numbers come from a Mac with an Apple M4 Max and 36 GB of memory running Ollama 0.40.14. Where nothing was tested (Windows, Linux, NVIDIA), the text says so, and the commands come from the documentation.

What Ollama is and when you need it

Ollama does three jobs. It downloads model weights from its library (or takes your own GGUF file). It runs the model on the GPU, and partly on the CPU if VRAM runs short. And it keeps an HTTP server running that the CLI, the Ollama app, your code and third-party tools talk to. If you’re choosing between Ollama and LM Studio, see the LM Studio vs Ollama comparison with benchmarks on the same model.

How a local model compares with a cloud API:

	Ollama (local)	Cloud API (Claude, GPT)
Data	Stays on your machine	Goes to the provider
Cost	Hardware and electricity	Pay per token
Quality	Below frontier models, depends on size	Frontier models
Speed	Depends on GPU and model size	Depends on provider and load
Availability	Works offline once the model is downloaded	Needs network, account, payment

The FAQ is explicit about privacy: when you run locally, Ollama doesn’t see your prompts or data. That’s a common reason to pick a local model: classifying personal data, parsing logs, experimenting without a token bill. One example is the Ambient AI for Mac postmortem, where llama3.2:3b running through Ollama analyzes work patterns and no data goes to the cloud.

In a production architecture, a local model usually becomes one provider among several rather than a replacement for the cloud. How to route tasks between models and keep a fallback is covered in Production LLM Stack.

How to install Ollama

The commands below come from the README and the documentation as of November 2026. The macOS install was tested; the other platforms are described from the documentation.

macOS

Requirements: macOS Sonoma (14) or newer. On Apple Silicon both CPU and GPU are used; Intel Macs run on CPU only (docs.ollama.com/macos).

curl -fsSL https://ollama.com/install.sh | sh

The second option is to download Ollama.dmg and drag the app into Applications. On first launch the app offers to create a CLI link in /usr/local/bin.

Homebrew has an ollama formula in homebrew-core. The official documentation doesn’t cover this method. The formula installs the CLI and server without the GUI app, and brew services start ollama runs the background service:

brew install ollama
brew services start ollama

Windows

Per the documentation: Windows 10 22H2 or newer, or Windows 11. NVIDIA needs driver 551.61+, AMD needs a stack with ROCm v7 support or the Radeon Vulkan driver. No admin rights are required; the installer puts Ollama in your home folder.

irm https://ollama.com/install.ps1 | iex

Or download OllamaSetup.exe. To run Ollama as a service, there is the ollama-windows-amd64.zip archive.

Linux

curl -fsSL https://ollama.com/install.sh | sh

The script creates an ollama user and a systemd service. For a manual install, the Linux page lists .tar.zst archives for amd64 and arm64 plus a separate ROCm package for AMD. Managing the service:

sudo systemctl status ollama
journalctl -e -u ollama          # server logs
sudo systemctl edit ollama       # environment variables

To update on Linux, run the same script again.

Docker

# CPU only
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# NVIDIA GPU (requires NVIDIA Container Toolkit)
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker exec -it ollama ollama run llama3.2

For AMD there is the ollama/ollama:rocm image (docs.ollama.com/docker). On macOS the Docker route does nothing for speed: the FAQ states that Docker Desktop on macOS has no GPU acceleration. On a Mac, install Ollama natively.

Checking the install

ollama --version
curl http://localhost:11434/api/version
# {"version":"0.40.14"}

If the CLI prints could not connect to a running Ollama instance, the server isn’t running. Start it with the app, the systemd service, brew services or ollama serve in a separate terminal.

Ollama models: how to pick one for your hardware

Memory is the main constraint. The model has to fit entirely in VRAM (on Apple Silicon, in the part of unified memory the GPU can see). Otherwise some layers move to the CPU and speed drops. Memory goes to both the weights and the context KV cache. The memory math, model families, and 27B benchmarks are in the local LLM guide.

A test on Ollama 0.40.14 shows the gap:

Model	File	In memory (ollama ps, context 32768)
qwen2.5-coder:7b	4.7 GB	6.6 GB
qwen3:8b	5.2 GB	9.9 GB

On macOS the GPU doesn’t get all of the memory. On a 36 GB machine, the Ollama server reported total_vram=28.1 GiB in its log.

The size guide below is based on the tag pages at ollama.com/library as of November 6, 2026. These are file sizes; context memory comes on top. Treat the table as a starting point; ollama ps on your own machine has the final word.

GPU memory	What is reasonable to run	Examples (file size)
8 GB	1-4B	qwen3:4b (2.5 GB), qwen3.5:4b (3.3-4.0 GB)
16 GB	7-9B	qwen2.5-coder:7b (4.7 GB), qwen3:8b (5.2 GB), qwen3.5:9b (6.6-7.6 GB)
24-36 GB	14-20B, MoE around 30B	qwen3:14b (9.3 GB), gpt-oss:20b (14 GB), qwen3:30b (19 GB)
48 GB and up	27-35B	qwen3.8:27b (18 GB), qwen3.6:35b (23 GB)
Servers, 80+ GB	100B+	gpt-oss:120b (65 GB), qwen3.5:122b (81 GB)

Tags and quantization

A model name has a family and a tag: qwen3:8b, qwen3.6:27b-q4_K_M. A tag without a suffix is the default variant. Suffixes such as q4_K_M, q8_0 and bf16 mark the quantization: fewer bits per weight mean a smaller file, less memory and a more noticeable quality loss. For qwen3.8:27b that is 18 GB at q4_K_M, 30 GB at q8_0 and 56 GB at bf16. Some recent models also have -mlx tags for Apple Silicon.

Each model page shows capability labels:

• tools: the model is trained to call tools (function calling);
• thinking: the model can produce reasoning before the answer, switched on and off with the think parameter;
• vision: accepts images;
• embedding: a model for vectors, not for chat (nomic-embed-text, bge-m3, qwen3-embedding);
• cloud: the model has a cloud variant that runs on Ollama’s servers.

The tools label doesn’t guarantee the model can handle an agent: qwen2.5-coder:7b has it, yet in OpenCode it printed tool calls as text.

Ollama and Qwen

Qwen models fill a good share of the top rows by downloads in the library. As of November 6, 2026 it includes:

Model	Sizes	Context	Labels
qwen2.5-coder	0.5b-32b	32K	tools
qwen3	0.6b-235b	40K or 256K, depending on size	tools, thinking
qwen3-coder	30b, 480b	256K (30b)	tools
qwen3.5	0.8b-122b	256K	vision, tools, thinking
qwen3.6	27b, 35b (35b-a3b)	256K	vision, tools, thinking
qwen3.8	27b	256K	vision, tools, thinking

For a 16 GB machine, qwen3:8b and qwen3.5:9b fit. qwen2.5-coder:7b handles short coding tasks in chat, but its page was last updated more than a year ago, and it does not work as an agent. The native context of qwen3:8b is 40960 tokens; qwen3.5 has 256K. For an agent this matters: the OpenCode system prompt grew to about 53,000 tokens in testing.

Family numbering and the library lineup change every few weeks: at the time of checking, qwen3.8 had been updated six days earlier and qwen3.6 a week earlier. Before you choose, open the model page and check the update date and the tags.

Ollama CLI commands

Command	What it does
ollama run qwen3:8b	Interactive chat; with a string argument, runs a single prompt
ollama pull qwen3:8b	Downloads a model; an interrupted download resumes on rerun
ollama ls	Lists downloaded models with sizes
ollama ps	Models loaded in memory: memory, PROCESSOR, CONTEXT, time until unload
ollama stop qwen3:8b	Unloads the model from memory right away
ollama rm qwen3:8b	Deletes the model from disk
ollama show qwen3:8b	Parameters, native context, capabilities; --modelfile prints the Modelfile
ollama create name -f Modelfile	Builds a model from a Modelfile
ollama cp src dst	Copies a model under another name
ollama serve	Runs the server in the current terminal
ollama launch [integration]	Configures and launches an external tool with Ollama

The --verbose flag on ollama run prints stats after the answer:

ollama run qwen2.5-coder:7b --verbose "Write a Python function is_prime(n)."
# total duration:  ...
# load duration:   ...   # loading the model into memory
# prompt eval rate: ...  # prompt processing speed, tok/s
# eval count:       ...  # tokens generated
# eval rate:        ...  # generation speed, tok/s

To compare models, look at eval rate, the answer generation speed. load duration shows how long loading the model took: seconds on the first request, a fraction of a millisecond for a model that is already loaded. For models with the thinking label, turn reasoning off for short answers: ollama run qwen3:8b --think=false.

Benchmark: qwen2.5-coder:7b and qwen3:8b on M4 Max

Setup: Mac with Apple M4 Max, 36 GB of memory, macOS 27.2, Ollama 0.40.14 from Homebrew (release 0.40.15 had not reached the formula at test time). Server on default settings.

Model	Load time (load / total)	Generation, first request	Generation, warm model	ollama ps
qwen2.5-coder:7b	2.11 s / 4.0 s	69.5 tok/s	68.9 tok/s	6.6 GB, CONTEXT 32768, 100% GPU
qwen3:8b (--think=false)	3.78 s / 6.5 s	59.5 tok/s	59.4 tok/s	9.9 GB, CONTEXT 32768, 100% GPU

How to read the numbers:

• Load time was measured while the model files were already in the macOS file system cache. This test doesn’t show loading from a cold disk, which takes longer.
• Generation speed for 7-8B models on this hardware stays around 60-70 tokens per second and barely depends on whether it is the first request.
• Downloading qwen2.5-coder:7b (4.7 GB) over a stable connection took about a minute and a half. That time depends entirely on the network.

The Ollama version affects memory. On Ollama 0.24.0 the same qwen2.5-coder:7b with the same 32768 context took 8.2 GB; on 0.40.14 it took 6.6 GB, while generation speed barely changed (72 and 70 tok/s versus 69.5 and 68.9). Numbers from older guides are worth rechecking on your version.

Ollama REST API

The native API lives at http://localhost:11434/api. No key is needed locally (docs.ollama.com/api). The full endpoint list is in the reference:

Endpoint	Purpose
POST /api/generate	Answer to a single prompt
POST /api/chat	Answer to a message history, with tools
POST /api/embed	Embeddings
GET /api/tags	Downloaded models
GET /api/ps	Loaded models, context_length, size_vram
POST /api/show	Model parameters and capabilities
POST /api/pull, /api/create, /api/copy, DELETE /api/delete	Model management
GET /api/version	Server version

A request to /api/generate:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "Write a one-line Python lambda that squares a number. Code only.",
  "stream": false,
  "options": {"num_predict": 60}
}' | jq -r .response
# ```python
# square = lambda x: x ** 2
# ```

On a warm model this request took 0.27 s. /api/chat takes a messages array instead of prompt:

curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Explain a Python decorator in one sentence."}],
  "think": false,
  "stream": false
}' | jq -r .message.content

The parameters you need most often:

• stream defaults to true: the answer arrives as a stream of chunks. For scripts and curl, set it to false.
• options holds generation parameters: temperature, top_k, top_p, seed, stop, num_predict, num_ctx.
• keep_alive sets how long the model stays in memory after the request ("10m"; 0 unloads it right away).
• think: false turns off reasoning for models with thinking. Without it, qwen3 spends tokens on reasoning even for a one-line answer.
• format takes "json" or a JSON schema for structured output.

The response includes total_duration, load_duration, prompt_eval_count, eval_count and eval_duration. Durations are in nanoseconds, so generation speed is eval_count / eval_duration * 1e9.

Python and JavaScript have official libraries, pip install ollama and npm i ollama:

from ollama import chat

response = chat(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Name three uses of embeddings."}],
    think=False,
)
print(response.message.content)

Ollama OpenAI-compatible API

The address is http://localhost:11434/v1. An existing OpenAI client connects by changing base_url. Any key works: the SDK requires a value, and Ollama doesn’t check it (docs.ollama.com/openai).

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")

resp = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Write a one-line Python lambda that squares a number."}],
    max_tokens=60,
)
print(resp.choices[0].message.content)

Supported endpoints are /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings, /v1/responses (stateless only) and the audio endpoints. Chat completions support streaming, JSON mode, seed, base64 images, tools and reasoning_effort. For qwen3, "reasoning_effort": "none" gives a short answer with no reasoning.

What /v1 lacks:

• tool_choice, logit_bias, user, n, logprobs;
• images by URL (base64 only);
• a context size parameter. If you need a different num_ctx, create a model with a Modelfile and call it by name.

For a tool that expects a specific model name such as gpt-3.5-turbo, give it a copy: ollama cp qwen3:8b gpt-3.5-turbo.

Both APIs are equally fast: on a warm qwen2.5-coder:7b, /v1/chat/completions answered in 0.23 s versus 0.27 s for /api/generate. The choice between them is about format. If your code is already written for the OpenAI SDK, or the tool only supports OpenAI-compatible providers, use /v1. If you need num_ctx, keep_alive and model management, use /api.

Context and memory in Ollama

Context size sets how many tokens the model sees at once and how much memory the KV cache takes. The default depends on VRAM (docs.ollama.com/context-length):

VRAM	Default context
under 24 GiB	4k
24-48 GiB	32k
48 GiB and up	256k

On the test machine the server saw 28.1 GiB and set 32768, as confirmed by the log line vram-based default context ... default_num_ctx=32768. Yet the Modelfile page lists a default num_ctx of 2048. The documentation disagrees with itself, so check the real value in the CONTEXT column of ollama ps.

For agents, web search and coding tools, the documentation recommends at least 64000 tokens. On a 16 GB machine the default is 4k, and an agent’s system prompt will not fit: Ollama truncates the input, and the model loses its instructions.

How to change the context:

# for the whole server
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# for a single /api request
curl -s http://localhost:11434/api/generate -d '{"model":"qwen3:8b","prompt":"hi","options":{"num_ctx":32768}}'

In the Ollama app, a slider in Settings controls the context. The third way is PARAMETER num_ctx in a Modelfile.

Raising the context has two limits:

1. The model’s native window. num_ctx above it has no effect. qwen3:8b has a 40960-token window, and when 65536 was requested by any of the three methods, ollama ps showed CONTEXT 40960. gemma4:e4b has a native window of 131072, and 65536 was applied with no capping. ollama show <model> shows the native window.
2. Memory. As the context grows, so does the KV cache. qwen3:8b took 11 GB at a 40960 context versus 9.9 GB at 32768. If memory runs out, part of the model moves to the CPU, and PROCESSOR shows a CPU/GPU split instead of 100% GPU.

Two variables from the FAQ save memory on the cache. OLLAMA_KV_CACHE_TYPE=q8_0 halves the KV cache compared with f16 at a very small quality loss; q4_0 cuts it to a quarter with a small to medium loss. A quantized cache only works with Flash Attention: OLLAMA_FLASH_ATTENTION=1. The Homebrew formula suggests exactly this pair of variables in its post-install notes.

Modelfile: building your own Ollama model

A Modelfile is a text file that describes a derived model: base weights, parameters and a system prompt (docs.ollama.com/modelfile). A tested example:

FROM qwen3:8b
PARAMETER num_ctx 65536
PARAMETER temperature 0.2
SYSTEM You are a concise Python code assistant. Answer with code first.

ollama create fc-qwen3-64k -f Modelfile
ollama show fc-qwen3-64k         # num_ctx 65536, temperature 0.2, System, Capabilities
ollama run fc-qwen3-64k --think=false "Reverse a string in Python."
ollama ps                        # CONTEXT 40960: capped at the native window

ollama show reports num_ctx 65536, but after the run ollama ps shows 40960. The model’s native window set the ceiling.

Modelfile instructions:

Instruction	Purpose
FROM	Required: a library model, a safetensors folder or a GGUF file
PARAMETER	Generation and context parameters
SYSTEM	System prompt
TEMPLATE	Prompt template (Go template syntax)
MESSAGE	Example dialogue in the history (system, user, assistant)
LICENSE	License text
REQUIRES	Minimum Ollama version for the model
CAPABILITY	Adds a capability to a GGUF or MLX model

Common parameters and their documented defaults: temperature 0.8, top_k 40, top_p 0.9, min_p 0.0, repeat_penalty 1.0, num_predict -1 (no limit), seed 0. stop can be set several times. To see the Modelfile of any model, including its library template, run ollama show --modelfile qwen3:8b.

Besides setting the context for the OpenAI API, a Modelfile is useful for a task-specific system prompt (code reviewer, ticket classifier) and a fixed temperature. Another use case is packaging your own GGUF under a clear name.

Ollama server settings and security

The server is configured through environment variables:

Variable	Default	What it changes
OLLAMA_HOST	127.0.0.1:11434	Server address and port
OLLAMA_MODELS	~/.ollama/models (macOS)	Model folder
OLLAMA_KEEP_ALIVE	5 minutes	How long a model stays in memory after a request
OLLAMA_CONTEXT_LENGTH	VRAM-based	Default context
OLLAMA_NUM_PARALLEL	1	Parallel requests per model
OLLAMA_MAX_LOADED_MODELS	3 × number of GPUs (3 on CPU)	How many models stay in memory at once
OLLAMA_MAX_QUEUE	512	Request queue; beyond it the server returns 503
OLLAMA_ORIGINS	127.0.0.1, 0.0.0.0	Allowed CORS origins

Parallel requests cost memory: per the documentation, it grows in proportion to OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH.

Where to set the variables depends on how Ollama was installed. For the macOS app, use launchctl setenv and restart the app. On Linux, use systemctl edit ollama and Environment= lines in the [Service] section. On Windows, set user environment variables and restart Ollama. If you started the server by hand with ollama serve, set the variables in the same terminal.

By default Ollama listens only on localhost. OLLAMA_HOST=0.0.0.0:11434 opens it to the local network, but the local API has no key. Anyone who can reach the port can run models, download new ones and delete existing ones. Expose Ollama only through a reverse proxy with authentication, a VPN or an SSH tunnel (ssh -L 11434:localhost:11434 host).

Cloud features (cloud models and web search) are turned off with OLLAMA_NO_CLOUD=1 or with "disable_ollama_cloud": true in ~/.ollama/server.json.

Integrations: Ollama for agents and IDEs

The ollama launch command configures an external tool to work with a local model and launches it. In version 0.40.14, ollama launch --help lists 18 integrations, among them claude, codex, opencode, copilot, droid, cline, vscode, openclaw, hermes and qwen. The Integrations section of the documentation also has pages for JetBrains, Zed, Xcode, Goose, n8n and other tools. The --model flag picks the model, --config sets up the integration without launching it, and --restore returns the tool to its original profile. launch itself was not run while preparing this article, because it changes the configs of installed tools.

Claude Code. Ollama has supported the Anthropic Messages API (/v1/messages) since version 0.14.0, so Claude Code can be pointed at a local model:

ollama launch claude --model qwen3.5
# or manually
export ANTHROPIC_AUTH_TOKEN=ollama      # required, but not checked
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
claude --model qwen3.5

The integration docs require a context of 64k or more for local models. Anthropic does not support running Claude Code on non-Claude models, so this setup is at your own risk.

OpenCode. This open-source terminal agent connects with ollama launch opencode, or manually through the @ai-sdk/openai-compatible provider with the address http://localhost:11434/v1. The config and results of testing local models are in the OpenCode guide.

IDE extensions. Cline, Roo Code, VS Code, JetBrains and Zed each have their own integration page in the Ollama docs. This is a common option for a free stack; where it sits among paid tools is shown in the overview of AI coding tools.

Your own code and RAG. For embeddings there are the nomic-embed-text, bge-m3 and qwen3-embedding models and the /api/embed and /v1/embeddings endpoints. Embeddings were not tested for this article. How retrieval on top of them works is covered in the RAG guide.

Ollama limitations and pitfalls

Good coding model ≠ good agent

An agent needs a model that reliably calls tools in the right format and holds a long system prompt. Writing code in chat is a different task. In an OpenCode test on the task “add a function and a test for it”, local models produced these results:

Model	Context	Result
qwen2.5-coder:7b	32768	Printed tool calls as JSON text; no files changed. Repeated on two Ollama versions
qwen3:8b	32768 and 40960	Native tool calls, but unstable results: one clean run on the older Ollama version, none clean out of four runs on the newer one (a neighboring function deleted, duplicate return, timeouts)
gemma4:e4b	65536	Task done in 79 s, but the agent deleted an existing function and its test

The tools label in the library doesn’t guarantee working tool calls in a given agent. Local 8B models can work in agent mode, but check their edits and keep the project under git. The full method and test logs are in the OpenCode guide.

The default context is too small for agents

Machines with less than 24 GiB of VRAM get 4k by default, and 24-48 GiB gets 32k. An agent’s system prompt with tool descriptions easily exceeds both. In the OpenCode test, where OpenCode picked up other agents’ skills from the home folder, the prompt grew to about 53,000 tokens, and Ollama truncated it to 32768 (truncating input prompt in the server log). Before connecting an agent, check CONTEXT in ollama ps and the model’s native window.

num_ctx hits the native window

A Modelfile, an environment variable and options.num_ctx cannot raise the context beyond what the model was trained on. There is no error: ollama show displays the value you set, and the real one is visible only in ollama ps.

Versions change fast

Between 0.24.0 and 0.40.14, memory use for the same model dropped by 1.6 GB. In the library, qwen3.6 and qwen3.8 were updated within the week before checking. Numbers in any guide, this one included, are tied to an Ollama version and a date.

Quality and speed are not cloud-level

In the checks for this article, 7-8B models answered short tasks such as a one-line lambda correctly, but none of them reliably solved a multi-step agent task that edited two files. 60-70 tokens per second on an M4 Max is single-user speed. Several parallel clients need OLLAMA_NUM_PARALLEL and memory for each stream.

Docker on Mac and cold starts

Docker Desktop on macOS has no GPU acceleration, so models there run on the CPU. The first request after idle time waits for the model to load (by default a model is unloaded after 5 minutes without requests). If latency matters, raise OLLAMA_KEEP_ALIVE or pass keep_alive in the request.

Where to start

1. Install Ollama for your OS and check curl http://localhost:11434/api/version.
2. Pick a model from the memory table: for 16 GB, qwen3:8b or qwen3.5:9b; for 8 GB, 4B models.
3. Run ollama run <model> --verbose and check eval rate, then ollama ps: the model should be at 100% GPU.
4. Connect your code through /v1 with the OpenAI SDK or through the ollama library.
5. For an agent, raise the context to 64k if the model allows it (ollama show), and check the result in ollama ps.
6. Before setting OLLAMA_HOST=0.0.0.0, put a proxy with authentication in front of the server.