Local models for OpenClaw and coding agents: context length and VRAM

Agents such as OpenClaw, and coding tools such as OpenCode and Claude Code, send far longer prompts than a chat, and Ollama's documentation asks for at least 64k tokens of context when they run on a local model. At that length the KV cache, the memory that holds the context, often decides whether a model stays on the GPU. On a 24GB RTX 3090, Qwen3.8 27B keeps 64k fully on the GPU with the default f16 KV cache, while Qwen3.6 35B-A3B needs a q8_0 cache to stay there. Once a model fits, watch prompt processing: every turn, the model reads everything the agent sends before it writes a word.

Why agents need a much longer context than chat

An agent turn carries far more than a chat message: the agent's instructions, a description of every tool it may call, the conversation so far, the files it opened, the output of the commands it ran, and room for the reply, all in the context at once. OpenClaw's page on local models asks you to leave room for all of that, and warns that a model which handles a short prompt can still fail a full agent turn.

Ollama's context-length page says agents, web search and coding tools should get at least 64k tokens, and its integration pages repeat it for OpenCode, OpenClaw and Claude Code. Ollama documents a one-line setup for each (ollama launch opencode, ollama launch openclaw, ollama launch claude), but with a local model the context is still yours to set.

Ollama's own default is much smaller on most cards. Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). A 16GB card therefore starts at 4k unless something sets the length, and a full window makes Ollama drop the oldest messages without an error (see Ollama slow or forgetting your prompt? Context length defaults). Each agent sets the length differently; the settings section below covers OpenClaw, OpenCode and Claude Code.

Thinking models spend the same window on their reasoning. Qwen3.6 35B-A3B and Qwen3.8 27B think by default, according to their model cards, and the Qwen3.6 card advises at least 128k of context to preserve its thinking.

Which coding models fit at 32k and 64k on 16GB and 24GB cards

The table adds up the graphics card memory each model needs at 64k with an f16 KV cache: the weights at CanRun's baseline quant (Q4_K_M, a 4-bit quant; MXFP4 for gpt-oss-20b), the KV cache, and a compute buffer, the runtime's working memory. Compare each total with what your card can use: in CanRun's calculation, with Windows using the card for the display, that is 15.4 GB on a 16GB card, 23.4 GB on a 24GB card and 31.4 GB on a 32GB card.

The totals do not follow the file sizes. Qwen3 30B-A3B (2507) has a smaller file than Qwen3.6 35B-A3B (18.6 GB against 22.3 GB), yet at 64k it needs 25.9 GB against 23.5 GB, because every one of its layers keeps a KV cache. Verdict names in this guide's text refer to 8k with an f16 KV cache; for longer contexts, the tables give a verdict for every length and KV cache type.

24GB: RTX 3090 and RTX 4090

Qwen3.6 35B-A3B is a mixture-of-experts (MoE) model that uses only a small part of its weights for each token. At the 8k baseline it needs 21.8 GB and Runs great on the RTX 3090. At 32k with an f16 KV cache it still stays fully on the GPU. At 64k it needs 23.5 GB, a little over what the card can use. CanRun then keeps as many expert layers on the GPU as fit, moves the rest to system RAM, and labels the speed a theoretical estimate (±30%). A q8_0 KV cache, which stores the cache in 8 bits, brings 64k down to 22.9 GB. That is back on the GPU with little margin, so check ollama ps after loading.

The 128k rows matter for this model, since its card advises at least 128k to keep its thinking intact. On a 24GB card that length leaves some experts in system RAM even with a q4_0 KV cache, and the table marks those cells as theoretical estimates. The RTX 4090 can use the same amount of memory, so the fit results match; at 64k with q8_0 it also keeps the model on the GPU.

Qwen3.8 27B is a dense model, so it reads all of its weights for every token, but only one of its layers in four keeps a KV cache. At the 8k baseline it Runs great on the RTX 3090. At 64k with an f16 KV cache it needs 21.9 GB, which leaves more than a gigabyte free. At 128k with f16 it needs 26.8 GB, so part of the weights move to system RAM (partial offload). With q8_0 it needs 22.8 GB and stays on the GPU with little margin.

Side by side, the two tables show the trade-off. Qwen3.6 35B-A3B reads only its active experts for each token, so its generation estimates are several times those of Qwen3.8 27B, while Qwen3.8 27B holds 64k on the GPU without changing the KV cache type.

Other models on the 24GB tier:

• Gemma 4 26B-A4B needs 19.4 GB at 64k with f16 and stays on the GPU.
• Qwen3 30B-A3B (2507) and Mistral Small 3.2 24B both Run great at the 8k baseline, but at 64k with f16 they need 25.9 GB and 25.8 GB. In CanRun's calculation Qwen3 30B-A3B then moves some experts to system RAM, while the dense Mistral Small 3.2 24B moves part of its weights there. A q8_0 KV cache keeps both on the GPU at 64k, at 22.9 GB (little margin) and 20.8 GB. Qwen3-Coder-30B-A3B-Instruct has the same layout as Qwen3 30B-A3B (2507) in its configuration, so these figures are a close guide for it. Our page for Qwen3 30B-A3B (2507) on the GeForce RTX 3090 has the full context table.
• Nemotron 3.5 Lightning 30B-A3B needs 25.1 GB even at 8k, more than a 24GB card can use, so some of its experts sit in system RAM at every length, and it Runs well as a theoretical estimate. Its KV cache is so small that 64k adds little: the total is 26.0 GB.

16GB: RTX 5060 Ti 16GB

On the GeForce RTX 5060 Ti 16GB, gpt-oss-20b Runs great at the 8k baseline, and that cell is a public benchmark rather than an estimate. At 64k with an f16 KV cache it needs 14.2 GB, still inside the 15.4 GB this card can use. At 128k with f16 it needs 16.4 GB, so some experts move to system RAM, a theoretical estimate. With q8_0 it needs 14.9 GB and stays on the GPU with little margin. The model handles at most 128k. It always thinks before it answers, and its model card lists function calling among its agentic capabilities.

Qwen3.6 35B-A3B needs 21.8 GB even at 8k, more than this card can use, so its experts sit partly in system RAM at every length. At the 8k baseline it Runs well, a theoretical estimate. Its small KV cache keeps the total at 23.5 GB at 64k, close to the 8k figure, so a longer context costs it little extra memory. The Qwen3.6 35B-A3B page compares other cards.

Qwen3.8 27B does not fit even at 8k, where it needs 17.6 GB. Part of its weights run from system RAM, and because a dense model reads every weight for every token, it Runs slowly.

Qwen3.5 9B, built like Qwen3.8 27B with one KV layer in four, is small enough to stay on the GPU at agent lengths. It needs 8.4 GB at 64k with f16. That fits on this card and on an RTX 3060 12GB as well. How well it handles agent work is a separate question these tables do not answer.

8GB, 12GB and 32GB cards

On an 8GB card such as the GeForce RTX 4060, even Qwen3.5 9B is tight at 64k. With f16, part of it moves to system RAM. With q8_0 it needs 7.4 GB against 7.4 GB usable, so in CanRun's calculation it stays on the GPU with almost nothing to spare. Check ollama ps before you rely on it.

On a 12GB card such as the GeForce RTX 3060 12GB, gpt-oss-20b keeps some experts in system RAM at 64k and Runs well there as a theoretical estimate.

A 32GB card such as the GeForce RTX 5090 reaches the length the Qwen3.6 model card asks for. At 128k with an f16 KV cache, Qwen3.6 35B-A3B needs 25.5 GB and stays fully on the GPU in CanRun's calculation. Qwen3.8 27B also stays on the GPU at 128k with f16, at 26.8 GB.

Why a small KV cache per token matters more than file size

The KV cache stores a key and a value for every token in every layer that attends to the whole context. The cost per token is fixed by the model's design, so the cache grows in step with the context. The table lists that cost and what it comes to at 32k, 64k and 128k with f16.

The gap comes from how many layers keep a cache, not from the parameter count. Per each model's configuration or model card:

• Full attention on every layer. Mistral Small 3.2 24B has 40 such layers with eight KV heads and no sliding window; Qwen3 30B-A3B (2507) has 48 with four KV heads.
• Hybrid linear attention. The Qwen3.6 35B-A3B, Qwen3.8 27B and Qwen3.5 9B model cards describe repeating blocks of three Gated DeltaNet layers and one Gated Attention layer. The linear layers carry a fixed-size state, so only one layer in four grows with the context.
• Sliding windows. gpt-oss-20b alternates sliding-window layers, which keep only a short stretch of recent tokens, with full-attention layers; in Gemma 4 26B-A4B, 25 of 30 layers use a 1024-token window.
• Mostly Mamba. Only 6 of Nemotron 3.5 Lightning 30B-A3B's 52 layers are attention layers; the rest are Mamba and MoE layers, which keep no per-token cache.

Between two models that fit at 8k, the one with the smaller cache per token is more likely to still fit at 64k. For sliding-window and hybrid models CanRun counts only the layers whose cache grows; it leaves out the fixed part (the short windows and the linear layers' state), so those figures run slightly low. KV cache explained: how context length eats VRAM covers the mechanics.

Waiting time: the model reads the prompt before it writes

A local model has two speeds: prompt processing, how fast it reads what it is sent, and generation, how fast it writes the reply. CanRun's verdicts and speed bands are generation estimates. In an agent the prompt is long: the first turn of a session carries the agent's instructions and every tool description, and later turns can add a whole file or a long command output. The model reads all of it before the first word of the reply appears.

Reading goes much faster per token than writing, because the GPU takes the prompt many tokens at a time and writes the reply one token after another. In llama.cpp's scoreboard for NVIDIA cards, the prompt rate (pp512) is many times the generation rate (tg128). CanRun estimates prompt speed from a card's FP16 compute and generation speed from its memory bandwidth. Prompt estimates are rough (±40%), and combo pages show them only when the whole model runs on the GPU; with experts or layers in system RAM, CanRun gives none. The prompt estimate scales with the model's size: on the same card, a dense 27B model such as Qwen3.8 27B reads prompts at about a quarter of a 7B model's speed.

Runtimes do not reread what has not changed. llama-server reuses the KV cache from the previous request for the part at the start that matches, so only the part that differs is processed (cache_prompt, on by default). Ollama's API reports prompt_eval_duration as the time spent on the uncached prompt tokens. In a session where the agent only adds to the end, each turn mostly reads what was just added. When something near the start of the prompt changes, everything after that point is read again. For sliding-window and hybrid models (six of the eight here), llama-server can only go back to a saved checkpoint (--ctx-checkpoints), so a change can mean rereading from an earlier point or from the start.

To see how fast your model reads prompts in Ollama, subtract prompt_eval_cached_count from prompt_eval_count and divide by prompt_eval_duration (in nanoseconds, so divide by a billion for seconds); ollama run <model> --verbose prints the same figure as "prompt eval rate", and load_duration shows how long loading took. llama-server's responses already carry a timings object with prompt_per_second and cache_n, the prompt tokens reused from the cache; a cache_n near zero on a later turn means the whole prompt was read again.

A reply that never comes may be a model that is still loading. OpenClaw's page on local models names a cold start as a common cause of what looks like a hang, and points to two timeouts for slow servers: models.providers.<id>.timeoutSeconds for the provider and agents.defaults.timeoutSeconds for the run. Ollama unloads a model after five minutes by default; its FAQ explains how OLLAMA_KEEP_ALIVE changes that. And if ollama ps shows a CPU/GPU split under PROCESSOR, every turn runs at the slower speed of the offloaded rows in the tables.

Setting the context: Ollama, llama.cpp, OpenClaw, OpenCode and Claude Code

These settings change often, so treat each tool's documentation (under Sources) as the reference; below are the parts that decide memory. Every combo page also prints Ollama and llama.cpp commands that follow the context and KV cache type you pick in its table; our page for gpt-oss-20b on the GeForce RTX 5060 Ti 16GB is an example.

Ollama

Set the server-wide context, then restart Ollama. On Windows, quit Ollama from its taskbar icon first, then in PowerShell:

$env:OLLAMA_CONTEXT_LENGTH="65536"; ollama serve

On macOS or Linux the same line is OLLAMA_CONTEXT_LENGTH=65536 ollama serve. For a permanent setting, use the context slider in the Ollama app or an account environment variable, as described in Ollama slow or forgetting your prompt? Context length defaults. Load the model and run ollama ps: CONTEXT should show the length you set, and PROCESSOR should read 100% GPU.

If the context table puts f16 over the line and q8_0 under it, quantize the KV cache. Ollama quantizes the KV cache only when the server starts with OLLAMA_KV_CACHE_TYPE (f16, q8_0 or q4_0) and flash attention on; the setting applies to every model the server runs (Ollama docs, checked 2026-09-29). In PowerShell:

$env:OLLAMA_CONTEXT_LENGTH="65536"; $env:OLLAMA_FLASH_ATTENTION="1"; $env:OLLAMA_KV_CACHE_TYPE="q8_0"; ollama serve

llama.cpp (llama-server)

The same line works in PowerShell, macOS and Linux:

llama-server -m model.gguf -c 65536 -ctk q8_0 -ctv q8_0 -fa on

Set -c yourself. Left unset, llama-server starts from the model's own maximum (256k for Qwen3.6 35B-A3B), and when memory runs short its automatic fitting (--fit, on by default) lowers the context first, down to a 4k floor set by --fit-ctx, before it moves any weights. The agent then gets far less context than it expects. With -c set, the context stays as given and --fit moves weights to system RAM instead, experts first for MoE models. The startup log shows the context it settled on. Leave out -ctk and -ctv to keep f16. Chat templates (--jinja), which the README names as what OpenAI-style tool calls need, are on by default.

OpenClaw

Point OpenClaw at Ollama's native address, http://localhost:11434, without /v1. Its documentation warns that the OpenAI-compatible /v1 URL breaks tool calling, and that models may then write tool calls out as plain text. For local models it discovers itself, OpenClaw normally sends 32k, and the value it sends replaces the server's: even with OLLAMA_CONTEXT_LENGTH at 64k, Ollama loads the model at 32k for OpenClaw until you raise that model's contextTokens or params.num_ctx, both described on its advanced Ollama page. The CONTEXT column of ollama ps shows which value won. Its managed llama.cpp setup uses a 64k context in its curated recipes.

OpenCode

OpenCode's documentation connects Ollama and llama-server as OpenAI-compatible providers, with a baseURL that ends in /v1. Give each local model a limit.context equal to the context the server really has, so OpenCode knows how much room is left. Its documentation also suggests raising Ollama's num_ctx when tool calls do not work.

Claude Code

ollama launch claude connects Claude Code through Ollama's Anthropic-compatible API. Ollama's page asks for a 64k+ context with local models, so set the length on the Ollama side as above and check CONTEXT in ollama ps.

How sure are these numbers?

The memory figures are calculated by formula from real GGUF file sizes, each file's header and each model's published configuration, so they carry no error band. The speeds are estimates with a band and a confidence label. Calibrated means ±12% for dense models fully on an NVIDIA or AMD card and ±20% for MoE models, partial offload and other hardware. Theoretical means ±30%, used where real speeds vary widely, such as MoE experts kept in system RAM. In this guide's three context tables, the cells with experts in system RAM are theoretical, gpt-oss-20b's 8k f16 cell on the RTX 5060 Ti 16GB is measured, and the rest are calibrated. Prompt-processing estimates on combo pages are rougher still, at ±40%.

That measured point is one public llama-bench run at a 2k context from llama.cpp's gpt-oss guide thread, not a CanRun run. It appears below next to CanRun's own 8k estimate band.

The memory for the whole length is reserved when the model loads, so the memory figures apply from the first turn; only the speed in each row assumes the window is full, and an agent session grows into that. The tables assume Windows with the card driving the display, 32GB of DDR5 system RAM, one model loaded and one request slot. Ollama's FAQ notes that memory scales with OLLAMA_NUM_PARALLEL times the context length, so parallel requests need more than the tables show. Ollama and llama.cpp make their own memory estimates, and llama.cpp's automatic fitting keeps some VRAM free on each card, so a model that only just fits here can end up partly in system RAM. On your PC, the PROCESSOR column of ollama ps or the llama.cpp startup log is the ground truth.

Models with a thinking mode face 1.5× stricter speed thresholds, which the legend below does not show, because thinking costs many more tokens per answer. Among the models in this guide, gpt-oss-20b always thinks; Qwen3.6 35B-A3B, Qwen3.8 27B, Qwen3.5 9B, Gemma 4 26B-A4B and Nemotron 3.5 Lightning 30B-A3B have a thinking mode; Qwen3 30B-A3B (2507) and Mistral Small 3.2 24B have none. CanRun does not rate how well a model codes or calls tools. The thresholds:

Where to go next

• Ollama slow or forgetting your prompt? Context length defaults: Ollama's default context, how it drops old messages, and every way to set the length.
• KV cache explained: how context length eats VRAM: how the KV cache grows and what q8_0 and q4_0 save.
• How much VRAM do local LLMs need? 8–32 GB tiers (2026): which models fit on each memory tier at 8k.

FAQ

How much context does a local coding agent need?

At least 64k tokens, by Ollama's documentation for agents and coding tools; the Qwen3.6 35B-A3B model card advises 128k or more for its thinking. Choose the longest context that still keeps the whole model on the GPU: at 64k with an f16 KV cache, Qwen3.8 27B needs 21.9 GB and gpt-oss-20b 14.2 GB, against 23.4 GB usable on a 24GB card and 15.4 GB on a 16GB card.

Why does my local agent take so long to start answering, or time out?

Usually because the model is reading a long prompt or still loading. Compare prompt_eval_duration and load_duration in Ollama's API response, keep the model loaded with OLLAMA_KEEP_ALIVE, and raise the agent's timeout (OpenClaw documents timeoutSeconds settings). If ollama ps shows a CPU/GPU split, lower the context or use a q8_0 KV cache until it reads 100% GPU.

Does a q8_0 KV cache hurt coding quality?

The Ollama FAQ calls the precision loss of q8_0 very small: it uses about half the memory of f16, while q4_0 uses about a quarter with a small-to-medium loss. CanRun does not measure output quality. A memory-based rule: use q8_0 when f16 does not fit the context you need, and q4_0 only when q8_0 still spills. On a 24GB card, q8_0 is what keeps Qwen3.6 35B-A3B on the GPU at 64k.

Why does the agent print tool calls as text instead of running them?

The connection, the context or the model can each cause it. OpenClaw's documentation says that pointing it at Ollama's OpenAI-compatible /v1 URL breaks tool calling and that models may then write the call as plain text; use the native address instead. OpenCode's documentation suggests raising Ollama's num_ctx when tool calls fail. The model has to support tool calls too: the Qwen3.6 35B-A3B and Qwen3 30B-A3B (2507) model cards describe tool calling, and the gpt-oss-20b card lists function calling.

Can I check a coding model that is not in CanRun's list?

Yes. Qwen3-Coder-30B-A3B-Instruct has the same layout as Qwen3 30B-A3B (2507) in its configuration (48 layers, four KV heads, 128 experts), and Devstral Small 2507 is fine-tuned from Mistral Small 3.1, with the same 40 layers and eight KV heads as Mistral Small 3.2 24B, so those rows are a close guide. For any other model, paste its Hugging Face id into the calculator and set the context to 64k; it runs the same memory calculation for your card. Results for loaded models are theoretical estimates (±30%), and the calculator cannot tell whether such a model has a thinking mode, so it does not apply the stricter thresholds for thinking models.

Sources

• Ollama docs: context length: at least 64k for agents, web search and coding tools, the VRAM-based default tiers, ollama ps, and the advice to avoid CPU offload.
• Ollama docs: OpenCode integration: ollama launch opencode, OpenCode requiring a context of 64k or more, and setting a 64k+ window for local models.
• Ollama docs: OpenClaw integration: ollama launch openclaw and the recommendation of at least 64k with local models.
• Ollama docs: Claude Code integration: ollama launch claude, the Anthropic-compatible API, and setting a 64k+ window for local models.
• Ollama FAQ: keeping models loaded (five minutes by default, OLLAMA_KEEP_ALIVE), KV cache quantization with its precision notes, and memory scaling with OLLAMA_NUM_PARALLEL.
• Ollama API docs: usage: prompt_eval_count, prompt_eval_cached_count, prompt_eval_duration (uncached prompt tokens), load_duration, timings in nanoseconds.
• Ollama source: api/types.go: the "prompt eval rate" that --verbose prints, computed from uncached prompt tokens.
• OpenClaw docs: Ollama: the native API and the warning that the /v1 URL breaks tool calling.
• OpenClaw docs: Ollama advanced configuration: the 32k cap for discovered local models, contextTokens and params.num_ctx taking precedence over Ollama's own default, and num_ctx in OpenAI-compatible mode.
• OpenClaw docs: local models: leaving room for the prompt, tools, history and output, 64k curated llama.cpp recipes, timeouts and cold starts.
• OpenCode docs: providers: Ollama and llama-server as OpenAI-compatible providers, limit.context, and the num_ctx tip for tool calls.
• llama.cpp server README: -c (0 loads the model's own length), --fit and --fit-ctx, -ctk/-ctv, -fa, --jinja and tool calls, cache_prompt, --ctx-checkpoints, the timings object with cache_n and prompt_per_second.
• llama.cpp source: server-context.cpp: for sliding-window and hybrid models, falling back to a context checkpoint or to full prompt re-processing.
• llama.cpp PR #16653: automatic fitting: --fit lowers an unset context before moving weights, and keeps MoE dense weights on the GPU first.
• llama.cpp discussion #15013: performance on NVIDIA CUDA: pp512 and tg128 results for NVIDIA cards.
• llama.cpp: guide to running gpt-oss: the measured gpt-oss-20b point on the RTX 5060 Ti 16GB.
• Qwen3.6-35B-A3B model card: hybrid layout, experts, native context, the 128k advice for thinking, thinking on by default, tool calling.
• Qwen3.8-27B model card: hybrid layout, KV heads, native context, thinking on by default.
• Qwen3.5-9B model card: the same hybrid layout of Gated DeltaNet and Gated Attention layers.
• Qwen3-30B-A3B-Instruct-2507 model card: 48 layers, four KV heads, non-thinking only, tool calling.
• Qwen3-Coder-30B-A3B-Instruct configuration: 48 layers, four KV heads, 128 experts with 8 per token, the same as Qwen3-30B-A3B-Instruct-2507.
• Devstral Small 2507 model card and configuration: fine-tuned from Mistral Small 3.1, 128k context, 40 layers and eight KV heads.
• Mistral Small 3.2 24B configuration: 40 layers, eight KV heads, no sliding window.
• gpt-oss-20b model card and configuration: function calling, alternating sliding-window and full-attention layers.
• Gemma 4 26B-A4B configuration: 25 sliding-window and 5 full-attention layers, 1024-token window.
• Nemotron 3.5 Lightning 30B-A3B configuration: 6 attention layers among 52, the rest Mamba and MoE.