Thinking models on local hardware: the speed cost and how to turn thinking off

A thinking mode does not change how much memory a model needs. It changes how many tokens each reply takes, so at the same speed you wait longer for the answer. On the RTX 3060 12GB, Qwen3 14B and Phi-4 both fit on the GPU at 8k with about the same estimated speed, yet Qwen3 14B Runs well and Phi-4 Runs great, because CanRun holds every model with a thinking mode to 1.5× stricter speed thresholds. The two lines that turn thinking off are --think=false in Ollama and --reasoning off in llama-server; gpt-oss cannot turn it off, only lower its reasoning effort.

Which models think: always, on by default, off by default

Many recent open models can write out their reasoning before the answer, in a block that runtimes show apart from the answer. In Ollama's command line it appears between Thinking... and ...done thinking. markers. For each model in our catalog, CanRun records whether it has such a mode and how it is switched, based on the model card and the chat template, the file that turns your messages into the model's prompt format. The models fall into five groups:

• Built to always think. gpt-oss-20b and gpt-oss-120b, the DeepSeek R1 distills, LFM2.5 2.6B and GLM-5.3 Flash, among others. gpt-oss lets you pick a reasoning effort of low, medium or high but has no off setting, and its chat template uses medium unless told otherwise. GLM-5.3 Flash likewise takes a reasoning_effort of low, high or max and has no off setting. The original DeepSeek R1 distill cards (Qwen 14B, Qwen 32B and Llama 8B) go further and recommend forcing every reply to start with a <think> tag, because the models sometimes skip their thinking, which can make answers worse. The later R1 0528 Qwen3 8B card says this is no longer required.
• On by default, can be turned off. Qwen3 8B, 14B and 32B, the Qwen3.5, Qwen3.6 and Qwen3.8 models and EXAONE 4.5 33B, among others.
• Off by default, can be turned on. The Gemma 4 models and EXAONE 4.0 (32B and 1.2B). Gemma 4 thinks when its system prompt starts with a <|think|> token, and EXAONE 4.0 when its template receives enable_thinking=True.
• The model decides. By default, HyperCLOVA X SEED Think 14B decides for itself whether to reason before answering, according to its model card. Our catalog marks the MiniCPM5 models the same way.
• No thinking mode. Llama 3.1 8B, Gemma 3, Phi-4 and Qwen3 30B-A3B (2507). The last is the Instruct-2507 release, whose card says it supports only non-thinking mode. Phi-4 carries a reasoning tag in our catalog, but its card shows a plain chat format with no thinking block. A use-case tag is not a thinking mode.

Thinking changes nothing about memory. The weights and the KV cache per token are the same whether the model thinks or not, so whether it fits depends only on the model, the quant and the context. At 8k, Qwen3 14B needs 10.5 GB and Phi-4 needs 11.0 GB. Qwen3 32B needs 22.0 GB, and DeepSeek R1 Distill Qwen 32B, which its card says is built on Qwen2.5 32B, needs 22.1 GB. What thinking costs is time, and context, which a later section covers.

Combo pages mark these models with a thinking-mode label, and model pages add a note under the table. In the calculator, a model loaded from Hugging Face gets the normal thresholds, because CanRun cannot tell from the files whether it thinks. When you enter a model by hand, tick the thinking-mode box if its card mentions one.

Why CanRun's verdicts are 1.5× stricter for thinking models

CanRun's verdicts compare the estimated generation speed with fixed thresholds, shown in the legend near the end. Those thresholds assume that most of what the model writes is the answer you read. A thinking model writes its reasoning first, at the same speed per token, and the answer starts only when the reasoning ends. If a model thinks for three times as many tokens as its answer, the whole reply takes four times as long, and three quarters of that wait passes before the first word of the answer.

How long a model thinks depends on the question, the model and settings such as gpt-oss's effort level, and CanRun has no data that would predict it. So instead of guessing a token count, it raises every speed threshold by 1.5× for any model with a thinking mode: Runs great, Runs well and the Won't run floor all need one and a half times the speed in the legend. The factor is a policy choice written into our engine spec, not a measurement. A 1.5× bar matches a reply whose thinking is half as long as its answer, so on long thinks the real wait is longer than the verdict suggests.

The rule also covers models that are off by default, because Ollama and llama-server, the two runtimes this guide covers, turn thinking on anyway. In Ollama's source, a local model that can switch thinking keeps it on whenever a request leaves think unset, and Ollama's tests list Gemma 4 as such a model. llama-server enables thinking whenever the chat template can switch it, unless you start it with --reasoning off.

The grid crosses six models with five NVIDIA cards. Phi-4 and Gemma 3 27B have no thinking mode, Qwen3 14B and Qwen3 32B think by default, Gemma 4 26B-A4B is off by default and gpt-oss-20b always thinks. Every cell assumes an 8k context, an f16 KV cache, 32GB of DDR5 system RAM and Windows.

On the GeForce RTX 3060 12GB, Phi-4 Runs great while Qwen3 14B, with about the same estimated speed, Runs well. Qwen3 14B's estimate clears the normal bar for Runs great but not the stricter one. Phi-4 needs 11.0 GB against the 11.4 GB this card can use, so it fits with little margin; check ollama ps for 100% GPU. The GeForce RTX 5060 Ti 16GB has more memory bandwidth, 448 GB/s against 360 GB/s, and there Qwen3 14B's estimate just clears the stricter bar, so it Runs great.

The GeForce RTX 3090 shows the same split one size up. Gemma 3 27B Runs great, while Qwen3 32B, the larger model, fits but its estimate sits just under the stricter bar, so it Runs well. DeepSeek R1 Distill Qwen 32B, outside the grid, gets the same result there. On the GeForce RTX 4090, Qwen3 32B's estimate is less than a tenth higher, but that lands it just over the line, so it Runs great. A gap that small says more about where the line sits than about the two cards.

The GeForce RTX 4060 row shows the rule at the low end. Gemma 4 26B-A4B keeps thinking off in its own template, but CanRun still applies the stricter thresholds. On this 8GB card part of its experts sit in system RAM, a setup that needs the speed of Runs great just to reach Runs well, and one and a half times that with a thinking mode. Its estimate clears even the stricter bar, though only just, so it Runs well, a theoretical estimate. Qwen3 32B is rated Won't run on the same card. With most of its weights in system RAM, its estimate sits above the normal Won't run floor but below the stricter one. The cell suggests Q2_K instead, a Heavy loss quant, which Runs slowly. Gemma 3 27B, with no thinking mode, Runs slowly there.

gpt-oss-20b always thinks, yet it is fast for its size. With part of its experts in system RAM it still Runs well on the RTX 4060 and the RTX 3060 12GB, both theoretical estimates. On the 16GB and 24GB cards it fits fully and Runs great, and public benchmarks back those cells.

The verdict follows the model, not your settings. Switch thinking off and CanRun still rates Qwen3 14B on the RTX 3060 12GB Runs well; the combo pages add a note that a model may do one grade better with thinking off. By our estimates, Qwen3 14B with thinking off answers at about the pace of Phi-4 on that card. Our page for Qwen3 14B on the GeForce RTX 3060 12GB has the run commands.

How to turn thinking off in Ollama and llama.cpp

Ollama

In PowerShell or any other terminal, pass --think=false to ollama run, or type /set nothink inside a session (/set think turns it back on):

ollama run qwen3:14b --think=false

--hidethinking is a different switch. It only hides the reasoning in the terminal; the model still thinks, so the reply takes as long as before.

Through the API, add think to the body of a request to /api/chat or /api/generate. false asks for no thinking "if the model permits it", in the words of Ollama's docs, true asks for it, and a string picks a named level for models that define them.

{
  "model": "qwen3:14b",
  "messages": [{ "role": "user", "content": "Summarize this paragraph in two lines." }],
  "think": false
}

Leaving think out uses the model's default. ollama show qwen3:14b lists thinking under Capabilities and, for models that report them, the accepted levels and the default. Set think yourself rather than rely on that default: for local models that can switch thinking, Ollama keeps it on when a request does not say otherwise, Gemma 4 included.

Apps that talk to Ollama's OpenAI-compatible endpoint (/v1/chat/completions) cannot send think; Ollama ignores it there. Send "reasoning_effort": "none" instead. In the Python library, pass think=False to chat().

gpt-oss has no off value in Ollama, which lists only low, medium and high for it, with medium as the default. Run ollama run gpt-oss:20b --think=low or send "think": "low" to shorten its reasoning.

llama.cpp

Since March 2026, llama-server, the server in llama.cpp, has a direct switch. --reasoning off (short form -rea off) passes enable_thinking=false to the chat template, the variable that the Qwen3, Qwen3.5, Qwen3.6, Gemma 4 and EXAONE 4.0 templates read. The default, auto, turns thinking on whenever the template can switch it. Older builds used --reasoning-budget 0 as the off switch. A start command with thinking off looks like this:

llama-server -hf unsloth/Qwen3-14B-GGUF:Q4_K_M -c 8192 -ngl all -fa on --reasoning off

For a single request, the server README documents "reasoning_effort": "none" and template arguments such as the following:

{
  "messages": [{ "role": "user", "content": "Summarize this paragraph in two lines." }],
  "chat_template_kwargs": { "enable_thinking": false }
}

For gpt-oss, --reasoning-effort low hands the level to its chat template, which otherwise uses medium. To cap thinking rather than switch it off, --reasoning-budget N ends the reasoning after N tokens (-1, the default, means no limit), and --reasoning-budget-message adds a line before the end-of-thinking tag to tell the model it ran out.

The model's own switches

• Qwen3 8B, 14B and 32B read /no_think and /think at the end of a user or system message, turn by turn, as long as the template switch is on. With /no_think the reply still carries an empty thinking block. enable_thinking=False is the hard switch, and with it the tags do nothing.
• Qwen3.5 and Qwen3.6 do not officially support those tags, according to their cards. Use the runtime switch.
• Gemma 4 thinks only when the system prompt starts with <|think|>. Ollama's Gemma 4 renderer adds that token when thinking is on, so the runtime switch still decides.
• EXAONE 4.0 reasons when its template receives enable_thinking=True and answers directly otherwise.
• HyperCLOVA X SEED Think 14B takes force_reasoning or skip_reasoning as template arguments, and its card describes an approximate length limit you can write into the message.
• gpt-oss reads its effort from the system prompt, as a line such as Reasoning: low. Ollama and llama.cpp write that line for you from the options above.
• The DeepSeek R1 distills have no switch of their own. In Ollama, --think=false still works for them: Ollama's deepseek-r1 template answers it with an empty thinking block, so the model skips its reasoning, which the original distill cards warn can make answers worse.

Keeping thinking on and still getting answers quickly

Thinking is worth its cost on some tasks. Qwen3's card describes thinking mode for complex logical reasoning, math and coding, and non-thinking mode for efficient general-purpose dialogue. When you want to keep it on, the ways to clear the stricter bar are the usual ways to gain speed, and the data shows which of them help.

A q8_0 KV cache. At 8k, Qwen3 32B on the RTX 3090 sits just under the stricter bar with an f16 cache and just over it with q8_0, which reads a smaller cache for every token, so it Runs great. DeepSeek R1 Distill Qwen 32B does the same. That speed gain has not been checked against public measurements, so read it as a small edge; how to set the cache type is in KV cache explained: how context length eats VRAM.

One quant step down, within Good balance. Qwen3 32B at IQ4_XS needs 20.0 GB instead of 22.0 GB, and with the smaller file it Runs great on the RTX 3090. It does not always work: Qwen3 14B at IQ4_XS still Runs well on the RTX 3060 12GB. For what the step costs in quality, see GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download.

A smaller model from the same family. Even under the stricter thresholds, Qwen3 8B Runs great on the RTX 3060 12GB and, just over the line, on the 8GB RTX 4060, and Qwen3.5 9B Runs great on the RTX 3060 12GB. On the RTX 4060, Qwen3 8B needs 6.5 GB at 8k against 7.4 GB usable, so confirm 100% GPU in ollama ps.

A mixture-of-experts (MoE) model with few active parameters. An MoE model reads only a few billion of its parameters for each token, so it generates quickly for its size. gpt-oss-20b Runs great on the RTX 5060 Ti 16GB, a measured cell, and Qwen3.6 35B-A3B, which thinks by default, fits fully on the RTX 3090 and Runs great there. On the RTX 5060 Ti 16GB, Qwen3.6 35B-A3B needs 21.8 GB, so part of its experts sit in system RAM and it Runs well, a theoretical estimate. For that setup, see Running MoE models bigger than your GPU: gpt-oss-120b with system RAM.

Thinking tokens fill the context too

While the model writes a reply, its reasoning sits in the same context window as your messages. Every thinking token adds to the KV cache, the memory that holds the context, just like any other token. Model cards plan for long reasoning. Qwen3's card recommends allowing 32k output tokens for most queries, and the Qwen3.5 and Qwen3.6 cards do the same, with up to 80k for hard math and programming. DeepSeek ran its R1 evaluations with up to 32k generated tokens. A 4k or 8k window can fill up in the middle of a long think.

Ollama makes this more likely on smaller cards. Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). When a reply fills the window, Ollama shifts it and keeps generating, so the start of the conversation drops out; Ollama slow or forgetting your prompt? Context length defaults shows how to raise the limit.

Earlier turns cost less. The Qwen3, Qwen3.5 and Gemma 4 cards say the history should keep only the final answers of past turns, and their chat templates drop the old thinking; Gemma 4 keeps it in tool-call turns. Qwen3.6's template also drops it by default, but the model was trained to use kept thinking and has a preserve_thinking option. llama-server's --reasoning-preserve, on by default, keeps reasoning in the history for templates that support it.

The table gives the memory each thinking model in this guide needs at 32k.

The dense Qwen3 models and the DeepSeek R1 distill use full attention on every layer, so their KV cache grows with every token and a long think costs them the most: at 32k, Qwen3 32B holds 8.6 GB of cache. gpt-oss-20b and Gemma 4 use sliding-window layers that keep only recent tokens, and Qwen3.6 35B-A3B keeps a full cache on only some of its layers, so their 32k caches stay small; gpt-oss-20b needs only 0.8 GB. For the mechanics, see KV cache explained: how context length eats VRAM.

On a 16GB card, a dense 14B thinking model still has room for its full window. The table shows Qwen3 14B on the GeForce RTX 5060 Ti 16GB by context and KV cache type; every cell is calibrated.

At 32k with an f16 KV cache, Qwen3 14B needs 14.7 GB against the 15.4 GB this card can use. In CanRun's calculation it stays fully on the GPU, with little margin, so check ollama ps. With an f16 cache it drops from Runs great at 8k to Runs well from 16k, because each new token reads a larger cache; a q8_0 cache keeps 16k at Runs great. On the RTX 3060 12GB, the same 32k with an f16 cache does not fit, and part of the model moves to system RAM. A q4_0 KV cache keeps it on that card's GPU, with little margin.

Where to go next

• How much VRAM do local LLMs need? 8–32 GB tiers (2026): which models fit on each memory tier.
• Ollama slow or forgetting your prompt? Context length defaults: Ollama's default context and how to change it.
• KV cache explained: how context length eats VRAM: what the KV cache costs per token and what q8_0 and q4_0 save.
• Model pages such as Qwen3 14B and gpt-oss-20b list every card.

How sure are these numbers?

The memory figures are calculated by formula from real GGUF file sizes, each file's header and each model's published configuration, so they carry no error band. The speeds are estimates with a band and a confidence label. Calibrated means ±12% for dense models fully on an NVIDIA or AMD card and ±20% for MoE models, partial offload and other hardware. Theoretical means ±30%, for setups whose real speed varies a lot, such as MoE experts in system RAM. In this guide's grid, most cells are calibrated, gpt-oss-20b on the 16GB and 24GB cards is measured, and the cells with experts in system RAM are theoretical.

The 1.5× factor is the least certain part. It is a fixed policy rather than something CanRun measured, and the real cost of thinking changes from one question to the next. Several verdicts in this guide also sit close to the stricter line, where a few percent of estimated speed moves a cell by one level. With thinking off, read a cell's speed band against the normal thresholds in the legend. With thinking on, how long you wait for the answer depends on how long the model thinks, which no speed estimate can show.

The measured point below is gpt-oss-20b on the GeForce RTX 3090, from llama.cpp's gpt-oss guide thread (llama-bench, llama.cpp's benchmark tool, at a 2k context; not a CanRun run), next to CanRun's own 8k estimate band.

llama-bench times a fixed run of generated tokens, so a figure like this says how fast tokens come out, not how many a reply takes. To see what thinking costs on your own prompts, run the same question with ollama run <model> --verbose, once with thinking on and once with --think=false. After each reply, Ollama prints the total duration, the number of generated tokens (eval count) and the generation speed (eval rate).

All of this assumes Windows with the card driving the display, 32GB of DDR5-5600 dual-channel system RAM, an 8k context, an f16 KV cache, one model loaded, one request slot and a llama.cpp-class runtime. Verdict names in the text refer to the 8k, f16 KV baseline unless a context is given; for other contexts, read the table. The legend below shows the normal thresholds. For models with a thinking mode, each speed in it is multiplied by 1.5.

FAQ

Does thinking mode need more VRAM?

Not for the model itself: the weights and the KV cache per token are the same with thinking on or off. Qwen3 14B needs 10.5 GB at 8k either way, which fits on a 12GB card such as the RTX 3060 12GB. Thinking does fill the context faster, and a longer context needs a larger KV cache: at 32k with an f16 cache, Qwen3 14B needs 14.7 GB.

Is a thinking model slower per token?

Not by itself. At the same context length, speed per token depends on the model's size, the quant and the card's memory bandwidth. But a long think fills the context, and each new token then reads a larger KV cache, so generation slows as the reply grows: for Qwen3 14B on the RTX 5060 Ti 16GB, CanRun's estimate goes from Runs great at 8k to Runs well at 32k (see the table above). A thinking model also feels slower because it writes more tokens for each reply, and the answer starts only after the thinking ends.

Why does gpt-oss keep thinking when I set think to false?

gpt-oss has no off setting. Its card offers low, medium and high reasoning effort, and Ollama lists only those three values for it, so use --think=low or "think": "low" instead. With other models, check that you used --think=false and not --hidethinking, which only hides the reasoning. If your app uses the /v1 endpoint, think is ignored; set reasoning_effort to none. Qwen3.5 and Qwen3.6 do not support Qwen3's /no_think tag, so use the runtime switch for them. DeepSeek R1 in Ollama does stop thinking with --think=false, but only by skipping the reasoning, which the original R1 distill cards warn can make answers worse. An old Qwen3 download can keep thinking even with --think=false: Ollama's source notes that an older Qwen3 template always opens a thinking block, so pull the model again.

Should I turn thinking off?

Turn it off for chat and quick questions, and keep it on for math, code and problems that need several steps. That split follows Qwen3's card, which describes thinking mode for complex reasoning, math and coding and non-thinking mode for efficient general dialogue. If you keep it on, a q8_0 KV cache, one quant step down, a smaller model or an MoE model can still clear the stricter bar, as shown above.

Can I limit how long a model thinks?

Partly. llama-server's --reasoning-budget N ends the reasoning after N tokens, and gpt-oss takes a low effort level in both Ollama and llama.cpp. HyperCLOVA X SEED Think 14B's card suggests ending the message with a line such as Think for maximum 1024 tokens. Ollama's documented think values are on, off and the named levels a model defines; its docs say numbers are not supported. If Qwen3 seems to think forever, check the sampling settings: its card warns that greedy decoding (temperature 0) in thinking mode can cause endless repetition, and suggests a presence penalty.

Sources

• Ollama docs: thinking: the think field (true, false "if the model permits it", null, named levels; numbers not supported), the model default from /api/show, gpt-oss levels low, medium and high with medium as default, and think in the Python library.
• Ollama blog: thinking: --think, --think=false, /set think, /set nothink, --hidethinking for scripting, and disabling thinking in DeepSeek R1.
• Ollama v0.40.4 source: cmd/cmd.go: the --think flag (true, false or a level), --hidethinking only suppressing display, the Thinking... and ...done thinking. markers, the levels shown by ollama show, and the --verbose timings.
• Ollama v0.40.4 source: cmd/interactive.go: /set think and /set nothink.
• Ollama v0.40.4 source: server/model_thinking.go and its tests: local models that can switch thinking default to on, Gemma 4 included, and the note that an older Qwen3 template always opens a thinking block.
• Ollama v0.40.4 source: openai/openai.go: the OpenAI-compatible request has no think field, and reasoning_effort none turns thinking off.
• Ollama v0.40.4 source: model/renderers/thinking.go: gpt-oss (harmony) levels low, medium and high, default medium.
• Ollama v0.40.4 source: model/renderers/gemma4.go: the <|think|> token added when thinking is on.
• Ollama v0.40.4 source: server/routes.go: thinking turned on when a request leaves think unset and the model has no default of its own, and shifting the window by default when a reply fills it.
• llama.cpp: server README: --reasoning, --reasoning-effort, --reasoning-budget, --reasoning-budget-message, --reasoning-preserve, and the chat_template_kwargs and reasoning_effort request fields.
• llama.cpp source: server-context.cpp: thinking is enabled unless --reasoning off is given and the template supports it.
• llama.cpp PR #20297: handle reasoning budget: --reasoning on and off, a real token budget and the budget message, merged in March 2026; before it, --reasoning-budget 0 only turned thinking off.
• Qwen3-14B model card: thinking on by default, enable_thinking=False, the /think and /no_think tags, the empty thinking block, the 32k output length, history without thinking, what each mode is for, and the warning against greedy decoding with the presence penalty advice.
• Qwen3.5-35B-A3B model card and Qwen3.6-35B-A3B model card: thinking on by default, no official support for the soft switch, enable_thinking through chat_template_kwargs, history without thinking for Qwen3.5, kept thinking and preserve_thinking for Qwen3.6, output lengths of 32k and up to 80k.
• Qwen3-30B-A3B-Instruct-2507 model card: non-thinking mode only.
• gpt-oss-20b model card and its chat template: low, medium and high reasoning effort set in the system prompt, medium when unset.
• DeepSeek-R1-Distill-Qwen-32B model card: distilled into Qwen2.5 32B, forcing a <think> start because skipping the thinking can make answers worse (the Qwen 14B and Llama 8B distill cards say the same), a maximum generation length of 32k in evaluation.
• DeepSeek-R1-0528-Qwen3-8B model card: forcing a <think> start is no longer required.
• Gemma 4 26B-A4B model card and its chat template: thinking turned on by <|think|> at the start of the system prompt, enable_thinking off by default, past thoughts dropped except in tool-call turns.
• EXAONE 4.0 32B model card: reasoning mode with enable_thinking=True.
• HyperCLOVA X SEED Think 14B model card: the model decides by default, force_reasoning and skip_reasoning, and the Think for maximum N tokens line for an approximate length limit.
• Phi-4 model card: a plain chat format without a thinking block.
• llama.cpp: guide to running gpt-oss: llama-bench results on RTX cards, the source of the measured gpt-oss-20b point on the RTX 3090.