Tokens per second for local LLMs: what's fast enough and what sets the speed

Generation speed, the tokens per second a model writes, depends mostly on your card's memory bandwidth and on whether the whole model fits in VRAM. Llama 3.1 8B fits on an 8GB RTX 4060 and Runs great there, backed by a public benchmark. Qwen3 14B does not fit on that card, so part of it runs from system RAM and it Runs slowly, a calibrated estimate. Prompt processing, how fast a model reads your input, is a second speed that depends on the GPU's compute instead.

How fast is fast enough: CanRun's verdict lines

Local runtimes report two speeds. Generation speed (Ollama's eval rate, llama-bench's tg) is how fast the reply appears. Prompt processing (prompt eval rate, or pp) is how fast the model reads what you send, and it decides how long you wait before the first word. CanRun's verdicts use generation speed, estimated at an 8k context; prompt processing has its own section further down.

The legend below shows where each verdict starts. CanRun set the top line above typical reading pace, so a reply appears faster than you can follow it. Between the two upper lines you can read along, though a long answer or a block of code takes a while. Under the bottom line a reply of a few paragraphs takes minutes, and CanRun rates the setup Won't run.

Speed is not the only test. Runs great is kept for models that run fully on the GPU; the legend lists the limits for a model split between GPU and system RAM and for a CPU-only PC.

For models with a thinking mode, every line is 1.5 times higher. They write out their reasoning before the answer, so you wait for many more tokens per reply. That covers models that always think, such as gpt-oss, and models that can switch thinking on and off, such as Qwen3 14B, where it is on by default. On the RTX 3060 12GB, Qwen3 14B Runs well, while Phi-4, a 14B model without a thinking mode and with about the same estimated speed, Runs great. How to turn thinking off, and what it costs to leave it on, is in Thinking models on local hardware: the speed cost and how to turn thinking off.

A token is a piece of a word. How much text one token holds depends on the model's tokenizer and on the language, so the same tok/s can put text on screen at different rates for two models.

What sets generation speed: memory bandwidth and bytes per token

To write one token, the GPU reads every weight the model uses for that token, plus the KV cache, the memory that holds the conversation so far. Memory bandwidth, how many gigabytes per second the card can read from its VRAM, limits how often it can do that. CanRun's estimate is the card's bandwidth times an efficiency factor, divided by the bytes read per token. For a dense model fully on an NVIDIA card, CanRun counts on about seven-tenths of the rated bandwidth; the factor differs by hardware class and was fitted to public benchmarks.

The rule of thumb follows from that division: double the bandwidth and the estimate roughly doubles; double the bytes per token and it halves. Bandwidth spans a wide range, from 272 GB/s on the RTX 4060 to 1,792 GB/s on the RTX 5090.

For a dense model, the bytes read per token are nearly the whole set of weights on the GPU. These are the four models used in the grid further down, at their default quant:

The Parameters column shows why Qwen3 30B-A3B (2507) is the odd one out. It is a mixture-of-experts (MoE) model: a router picks a few expert blocks for each token, so only about a tenth of its parameters are active per token (the column gives both counts). It reads far fewer bytes per token than its file size suggests, but the whole file still has to fit in memory, because the router can pick any expert.

Quantization works the same way. A Q8_0 file of Qwen3 14B is 15.7 GB against 9.0 GB at Q4_K_M, so on a 24GB card where both fit, CanRun's Q8_0 estimate is about three-fifths of the Q4_K_M one. A smaller quant is faster as well as smaller, at some cost in quality; GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download compares the levels.

Longer chats get slower

Each new token also reads the whole KV cache, and the cache grows with every token in the conversation. CanRun's speed figures assume the context is filled to the length in question. By that calculation a fresh chat runs faster than the 8k figure and a long one slower. The table for Qwen3 30B-A3B (2507) on the GeForce RTX 3090 shows the effect on an MoE model, where it is larger:

With an f16 KV cache, the estimate at 32k is a little over half the 8k one. This model reads only a tenth of its weights per token, so the cache makes up a large share of each read. For a dense model such as Llama 3.1 8B, the weights make up most of the read at 8k, so the drop over the same range is smaller, about a third by CanRun's calculation. At 32k the model needs 22.4 GB and still fits in the 23.4 GB the card can use. At 64k it needs 25.9 GB, so in CanRun's calculation part of the experts move to system RAM and the speed falls further, a theoretical estimate.

The q8_0 and q4_0 columns come out faster because CanRun counts the smaller cache it reads. That part of the calculation has not been checked against public measurements, so read those columns mainly for memory. What they save is explained in KV cache explained: how context length eats VRAM.

The same model on different cards

The grid puts four models on eight setups, ordered from the least memory bandwidth to the most: a PC with no graphics card (dual-channel DDR5-5600 system RAM), then seven NVIDIA cards. Every cell assumes an 8k context and an f16 KV cache, with 32GB of DDR5 system RAM on the graphics card rows. Where a cell is labelled measured, the first figure is a public benchmark and the line under it is CanRun's estimate.

When the model fits, bandwidth sets the order

Llama 3.1 8B fits on every graphics card in the grid and Runs great on all of them. CanRun's estimate for it climbs with bandwidth down the rows. The RTX 4060 row shows a higher first figure than the RTX 4060 Ti 16GB row only because the 4060's cell carries a public benchmark, measured at a 4k context, where a card runs faster than at CanRun's 8k. Its estimate, on the line below, is the lower one.

Memory size does not change the speed of a model that already fits. The GeForce RTX 4060 Ti 16GB and the GeForce RTX 5060 Ti 16GB both have 16GB, but the newer card reads 448 GB/s against 288 GB/s. Qwen3 14B Runs well on the older card and Runs great on the newer one.

The older RTX 3060 12GB has less memory than the RTX 4060 Ti 16GB but more bandwidth, 360 GB/s, so CanRun estimates it quicker with Llama 3.1 8B and Qwen3 14B. The extra memory still counts for models that need it. gpt-oss-20b runs fully on the GPU of the RTX 4060 Ti 16GB and Runs great, while on the RTX 3060 12GB part of its experts go to system RAM and it Runs well, a theoretical estimate.

When it does not fit, bandwidth stops deciding

Gemma 3 27B needs 17.8 GB at 8k, more than a 16GB card can use. On the five graphics cards with 16GB or less, part of it runs from system RAM and it Runs slowly. The order no longer follows bandwidth. The RTX 5070 reads 672 GB/s, more than either 16GB card, yet its estimate is lower than both, because its 12GB keep less of the model on the GPU. On the RTX 3090 the whole model fits and Runs great.

Qwen3 30B-A3B shows the MoE case. On the cards with 16GB or less, CanRun's calculation keeps the experts that do not fit in system RAM, and each token reads only the few it needs from there. It Runs well on all five of those cards, as theoretical estimates. On the RTX 3090 it fits fully and Runs great, and CanRun estimates it faster there than Qwen3 14B, although its file is about twice as large. The gain is smaller than the gap in bytes read, because each token also pays a fixed time for every expert it runs in every layer, which CanRun fits to public benchmarks. On a top-end card such as the RTX 5090 that fixed time is a larger share of each token, so the estimate rises less than the bandwidth does.

On the CPU row, Llama 3.1 8B Runs slowly because its estimate is under the Runs well line. A CPU-only setup is also capped at Runs slowly, however high its speed: Qwen3 30B-A3B's estimate on that row clears the Runs well line, yet it is still rated Runs slowly (a calibrated estimate). Which models fit on each memory size is covered in How much VRAM do local LLMs need? 8–32 GB tiers (2026).

Why spilling into system RAM costs so much speed

When a model does not fit, the runtime keeps the layers that do not fit in system RAM, and every token has to pass through them. Dual-channel DDR5-5600, the system RAM CanRun assumes, reads 90 GB/s, about a third of the RTX 4060's 272 GB/s. CanRun also counts on only about half of the RAM's rated bandwidth for that part, against about seven-tenths of a card's VRAM bandwidth. The time spent on the RAM part adds to the time on the GPU part, so the slow part sets the pace.

Qwen3 14B fits on the RTX 4060 Ti 16GB and Runs well there. On the 8GB RTX 4060, which has almost the same bandwidth, it needs 10.5 GB against 7.4 GB usable, so about two-fifths of its weights go to system RAM, and it Runs slowly, a calibrated estimate. In CanRun's calculation that RAM share takes about three-quarters of the time for each token. The ladder for Qwen3 14B on the GeForce RTX 4060 shows the speed following the share left in RAM:

Q8_0 leaves the most in system RAM and is the slowest. IQ4_XS moves a little more onto the GPU but still Runs slowly. Only Q2_K, which fits all but a sliver, gets past the line to Runs well, and Q2_K belongs to the Heavy loss quality group. CanRun rates a dense model with less than nine-tenths of its weights on the GPU no higher than Runs slowly, whatever the estimate.

MoE models lose less, because each token reads only the chosen experts from system RAM. That is why Qwen3 30B-A3B still Runs well on the RTX 4060, a theoretical estimate. Running MoE models bigger than your GPU: gpt-oss-120b with system RAM covers that setup, and Ollama or LM Studio not using your GPU? Check the split and fix the offload shows how to see where your runtime put the model and what to shrink first.

Prompt processing: why the first word can take a while

Before it writes anything, the model reads the whole prompt: the system prompt, the chat history and your new message. It takes these tokens in large batches, so this step is limited by the GPU's compute rather than its memory bandwidth. Generation goes one token at a time and waits on memory.

llama.cpp's public scoreboard for NVIDIA cards, which runs Llama 2 7B at Q4_0 through llama-bench, shows the split. The RTX 4090 has only a little more bandwidth than the RTX 3090 (1,008 GB/s against 936 GB/s). In that scoreboard it generates less than a fifth faster but processes prompts more than twice as fast. The RTX 3060 (12GB) generates nearly a fifth faster than the RTX 4060 Ti (8GB), while the RTX 4060 Ti reads prompts about one and a half times as fast.

The wait before the first word is roughly the new prompt tokens divided by the prompt speed, plus loading time if the model was not in memory yet. In a chat it stays short, because runtimes reuse the part of the prompt they have already processed: Ollama reports those tokens as prompt eval cached, and llama-server as cache_n. It grows when you paste a long document or run an agent, whose instructions and tool descriptions alone are long; Local models for OpenClaw and coding agents: context length and VRAM covers that case.

CanRun's combo pages show a prompt speed as a rough estimate (±40%), shown only when the whole model is on the GPU. It starts from the card's FP16 compute, which sets the speed for a 7B model, and is divided by the model's size, because reading a prompt takes about two operations per parameter for every token. A 32B model reads prompts at about a fifth of a 7B model's speed on the same card. A 1B model gains less than its size suggests, because part of the time per token does not depend on the model, and that part weighs more on fast cards. A mixture-of-experts model counts as more than its active parameters, because each expert gets only a slice of every batch of prompt tokens. The estimate is for a short prompt; as the context fills, attention takes a larger share of the work and prompt speed falls.

Measuring speed on your own PC

Ollama

Run a model with --verbose. The command is the same in PowerShell, macOS and Linux:

ollama run llama3.1:8b --verbose

In a session that is already running, /set verbose turns on the same output. After each reply Ollama prints its timings:

• eval rate is the generation speed, the figure to compare with CanRun's tables.
• prompt eval rate is the prompt speed, counted only over the tokens it actually processed. prompt eval cached shows how many it reused.
• load duration is the time spent loading the model. It counts toward total duration but not toward either rate.

Ask for a reply of several paragraphs, since a very short reply gives a noisy rate. While the model is loaded, run ollama ps in a second window. If CanRun shows the model fully on the GPU, PROCESSOR should read 100% GPU, and a CPU/GPU split explains a low rate. For an MoE model that CanRun places partly in system RAM, a split is expected. The rate also drifts down as a chat grows, for the reason in the context table above.

llama.cpp: llama-bench and llama-server

llama-bench ships with llama.cpp and runs a fixed test. By default it reads a 512-token prompt (pp512) and generates 128 tokens (tg128), five times each, and reports the averages. Those tests start from an empty context, so for most models tg128 comes out above CanRun's 8k figure. The -d option fills the KV cache first: -d 0,8192 prints the empty-context rows plus rows marked @ d8192, which are the closest to CanRun's baseline. On Windows, in PowerShell from the llama.cpp folder:

.\llama-bench.exe -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -d 0,8192

On macOS and Linux, leave out .\ and .exe; -m takes a local GGUF file instead of -hf. llama-bench puts every layer on the GPU by default. For a model larger than your VRAM, set -ngl to fewer layers or give -fitt a memory margin in MiB so it fits the model for you, and use -ncmoe for MoE models. llama-bench leaves tokenization and sampling out of its timings, so real use can be slightly slower than it reports.

llama-server returns a timings object with each response: predicted_per_second is the generation speed, prompt_per_second the prompt speed and cache_n the prompt tokens reused from the cache.

LM Studio

LM Studio's REST API (POST /api/v1/chat) returns a stats object with tokens_per_second, the generation speed, and time_to_first_token_seconds. When the request had to load the model first, model_load_time_seconds appears too.

Comparing your number with CanRun's

Match the conditions before you compare. Use the same quant (a Q4_K_XL file differs in size from a Q4_K_M, so its speed differs too), the same context depth (CanRun's verdicts assume 8k), the same backend (CUDA, ROCm, Vulkan or Metal) and nothing else loaded on the card. A laptop GPU is often a different chip from the desktop card with the same name, with less memory and bandwidth (the GeForce RTX 4090 Laptop reads 576 GB/s), so compare with its Laptop entry; power and heat limits can lower it further. If your rate is far below the band in CanRun's table, check ollama ps for a CPU/GPU split first, since that costs the most speed.

How sure are these numbers?

Every speed on CanRun is an estimate with a band and a label. Measured means a public benchmark of that card, model and quant; it is shown first, with CanRun's estimate next to it. Calibrated means ±12% for dense models fully on an NVIDIA or AMD card and ±20% for MoE models, partial offload and other hardware. Theoretical means ±30%, used where real speeds vary widely, such as MoE experts in system RAM. Against 12 reference benchmarks, the speed formula's mean error is 4.2% and its largest 10.5%. For MoE models fully on the GPU, against 49 public results from 2026 llama.cpp builds, it is 8.3% and 19.9%. Public results outside that reference set can miss by more; the RTX 4090 result below is under CanRun's band.

One efficiency factor per hardware class is a simplification. In the public results CanRun was fitted to, slower cards got closer to their rated bandwidth than top-end cards did. Two public results for Llama 3.1 8B from LocalScore, both at a 4k context, show it. The RTX 4090 has close to four times the RTX 4060's bandwidth, yet its result is less than two and a half times as high. The RTX 4090's result sits below CanRun's band even allowing for the shorter context. Llama 3.1 8B Runs great on both cards either way.

Each is one public result measured with LocalScore's own method, not a CanRun run, so treat any one number, ours or a benchmark's, as a point inside a range. Our page for Llama 3.1 8B on the GeForce RTX 4090 lists the commands for that setup.

All estimates assume Windows with the card driving the display, 32GB of DDR5-5600 dual-channel system RAM, an 8k context filled to that length, an f16 KV cache, one model loaded, one request at a time and a llama.cpp-class runtime. Verdict names in this guide refer to that 8k baseline; for other contexts, read the tables. The verdict lines are in the legend near the top.

FAQ

How many tokens per second is good for a local LLM?

For chat, CanRun's top verdict line, in the legend in the first section, sits above typical reading pace: a speed above it puts text on screen faster than you can read it, and the band between the two upper lines is usable if you read along. Thinking models need lines 1.5 times higher, because they write their reasoning before the answer. For agents and long documents, prompt speed matters as much as generation speed. On an 8GB RTX 4060, Llama 3.1 8B clears the top line and Runs great.

Why is my local model slower than the benchmarks I see online?

Usually the conditions differ. Many published figures, including llama-bench's default tg128, start from an empty context, while a long chat makes each token read a larger KV cache. Other causes are a different quant, a different backend, a laptop GPU that is a smaller chip than its desktop namesake and, costing the most, a model that spilled into system RAM. Check PROCESSOR in ollama ps, then compare the eval rate from ollama run --verbose with the band in CanRun's table.

Does more VRAM make a model run faster?

Only when the extra memory stops a spill into system RAM. Once a model fits, bandwidth sets the speed: Qwen3 14B Runs well on the RTX 4060 Ti 16GB and Runs great on the RTX 5060 Ti 16GB, two 16GB cards with different bandwidth.

Why does the reply take so long to start?

The model reads the whole new prompt before it writes, and that step depends on the GPU's compute. A long pasted document or an agent's instructions take the longest, and loading the model counts too (load duration in Ollama's verbose output). In a chat, runtimes reuse what they processed before, so later turns start faster unless something near the start of the prompt changes.

Why do MoE models like Qwen3 30B-A3B run faster than smaller dense models?

They read only the experts the router picks for each token, about a tenth of the weights for Qwen3 30B-A3B, so each token costs fewer bytes. The whole file must still fit in memory. On a 24GB RTX 3090 it fits and Runs great, and CanRun estimates it faster there than the dense Qwen3 14B.

Sources

• Ollama source: api/types.go: the verbose summary, with eval rate (generated tokens over generation time), prompt eval rate (uncached prompt tokens over prompt time), prompt eval cached and load duration.
• Ollama source: cmd/cmd.go and cmd/interactive.go: the --verbose flag of ollama run and the /set verbose command.
• llama.cpp: llama-bench README: the pp512 and tg128 defaults, five repetitions, -d to prefill the KV cache, -hf and -m, -ngl (all layers by default), -fitt (off by default) and -ncmoe, and timings that leave out tokenization and sampling.
• llama.cpp: server README: the timings object with predicted_per_second, prompt_per_second and cache_n.
• llama.cpp discussion #15013: performance on NVIDIA CUDA: pp512 and tg128 results for Llama 2 7B Q4_0 on the RTX 4090, RTX 3090, RTX 3060 and RTX 4060 Ti.
• LM Studio docs: chat with a model: the stats fields of POST /api/v1/chat.
• LocalScore: RTX 4060 and LocalScore: RTX 4090: the public Llama 3.1 8B results shown in the measured boxes.
• Qwen3-30B-A3B-Instruct-2507 model card: total and activated parameters, 128 experts with 8 activated, non-thinking mode only.
• Qwen3-14B model card: thinking mode on by default, with a switch to turn it off.
• Phi-4 model card: a 14B model, used here as the example without a thinking mode.