CanRun

How CanRun calculates: verdicts, speed estimates and error ranges

Every answer on CanRun comes from the same calculation, whether you see it on the calculator, a GPU page, a model page or a guide. This page explains how that calculation works, how well it has matched real benchmarks and where it can be wrong.

Verdicts

Each model gets one of four verdicts. The baseline is the Q4_K_M quant, an 8k context and an f16 KV cache, the setup most people start with.

How to read the verdicts

Runs great
Fully on the GPU at 20 tok/s or more
Runs well
Fully on the GPU at 8–20 tok/s; MoE experts in system RAM at 20 tok/s or more; or at least 90% on the GPU at 8 tok/s or more
Runs slowly
2–8 tok/s; CPU-only; less than 90% on the GPU; or MoE experts in system RAM below 20 tok/s
Won't run
Does not fit, or under 2 tok/s

For models with a thinking mode, which write out their reasoning before answering, the speed thresholds are 1.5 times higher, because you wait for many more tokens per answer. Models that let you switch thinking on and off use the same thresholds. We check the official model card and chat template to see whether a model has a thinking mode. When a model does not reach a good verdict at Q4_K_M, the pages also show what a smaller quant or more system RAM would change.

Memory

The memory a model needs is the sum of three parts:

  • The model weights. We start from the real size of the GGUF file on Hugging Face for each quant, not an estimate from the parameter count. On a graphics card we leave out what llama.cpp does not put there: the input embedding table, which it always keeps in system RAM, and the multi-token prediction (MTP) block some newer models carry, which it loads only when you turn MTP on (EXAONE 4.5's MTP block is not used by llama.cpp yet, so it is never loaded). We read both sizes from each file's header. On a Mac we count the whole file. The one exception is a model whose numbers you type into the calculator yourself: there is no file behind it, so we estimate from the parameter count (see Known limits below).
  • The KV cache, which holds the conversation so far. It grows with the context length, and how fast it grows depends on the model's architecture: models with sliding-window layers need much less at long contexts. We calculate it from each model's published configuration.
  • A compute buffer, the working memory the runtime needs while it generates text.

That total is compared with the memory the card can actually use, not the number on the box. A card that also drives your display keeps some memory back for the operating system, and a Mac shares its unified memory with macOS. On a Mac or another unified-memory PC we count about 70% of the memory as the GPU's. On a DGX Spark we count what Linux and the desktop leave free, starting from the lowest figure in owners' public boot logs. When the model does not fit, we work out whether part of it can run from system RAM and how much that slows it down.

Speed

Generating each token means reading the model's weights and the KV cache from memory, so memory bandwidth sets the pace. Our estimate starts from the card's bandwidth and the bytes read per token, then applies an efficiency factor for each class of hardware (NVIDIA, AMD, Intel and Apple GPUs, and CPUs). Those factors are calibrated against public benchmarks. When part of the model runs from system RAM, the slower RAM bandwidth is added for that part.

A mixture-of-experts (MoE) model reads only its shared layers and the few experts the router picks for each token, so we count those bytes from the tensor sizes in each GGUF file's header. On top of the reading time, each token pays a small fixed cost for every expert it runs in every layer, for the routing and the many small matrix operations. For small MoE models on fast hardware that cost can be as large as the reading time itself. We fitted it for each class of hardware to public benchmarks from 2026 llama.cpp builds.

Confidence labels

Every speed on CanRun is a range with a label:

  • Measured: a benchmark of the same GPU, model and quant, either a public one or one we ran ourselves and published with its raw log. We show where it came from and under what conditions.
  • Calibrated: our estimate, checked against benchmarks of similar setups. The range is ±12% for dense models running fully on an NVIDIA or AMD card, and ±20% for other hardware, mixture-of-experts (MoE) models and setups that spill into system RAM.
  • Theoretical: an estimate for a setup we have not been able to check against benchmarks yet, shown with a ±30% range.

Prompt processing speed, the time before the first token, is shown as a rough ±40% range. It starts from the GPU's FP16 compute, which sets the speed for a 7B model, and is divided by the model's size, because reading a prompt takes about two operations per parameter for every token: a 70B model reads prompts at about a tenth of a 7B model's speed. Part of the time per token does not depend on the model, so small models gain less than their size suggests. A mixture-of-experts (MoE) model counts as more than its active parameters, and the more experts it has per active one, the more it counts, because each expert gets only a slice of every batch of prompt tokens. For Radeon cards we use the FP16 matrix figure AMD publishes. Apple's M5-series chips, which have a matrix accelerator in each GPU core, get their own factor, fitted to their public benchmarks; on them and on the Strix Halo, K-quant files such as Q4_K_M read prompts more slowly than Q4_0 or Q8_0 files, and we count that. Apple does not publish its chips' compute, so for Apple chips we use the theoretical figures in Wikipedia's Apple silicon table; their GPUs run FP16 at the same rate as FP32.

How accurate it has been

We check the calculation against real data, and the check runs automatically every time the code or data changes:

  • Model file sizes: an average error of 0.7% and at most 1.6% across 11 real GGUF files.
  • KV cache: an exact match with the formula for all 7 architectures we checked.
  • Speed: an average error of 4.2% and at most 10.5% across 12 reference benchmarks. The largest misses were an Apple M5 Pro and a CPU-only setup, both about 10% too low.
  • MoE speed: an average error of 8.3% and at most 19.9% across 49 public benchmarks from 2026 llama.cpp builds, on NVIDIA, AMD and Apple GPUs, the DGX Spark and the Strix Halo.
  • Prompt processing: an average error of 11.6% and at most 35% across 91 public benchmarks from 2026 llama.cpp builds, with models from 0.6B to 120B, dense and MoE, on the same kinds of hardware. Before October 16, 2026, the estimate was the same for every model on a card; on the 83 of these benchmarks it covered (the DGX Spark had no estimate then), it was off by 92% on average and up to eight times too high.

We first checked these figures on August 9, 2026. On October 15, 2026, we replaced the two Apple speed benchmarks, which came from a 2023 llama.cpp build, with results from current builds. Later that day we changed how MoE speed is calculated and added the MoE check. On October 16, 2026, we refitted the Strix Halo factors to results from current llama.cpp builds and added four Strix Halo results to the MoE check. The same day we made the prompt-processing estimate depend on the model's size and added the prompt-processing check. The automated check keeps the figures from getting worse.

Known limits

  • MoE models with their experts in system RAM are estimated with as many experts on the GPU as fit and only the rest in RAM. That matches how current llama.cpp and Ollama split these models on their own, and the --n-cpu-moe value in our commands follows the same split. Against 35 public benchmarks on NVIDIA cards the average error was about 10%, and 89% landed within ±20%. Some setups were off by about a third depending on the OS, backend and RAM configuration, so these results stay labeled theoretical (±30%) (checked October 1, 2026, and again on October 15, 2026).
  • MoE speed depends on how the model is built, not only on its size. We use one fixed cost per expert for each class of hardware, so a model whose layers do more or less work than most can land up to about 20% off. Q8_0 files tend to run a little faster than we estimate. Our figures follow 2026 llama.cpp builds; Mac MoE benchmarks from 2025 builds came in up to about 30% below them.
  • Prompt processing is the roughest figure. It is for a short prompt with llama.cpp's default batch of 512 tokens. As the context fills, attention takes a larger share of the work and prompt speed falls. Larger batches (-ub 2048, which llama.cpp's gpt-oss guide suggests) make MoE models read prompts faster than we show. Prompt speed also changes quickly between llama.cpp builds: on NVIDIA cards, MoE prompt processing got about 40% faster in mid-2026. For Intel Arc cards we have no recent benchmark of a standard model, so their prompt figures are the least certain.
  • The backend can change speed. Our Strix Halo figures follow llama.cpp's Vulkan backend, which is faster there. In public comparisons on the same machine and build, ROCm generated text about 3% to 12% slower.
  • Benchmarks differ in how they measure. Some average over a long context and some use a short prompt, so we record the context of every benchmark and compare like with like.
  • Your own machine can differ: laptop power and heat limits, single-channel RAM, drivers and background apps can all change real speeds. The ranges absorb some of this, not all of it.
  • Runtimes make their own decisions. Ollama and llama.cpp estimate memory in their own way and may split a model between GPU and CPU differently from our calculation. llama.cpp's automatic fitting, which current Ollama versions also go through, keeps some VRAM free on each GPU, so a model that only just fits in our calculation can end up partly in system RAM. On your machine, ollama ps shows what actually happened.
  • The DGX Spark's GPU share is on the low side. We start from the lowest memory figure in owners' public boot logs, which comes from a DGX OS release whose kernel reserves several gigabytes more than earlier ones, and subtract what the desktop uses at idle. On an earlier or fixed kernel, or without a desktop, a somewhat larger model can fit than we show. The 64GB model has no public figure yet, so we subtract the same fixed amount as on the 128GB model; part of that amount normally grows with memory size, so a little more may be left in practice.
  • Turning on MTP needs more memory. When a model fits fully on a graphics card, our figures assume llama.cpp's default, which leaves the MTP block unloaded, so the block is left out. On a Mac, and when part of the model sits in system RAM, the block is counted. If you turn on MTP speculative decoding, the block and the KV cache it uses come on top; combo pages where the model fits fully on the graphics card show the block's size under the memory chart. For EXAONE 4.5, llama.cpp does not use the MTP block yet, so it cannot be turned on.
  • Models you enter by hand in the calculator have no real file behind them, so we estimate the file size from the parameter count and the bits per weight of each quant. Checked against the models in our list, the estimate lands within 9% of the real file for dense models, but it can miss by more for MoE models (34% in the worst case we found). The KV cache uses the layer counts you enter. Models whose config.json has kv_lora_rank (MLA, as in DeepSeek V3) store the KV cache differently, so numbers entered by hand can't calculate them correctly. With no real file and no benchmark to check against, results for these models are always shown as theoretical estimates (±30%).
  • Several GPUs in one machine are a calculator-only beta. We assume llama.cpp's default, which splits the layers across the cards in proportion to their free memory. VRAM adds up, but memory bandwidth doesn't. Each token passes through the cards one after another, so when you mix different cards, the slower card's share drags the whole run down. There are few public benchmarks of multi-GPU setups to check against, and only a handful for mixed cards, so these results are always labeled theoretical (±30%). GPU, model and combo pages assume a single graphics card.
  • Linked DGX Spark units (two to four, in the GPU page tables and the calculator) are worked out for llama.cpp RPC, which splits the layers across the units. Memory adds up, but each token passes through every unit, so we start from the time one unit takes per token and add a fixed time for every hop between units, measured on two units linked over TCP. With a faster link setup (RDMA, or CPU idle states turned off), our figure came in up to about 16% below the measured speed, and at very long contexts it came in above it. Three and four units have no public benchmarks, so we extend the two-unit figure, and every linked figure is a theoretical estimate. In those tables, a model whose baseline quant (Q4_K_M, or MXFP4 for gpt-oss and DeepSeek V4 Flash) already fits fully on the GPU of one unit with the same memory shows “One unit is enough” instead of a linked verdict, because linking would only slow it down. vLLM tensor parallelism, which NVIDIA's official guides use, can be faster than llama.cpp; we don't calculate it.

Independence

Verdicts and recommendations are calculated from the data above. They do not depend on ads, affiliate links or any arrangement with a manufacturer. Where prices appear, they are shown with the date they were checked.

Sources