How much VRAM do local LLMs need? GPU requirements for 8–32 GB (2026)

A model runs fully on the GPU when the weights it puts there (nearly all of the model file), its KV cache (the memory that holds the context) and a small compute buffer (the runtime's working memory) fit in the VRAM your card can use. Llama 3.1 8B at Q4_K_M, a 4-bit quant, needs 6.3 GB with an 8k-token context, inside the 7.4 GB an 8GB RTX 4060 can use, and it Runs great. At 8k, 12GB holds 12B to 14B models (Qwen3 14B with little margin); 16GB holds gpt-oss-20b and, at IQ4_XS, Gemma 4 26B-A4B; 24GB holds the dense Qwen3.6 27B and most 30B-class models, such as Qwen3 30B-A3B; and 32GB mainly adds room for longer context. With 32GB of system RAM, gpt-oss-120b fits on none of these cards; in CanRun's calculation it needs 64GB of system RAM on a 12GB or larger card and 96GB on an 8GB card.

How much VRAM a model needs: weights, KV cache and buffer

Three things share a graphics card's memory while a model runs: the model's weights at the quant you downloaded (Q4_K_M, a 4-bit GGUF quant, is the usual default), the KV cache, which holds keys and values for every token in the context, and a compute buffer, the runtime's working memory, which grows slightly with the context.

The weights on the GPU are the model file minus two parts that llama.cpp leaves off the graphics card. It always keeps the input embedding table, which turns tokens into vectors, in system RAM. Some newer models, such as Qwen3.5, also ship a multi-token prediction (MTP) block, which llama.cpp loads only when you turn MTP on. EXAONE 4.5 carries one too, but llama.cpp does not use it yet and never loads it. CanRun reads both sizes from each GGUF file's header; together they come to a few hundred megabytes for most models and around a gigabyte for the Qwen3.5 models. The largest, Qwen3.5 122B, passes two gigabytes even at Q4_K_M. Models that reuse the embedding table as their output layer, such as Gemma 3, keep a copy on the GPU, so for them the weights equal the file.

CanRun's verdicts use Q4_K_M (MXFP4 for gpt-oss, its native format), an 8k context and an f16 (unquantized 16-bit) KV cache. These are the totals for the models this guide follows:

Compare the Total column with the VRAM your card can use, not the number on the box. When the card also drives a Windows display, CanRun sets some VRAM aside for the desktop (the note under the table gives the amount). That leaves an 8GB card such as the RTX 4060 with 7.4 GB and a 12GB card such as the RTX 3060 12GB with 11.4 GB.

The weights are the largest part. CanRun takes file sizes from the real GGUF files on Hugging Face. The quantize table in llama.cpp gives a rule of thumb: Q4_K_M averages a little under five bits per weight, so a Q4_K_M file in gigabytes is about three-fifths of the parameter count in billions. Llama 3.1 8B comes to 4.9 GB, of which 4.6 GB goes on the GPU, and Qwen3 14B to 9.0 GB. Q8_0 is roughly one gigabyte per billion parameters, so Qwen3 14B at Q8_0 is 15.7 GB.

At 8k the KV cache is small next to the file (1.1 GB for Llama 3.1 8B, 1.3 GB for Qwen3 14B), but it grows with the context, as the section on context length shows.

Qwen3 14B on the RTX 3060 12GB shows how the parts fill a 12GB card: it needs 10.5 GB at 8k against 11.4 GB usable, so in CanRun's calculation it fits with an f16 KV cache, with less than a gigabyte free, and Runs well. Check ollama ps after loading it: PROCESSOR should read 100% GPU. Our page for Qwen3 14B on the GeForce RTX 3060 12GB has the commands.

Active parameters do not set the memory need: mixture-of-experts (MoE) models such as gpt-oss-20b use only a few billion parameters per token, but all of their weights must be in memory (the Parameters column shows both counts). Macs use unified memory instead, of which the GPU gets about 70% by default in CanRun's calculation; see Running local LLMs on a Mac (M4, M5): how much unified memory?.

What fits on 8, 12, 16, 24 and 32 GB graphics cards

The grid crosses six popular models with eight NVIDIA cards. Every verdict assumes an 8k context, an f16 KV cache, 32GB of DDR5 system RAM and Windows. Each cell shows a verdict, an estimated speed band and a confidence label; cells labelled theoretical get an extra note under the grid.

The second grid uses the same cards and assumptions for twelve newer or larger models people often look up: Gemma 4, Qwen3.5, Qwen3.6, Qwen3.8, Qwen3 8B and 32B, and Llama 3.3 70B.

8GB: RTX 4060

On the GeForce RTX 4060, Llama 3.1 8B fits fully with about a gigabyte to spare (6.3 GB against 7.4 GB usable; ollama ps should show 100% GPU) and Runs great, a measured cell. Our page for Llama 3.1 8B on the GeForce RTX 4060 covers longer contexts.

Gemma 4 12B needs 7.8 GB, more than the card's 7.4 GB, but with over 90% of it on the GPU (partial offload) it Runs well, a calibrated estimate. At IQ4_XS it needs 7.1 GB and fits fully with little margin (check ollama ps); it still Runs well. Gemma 3 12B needs 8.4 GB and Qwen3 14B needs 10.5 GB, so part of their weights go to system RAM too; both Run slowly, as calibrated estimates. So do the dense Qwen3.6 27B, Qwen3.8 27B and Gemma 4 31B.

Two of the MoE models, gpt-oss-20b and Qwen3 30B-A3B, run on this card only with part of their expert weights in system RAM, and both cells are theoretical estimates (see the MoE section). gpt-oss-20b always thinks before it answers, so it is held to stricter thresholds, and it only just clears the bar for Runs well; Qwen3 30B-A3B Runs well too. Outside the grid, Gemma 4 26B-A4B and Qwen3.6 35B-A3B Run well the same way, both theoretical estimates. gpt-oss-120b is rated Won't run.

The GeForce RTX 5060 Ti 8GB shares its name with a 16GB card, so check the memory on the box. It can use the same 7.4 GB as the GeForce RTX 4060 but has more bandwidth (448 GB/s against 272 GB/s). Gemma 4 12B at IQ4_XS fits fully there too and Runs great. Gemma 4 has a thinking mode (off by default), so it faces 1.5× stricter speed thresholds, and only the faster card's estimate clears the bar for Runs great.

In CanRun's calculation, fit depends only on usable memory, and every 8GB card in our catalog can use the same 7.4 GB, so the same models fit on each; the table below, highest bandwidth first, shows how speed alone shifts the verdicts.

12GB: RTX 3060 12GB, RTX 4070 and RTX 5070

The grid has three 12GB cards: the GeForce RTX 3060 12GB, the GeForce RTX 4070 and the GeForce RTX 5070. Each can use 11.4 GB, so in CanRun's calculation they fit the same models.

Gemma 3 12B and Gemma 4 12B need 8.4 GB and 7.8 GB, leaving room to spare, and both Run great on all three. Even at 32k with an f16 KV cache, Gemma 4 12B needs only 8.4 GB, inside the 11.4 GB these cards can use. Qwen3 14B fits at 8k with little margin (see the bar in the first section, and check ollama ps). At 16k with an f16 KV cache it spills over; see the context table further down. gpt-oss-20b needs 12.3 GB, more than a 12GB card can use, so the expert weights that do not fit go to system RAM and the cell reads Runs well as a theoretical estimate.

The MoE Gemma 4 26B-A4B and Qwen3.6 35B-A3B keep part of their experts in system RAM and Run well, both theoretical estimates. The dense Qwen3.6 27B and Gemma 4 31B need 17.2 GB and 19.5 GB, so on both the RTX 3060 12GB and the RTX 5070 part of each stays in system RAM, and they are rated Runs slowly.

Memory bandwidth largely sets the speed. The RTX 3060 12GB has 360 GB/s, the RTX 4070 504 GB/s and the RTX 5070 672 GB/s. Qwen3 14B Runs well on the RTX 3060 12GB but Runs great on the RTX 4070 and the RTX 5070. The gap comes from the thinking-mode rule: Qwen3 14B can think before it answers, which costs many more tokens per answer, so CanRun holds it to 1.5× stricter speed thresholds. The faster cards clear them and the RTX 3060 12GB does not. Phi-4, a 14B model without a thinking mode, has about the same estimated speed yet Runs great on all three. It needs 11.0 GB at 8k and fits on all three with little margin, so check ollama ps.

Our catalog's other 12GB cards, such as the RTX 4070 Super, the RTX 3080 12GB and Intel's Arc B580, can use the same 11.4 GB and fit the same models; the table lists them by bandwidth.

16GB: RTX 5060 Ti 16GB

On the GeForce RTX 5060 Ti 16GB, gpt-oss-20b needs 12.3 GB against 15.4 GB usable, so it fits fully on the GPU, in line with its model card, which says it runs within 16GB of memory. It Runs great, a measured cell. Our page for gpt-oss-20b on the GeForce RTX 5060 Ti 16GB has the full breakdown.

Qwen3 14B Runs great here as well, and the extra memory goes to context: at 32k with an f16 KV cache it needs 14.7 GB, still inside the 15.4 GB this card can use but with little margin (check ollama ps).

Gemma 4 12B Runs great here, and even at Q8_0 it fits fully, needing 13.4 GB, and Runs well. Gemma 4 26B-A4B needs 17.7 GB at Q4_K_M, over the 15.4 GB the card can use, so in CanRun's calculation part of its experts go to system RAM; it Runs well, a theoretical estimate. At IQ4_XS it needs 14.3 GB and fits fully with about a gigabyte to spare, and it Runs great, a calibrated estimate. Gemma 4 31B needs 19.5 GB and Runs slowly, even at IQ4_XS (17.6 GB), where part of it still stays in system RAM. For the three sizes side by side, see Gemma 4 12B vs 26B-A4B vs 31B: differences, VRAM and which size fits your GPU.

The dense Qwen3.6 27B and Qwen3.8 27B need 17.2 GB and 17.6 GB at Q4_K_M, so part of each runs from system RAM and both are rated Runs slowly, as calibrated estimates. At IQ4_XS (15.8 GB and 15.7 GB) they still need more than the card can use, but enough stays on the GPU that both Run well, again calibrated; ollama ps should show a CPU/GPU split. The MoE Qwen3 30B-A3B and Qwen3.6 35B-A3B need 19.8 GB and 21.8 GB, so the experts that do not fit go to system RAM, and both Run well as theoretical estimates; Qwen3.6 35B-A3B does so even at IQ4_XS. For the version differences, see Qwen3.6 vs Qwen3.8, 27B vs 35B-A3B: differences and the VRAM each needs.

Every 16GB card in our catalog can use the same 15.4 GB, so in CanRun's calculation they fit the same models: the AMD Radeon RX 9060 XT 16GB also Runs great with gpt-oss-20b, and only speed separates the cards, as the table shows.

The table includes laptop GPUs by their own memory: the GeForce RTX 4090 Laptop has 16 GB and sits here rather than with the desktop card, and the GeForce RTX 5090 Laptop has 24 GB and sits in the 24GB table. Look up the laptop chip itself.

24GB: RTX 3090 and RTX 4090

The GeForce RTX 3090 and the GeForce RTX 4090 can each use 23.4 GB. That is enough for Qwen3 30B-A3B, which needs 19.8 GB: it runs fully on the GPU and Runs great on both. Our page for Qwen3 30B-A3B (2507) on the GeForce RTX 3090 shows how it holds up at longer contexts.

The dense Qwen3.6 27B, Qwen3.8 27B and Gemma 4 31B need 17.2 GB, 17.6 GB and 19.5 GB, so they fit fully and Run great on both the RTX 3090 and the RTX 4090, as do the MoE Gemma 4 26B-A4B and Qwen3.6 35B-A3B.

The dense Qwen3 32B needs 22.0 GB at 8k, so it fits on the RTX 3090 with a little over a gigabyte to spare (confirm 100% GPU in ollama ps). A longer context with an f16 KV cache pushes it partly off the GPU, and Ollama's default on 24GB cards is already 32k; for that case, see Ollama slow or forgetting your prompt? Context length defaults. At 32k with an f16 KV cache, Gemma 4 31B (21.8 GB) and Qwen3.6 27B (19.0 GB) stay fully on the GPU, and Qwen3.6 35B-A3B (22.6 GB) fits with little margin (check ollama ps).

Speed can still separate two cards with the same memory. Qwen3 32B and DeepSeek R1 Distill Qwen 32B are both rated Runs well on the RTX 3090 but Runs great on the RTX 4090. Both have a thinking mode, so they face 1.5× stricter speed thresholds, which the faster RTX 4090 clears and the RTX 3090 does not. gpt-oss-120b is rated Won't run on both.

Our catalog's other 24GB cards, such as the RTX 3090 Ti, the RX 7900 XTX and the RTX 5090 Laptop, can use the same 23.4 GB and fit the same models; the table lists them by bandwidth.

32GB: RTX 5090

For the six grid models, the RTX 5090 gets the same 8k verdicts as the 24GB cards. Its extra memory goes to context: Qwen3 32B at 32k with an f16 KV cache needs 28.7 GB, inside the 31.4 GB this card can use but over the 23.4 GB of a 24GB card. At the 8k baseline it Runs great on the RTX 5090.

Qwen3.6 27B and Qwen3.8 27B fit fully here even at Q8_0 (28.4 GB and 28.4 GB) and Run great, while on the RTX 3090 those files spill into system RAM and Run slowly. At 128k with an f16 KV cache, Gemma 4 31B needs 30.7 GB, inside the 31.4 GB this card can use, with little margin (check ollama ps), and Qwen3.6 35B-A3B stays fully on the GPU even at its native 256k context.

At Q4_K_M, 32GB still does not hold a dense 70B model. Llama 3.3 70B needs 45.2 GB at that quant, so it runs with part of its weights in system RAM and is rated Runs slowly on the RTX 5090. On the RTX 4090 it is rated Won't run at Q4_K_M because partial offload is too slow. Only Heavy loss quants fit Llama 3.3 70B fully on the RTX 5090. gpt-oss-120b is still rated Won't run with 32GB of system RAM, even here.

The whole catalog, card by card

The table below counts every model in CanRun's catalog by verdict for each of the eight cards (the caption gives the total) and names the largest model that fits on the GPU alone, with its quant.

At 8k, the set of models that fit fully on the GPU grows at each step from 8GB to 24GB, and the 32GB RTX 5090 also fits Nemotron 3.5 Lightning 30B-A3B, which needs 25.1 GB, more than the 23.4 GB a 24GB card can use. On the RTX 4090 some of its expert weights go to system RAM, and CanRun rates it Runs well, a theoretical estimate; on the RTX 5090 it fits fully on the GPU and Runs great, also a theoretical estimate. The counts change as models join the catalog.

Memory targets by model size

• 8B-class models: 8GB of VRAM. Gemma 4 12B also fits fully on 8GB at IQ4_XS, with little margin.
• 12B to 14B models: 12GB, or 16GB for longer context.
• gpt-oss-20b, or Gemma 4 26B-A4B at IQ4_XS, fully on the GPU: 16GB.
• Dense 27B to 31B models such as Qwen3.6 27B, Qwen3.8 27B and Gemma 4 31B: 24GB.
• 30B-class models: 24GB for most, such as Qwen3 30B-A3B and Qwen3.6 35B-A3B, or 32GB for longer context. Nemotron 3.5 Lightning 30B-A3B needs 32GB to fit fully on the GPU even at 8k.

MoE models: the whole file counts, but experts can sit in system RAM

In a mixture-of-experts (MoE) model, the bulk of each layer is split into many expert blocks, and a router picks a few for every token. Qwen3 30B-A3B has 128 experts with 8 active per token, according to its model card; gpt-oss-20b and gpt-oss-120b work the same way, as do Gemma 4 26B-A4B and Qwen3.6 35B-A3B. Fit needs the whole file, because the router can pick any expert; speed depends mostly on the active part.

When the model does not fit fully in VRAM (weights, KV cache and buffer together), CanRun's calculation first keeps the shared and attention weights, the KV cache and the buffer on the GPU. It then fills the remaining VRAM with as many layers of experts as fit, leaving one layer's worth free as a safety margin, and keeps only the experts of the other layers in system RAM. CanRun reads the expert and non-expert sizes from each GGUF file's header. llama.cpp's --n-cpu-moe N option makes this split: it keeps the experts of the first N layers on the CPU, and our combo pages print the N that CanRun calculated. Qwen3 30B-A3B (2507) needs 19.8 GB, against 11.4 GB usable on the RTX 3060 12GB, so the bar shows the GPU filled and the remaining experts in system RAM:

In this mode the verdict follows a stricter rule, shown in the verdict legend further down: the best verdict is Runs well, which needs the speed Runs great needs on a full GPU; slower than that is Runs slowly, and very slow is Won't run. The gpt-oss models have a thinking mode, so their thresholds are 1.5× stricter again. Qwen3 30B-A3B (2507) on the GeForce RTX 3060 12GB reaches Runs well, the best verdict under that rule, as a theoretical estimate.

Treat these speeds as theoretical estimates (±30%). For each token, CanRun's calculation reads the chosen experts of the layers in system RAM at RAM speed, and everything on the GPU, including the experts kept there and the KV cache, at GPU speed. Against 35 public benchmarks on NVIDIA cards, the average error was about 10% and 89% landed within ±20%. Some setups were off by close to 40%, depending on the OS, the backend and the RAM configuration, so the label stays theoretical.

For the largest MoE model in the grid, system RAM becomes the limit. The gpt-oss-120b file is 63.4 GB. With 32GB of system RAM the experts that do not fit in VRAM have nowhere to go, so it is rated Won't run on every card in the grid, from the 8GB RTX 4060 to the 32GB RTX 5090. CanRun also checks 64GB, 96GB and 128GB of RAM. In its calculation, 64GB is enough on every card in the grid from 12GB up, while the 8GB RTX 4060 needs 96GB. The line under each badge in the grid shows that RAM size with the verdict it would get, a theoretical estimate. Its model card targets a single 80GB GPU.

Current runtimes make this kind of split on their own. llama.cpp b7410 (December 2025) and later fit the model to your VRAM by default (--fit), putting the non-expert weights on the GPU first and moving expert weights to the CPU only as needed. Ollama 0.30 and later runs models through llama.cpp and keeps part of the experts on the GPU the same way; older Ollama versions split whole layers instead, which is much slower. ollama ps shows what Ollama chose under PROCESSOR. Combo pages print the full llama.cpp command with the split CanRun assumed (-ngl all --n-cpu-moe N); for the setup above, see our page for Qwen3 30B-A3B (2507) on the GeForce RTX 3060 12GB.

How quantization and context length change the answer

On a given card, two settings move a model across the lines above: the quant you download and the context you run it at.

Quantization: a smaller file, at a cost in quality

A quant stores each weight in fewer bits, which makes the file smaller. CanRun sorts quants into four quality groups: Near-lossless (Q5_K_M, Q6_K, Q8_0, and MXFP4, gpt-oss's native format), Good balance (Q4_K_M and IQ4_XS), Noticeable loss (3-bit quants such as Q3_K_M and IQ3_XS) and Heavy loss (2-bit quants such as Q2_K and IQ2_M). A Q8_0 file is well over one and a half times the Q4_K_M file: Gemma 3 12B is 12.5 GB at Q8_0 against 7.3 GB at Q4_K_M. The ladder below shows each step for Gemma 3 12B on the GeForce RTX 4060; every cell is calibrated.

At Q8_0 and Q4_K_M the model spills into system RAM; Q4_K_M, the baseline, is rated Runs slowly. IQ4_XS needs 7.7 GB, still over the 7.4 GB the card can use, so it remains partial offload, but with at least 90% on the GPU the table shows Runs well. Q2_K fits fully and the table shows Runs great, but it belongs to the Heavy loss group. CanRun suggests a Heavy loss quant only when every other quant is rated Won't run; the better first move is one level down within Good balance, from Q4_K_M to IQ4_XS. For a closer comparison of the levels, see GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download.

More VRAM lets you move up in quality instead, at a cost: a bigger file is slower per token and leaves less room for context, so CanRun does not suggest it on its own. Qwen3 14B at Q8_0 needs 16.8 GB, more than the 15.4 GB of a 16GB card but inside the 23.4 GB of a 24GB card, where in CanRun's calculation it stays fully on the GPU.

Context length: the KV cache grows

In a model with full attention on every layer, such as Llama 3.1 8B, Qwen3 14B, Qwen3 32B and Qwen3 30B-A3B, the KV cache grows in step with the context: double the context, double the cache. Gemma 3, Gemma 4 and gpt-oss mix in sliding-window layers that only see a short stretch of recent tokens (as the Gemma 3 technical report, the Gemma 4 model cards and the gpt-oss configuration files describe), so their cache stays small for their size. In Qwen3.6 27B and Qwen3.8 27B, three of every four layers use linear attention (Gated DeltaNet) and keep no growing cache, so per token their KV cache is half of Llama 3.1 8B's and a quarter of Qwen3 32B's (see the table). At 32k with an f16 cache, Llama 3.1 8B needs 4.3 GB for the cache alone, gpt-oss-20b only 0.8 GB.

Qwen3 14B and Qwen3 32B support 32k natively, so their 128k column shows a dash. The context table for Qwen3 14B on a 12GB card shows where the line falls; every cell is calibrated.

At 16k with an f16 KV cache, Qwen3 14B needs 11.9 GB, over the 11.4 GB the card can use, so part of the model moves to system RAM. A q8_0 (8-bit) KV cache brings 16k back onto the GPU, at 10.6 GB, with little margin. At 32k only a q4_0 (4-bit) cache keeps it on the GPU, at 10.9 GB with little margin; in both cases confirm 100% GPU in ollama ps. The 32k f16 and q8_0 cells are partial offload; the table gives each cell's verdict.

Ollama adds its own default. Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). So 8GB to 16GB cards start at 4k and 24GB and 32GB cards at 32k, while CanRun's verdicts use 8k. Ollama quantizes the KV cache only when the server starts with OLLAMA_KV_CACHE_TYPE (f16, q8_0 or q4_0) and flash attention on; the setting applies to every model the server runs (Ollama docs, checked 2026-09-29). The Ollama FAQ puts q8_0 at about half and q4_0 at about a quarter of f16; CanRun's tables use about 54% and 29%. Parallel requests multiply context memory (OLLAMA_NUM_PARALLEL); the tables assume one slot. For the settings, see Ollama slow or forgetting your prompt? Context length defaults and KV cache explained: how context length eats VRAM.

Where to go next

• KV cache explained: how context length eats VRAM: how the KV cache works and what q8_0 and q4_0 save.
• GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download: quant levels, file sizes and quality groups.
• Ollama slow or forgetting your prompt? Context length defaults: Ollama's default context and how to change it.
• Running local LLMs on a Mac (M4, M5): how much unified memory?: the same questions for unified memory.
• Gemma 4 12B vs 26B-A4B vs 31B: differences, VRAM and which size fits your GPU: the three Gemma 4 sizes on 8GB to 32GB cards.
• Qwen3.6 vs Qwen3.8, 27B vs 35B-A3B: differences and the VRAM each needs: Qwen3.6 27B, Qwen3.8 27B and Qwen3.6 35B-A3B compared.
• Card and model pages, such as the GeForce RTX 4060, the GeForce RTX 3090 and Qwen3 14B, list every combination.

How sure are these numbers?

The memory figures are calculated by formula from real GGUF file sizes, each file's header (for the parts llama.cpp keeps off the graphics card), each model's published configuration and the compute buffer. Where a real file size is missing, CanRun falls back to a weight-size formula, which matched eleven real GGUF files with a mean error of 0.7%. CanRun's KV formulas matched the published values on seven architectures.

The speeds are estimates with a band and a label. Measured means a public benchmark for that card, model and quant. Calibrated means ±12% for dense models fully on an NVIDIA or AMD card and ±20% for MoE models, partial offload and other hardware. Theoretical means ±30%, for setups with few measurements or widely varying real speeds, such as MoE experts in system RAM; there the estimate keeps as many experts on the GPU as fit and only the rest in RAM. Against 12 public measurements, the speed formula has a mean error of 4.2% and a maximum of 10.5%.

In the grid, about half the cells are calibrated and several are measured; the cells with MoE experts in system RAM, including the RAM hints for gpt-oss-120b, are theoretical.

Below is one public measurement, for gpt-oss-20b on the GeForce RTX 5060 Ti 16GB, from llama.cpp's gpt-oss guide thread (llama-bench, llama.cpp's benchmark tool, at a 2k context; not a CanRun run), next to CanRun's own 8k estimate band.

All of this assumes Windows with the card driving the display, 32GB of DDR5-5600 dual-channel system RAM, an 8k context, an f16 KV cache, one model loaded, one request slot and a llama.cpp-class runtime. Ollama and llama.cpp make their own estimates. llama.cpp's automatic fitting (--fit, on by default) keeps some VRAM free on each GPU, and Ollama 0.30 and later runs models through llama.cpp, so a model that only just fits here can end up partly in system RAM. On your PC the PROCESSOR column of ollama ps is the ground truth: 100% GPU, or a CPU/GPU split.

Verdict names in the text of this guide refer to the 8k, f16 KV baseline; for other contexts and quants, read the tables. Models with a thinking mode face 1.5× stricter speed thresholds, which the legend does not show: with thinking on, each answer costs many more tokens. That covers models that always think, such as gpt-oss and the DeepSeek R1 distills, and models that can switch thinking on and off, whether it is on by default, as in Qwen3 14B and 32B, Qwen3.6 and Qwen3.8, or off, as in EXAONE 4.0 32B and Gemma 4. Llama 3.1 8B, Gemma 3, Phi-4 and Qwen3 30B-A3B (2507) have no thinking mode. The thresholds:

FAQ

Is 8GB of VRAM enough to run a local LLM?

Yes, for 8B-class models at Q4_K_M with an 8k context. Llama 3.1 8B needs 6.3 GB against the 7.4 GB an 8GB card can use, and it Runs great on the RTX 4060, with about a gigabyte to spare. At Q4_K_M, the 12B to 14B dense models in the catalog do not fit fully: Qwen3 14B needs 10.5 GB, so part of it runs from system RAM and it Runs slowly. Longer context makes it tighter: at 16k with an f16 KV cache, Llama 3.1 8B needs 7.4 GB, about the same as what the card can use; in CanRun's calculation it goes over, so part of it moves to system RAM. A q8_0 (8-bit) KV cache or a shorter context keeps it on the GPU.

Is 16GB of VRAM enough for Gemma 4 and Qwen3.6?

For Gemma 4 12B, and for Gemma 4 26B-A4B at IQ4_XS, yes: at 8k both fit fully on a 16GB card such as the RTX 5060 Ti 16GB and Run great. Qwen3.6 27B and Qwen3.6 35B-A3B keep part of their weights in system RAM there, even at IQ4_XS. At Q4_K_M, Gemma 4 26B-A4B needs 17.7 GB, more than the 15.4 GB such a card can use, so part of its experts sit in system RAM. Gemma 4 31B Runs slowly even at IQ4_XS. Qwen3.6 27B and Qwen3.8 27B Run slowly at Q4_K_M but Run well at IQ4_XS with a small part in system RAM, as calibrated estimates. For Qwen3.6 35B-A3B, which needs 21.8 GB, CanRun's speed with experts in system RAM is a theoretical estimate.

What does 16GB of VRAM run that 12GB does not?

On 16GB, gpt-oss-20b and Gemma 4 26B-A4B at IQ4_XS fit fully on the GPU, while a 12GB card keeps part of their experts in system RAM. 16GB also holds more context: Qwen3 14B at 16k with an f16 KV cache needs 11.9 GB, over the 11.4 GB of a 12GB card but inside the 15.4 GB of a 16GB card. Neither holds Qwen3.6 27B fully at Q4_K_M; for that model, aim for 24GB. Within one size, only speed differs, as the tables in the 12GB and 16GB sections show.

How do I work out how much VRAM a model needs?

Start from the GGUF file size at your quant, add the KV cache for your context and a small compute buffer, then compare the total with the VRAM your card can use: the number on the box minus what the OS keeps for the display. At Q4_K_M a file in gigabytes is about three-fifths of the parameter count in billions, at Q8_0 roughly one gigabyte per billion; the real file size is better still. The weights on the GPU are a little smaller than the file, because llama.cpp keeps the input embedding table in system RAM. For Qwen3 14B, 8.6 GB of the 9.0 GB file goes on the GPU; with 1.3 GB of KV cache at 8k and the buffer, the total is 10.5 GB.

Does a longer context need more VRAM?

Yes. The KV cache stores every token in the context, so for models with full attention on every layer, doubling the context doubles the cache. How much depends on the architecture: at 32k with an f16 cache it is 4.3 GB for Llama 3.1 8B but 0.8 GB for gpt-oss-20b, whose sliding-window layers keep only recent tokens. Mind the runtime default as well: Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). A q8_0 or q4_0 (8-bit or 4-bit) KV cache shrinks it; see KV cache explained: how context length eats VRAM.

Do MoE models like gpt-oss-20b or Qwen3 30B-A3B need less VRAM than dense models of the same size?

Not to fit. The whole file has to be in memory; the small number of active parameters reduces the work per token, not the file size. gpt-oss-20b needs 12.3 GB at 8k, so it fits fully on a 16GB card such as the RTX 5060 Ti 16GB and Runs great there, in line with its model card. What MoE does allow is keeping part of the expert weights in system RAM (llama.cpp's --n-cpu-moe, a split that current llama.cpp and Ollama versions also make on their own); CanRun marks that setup's speed as a theoretical estimate.

Can system RAM make up for too little VRAM?

Partly. Runtimes put whatever does not fit into system RAM, so the model still runs, but slower, and for dense models the drop is steep: Qwen3 14B on an 8GB RTX 4060 Runs slowly. For large MoE models, system RAM itself becomes the limit. The gpt-oss-120b file is 63.4 GB, and with 32GB of RAM it is rated Won't run even on the 32GB RTX 5090; in CanRun's calculation that card needs 64GB of RAM, and an 8GB card needs 96GB. ollama ps or the llama.cpp log shows what your runtime did.

Sources

• llama.cpp: quantize README: bits per weight and file sizes for Q4_K_M, IQ4_XS, Q8_0 and other quants of Llama 3.1 8B, the basis of the file-size rule of thumb.
• llama.cpp: server README: -ngl/--n-gpu-layers, --cpu-moe (all MoE weights on the CPU), --n-cpu-moe N (MoE weights of the first N layers on the CPU), -c/--ctx-size and --cache-type-k (f16 by default, q8_0, q4_0).
• llama.cpp source: layer placement: the input layer, which holds the token embedding table, always stays on the CPU.
• llama.cpp source: Qwen3.5 model loading: the MTP layers are skipped unless MTP is requested, which it is not by default.
• llama.cpp source: EXAONE 4 model loading: the MTP layers stay in the file but are not executed yet, so they are always skipped.
• llama.cpp PR #16653: automatic fitting: --fit, on by default, sets unset options to make the most of GPU memory; for MoE models it keeps dense weights on the GPU first and moves expert weights to the CPU as needed. Merged in December 2025.
• Ollama v0.30.0 release notes: models run with llama.cpp for wider compatibility and performance.
• Ollama docs: context length: the default context by VRAM tier, the advice to avoid offloading to the CPU, and checking with ollama ps.
• Ollama FAQ: the PROCESSOR column of ollama ps, KV cache quantization (q8_0 about half and q4_0 about a quarter of f16, with flash attention), and memory scaling with OLLAMA_NUM_PARALLEL.
• gpt-oss-20b model card: total and active parameters, MoE weights in MXFP4, runs within 16GB of memory.
• gpt-oss-120b model card: total and active parameters, MXFP4, fits a single 80GB GPU.
• gpt-oss-20b configuration and gpt-oss-120b configuration: layer_types alternating sliding-window and full attention, with a short sliding_window.
• Qwen3-30B-A3B-Instruct-2507 model card: total and activated parameters, 128 experts with 8 activated, 48 layers, native context.
• Qwen3-14B model card: parameters, layers, KV heads and native context, with the YaRN extension.
• Qwen3-32B model card: parameters, layers, KV heads and native context.
• Llama 3.1 8B Instruct model card: parameters, 128k context, grouped-query attention.
• Llama 3.3 70B Instruct model card: parameters and 128k context, used for the 70B example in the 32GB section.
• Gemma 3 12B model card: 128k input context (downloading the files requires accepting Google's license; the card text is public).
• Gemma 3 Technical Report: more local than global attention layers with a short local span, to limit KV cache growth at long context.
• llama.cpp: guide to running gpt-oss: llama-bench results on RTX cards, the source of the measured gpt-oss-20b point on the RTX 5060 Ti 16GB.
• Gemma 4 12B, Gemma 4 26B-A4B and Gemma 4 31B model cards: the 26B-A4B is mixture-of-experts with 128 experts (8 active plus 1 shared); local sliding-window layers (1024 tokens) interleaved with global attention; 256k context; thinking off unless enabled in the system prompt.
• Qwen3.6-27B and Qwen3.8-27B model cards: 64 layers in which three Gated DeltaNet (linear attention) layers alternate with one full-attention layer; 256k native context; thinking on by default and can be turned off per request.
• Qwen3.6-35B-A3B model card: 35B total and 3B active parameters, 256 experts with 8 routed and 1 shared active per token, 256k native context, thinking on by default.