Gemma 4 12B vs 26B-A4B vs 31B: differences, VRAM and which size fits your GPU

At Q4_K_M, the usual 4-bit quant, and an 8k context, Gemma 4 12B fits fully on a 12GB card, while 26B-A4B and 31B need 24GB to sit fully on the GPU. On the 16GB RTX 5060 Ti, the MoE 26B-A4B keeps part of its experts in system RAM, and the dense 31B Runs slowly with part of it in RAM. The small E4B that ollama run gemma4 pulls fits fully even on an 8GB card.

Three sizes, two designs

Besides two small edge models, Gemma 4 comes in three sizes for desktop graphics cards, built in two ways. The 12B and the 31B are dense models: every token passes through every weight. The 26B-A4B is a mixture-of-experts (MoE) model. According to its model card and configuration file, each layer holds 128 experts, small blocks of weights, and a router picks 8 of them per token, plus one shared expert. That makes about 26B parameters in total and about 4B active per token, hence the name.

Google released the 26B-A4B and the 31B in April 2026, together with the two edge models, E2B and E4B. The 12B followed in June 2026. Its model card lists it as "12B Unified", a dense model without separate image or audio encoders that takes text, image and audio input. The other two take text and images. All three have a 256k context and the Apache 2.0 license. Their configuration files give 48 layers for the 12B, 30 for the 26B-A4B and 60 for the 31B.

Google's model card says the 26B-A4B runs almost as fast as a 4B-parameter model, and the 12B release post says the 12B is small enough for 16GB of VRAM or unified memory. In CanRun's calculation, the 12B at Q4_K_M fits fully even on a 12GB card. The edge models come up again in the section on Ollama's tags.

How much memory each model needs

At an 8k context with the Q4_K_M file, the 12B needs 7.8 GB, the 26B-A4B 17.7 GB and the 31B 19.5 GB. CanRun counts the weights that go on the graphics card, the KV cache (the memory that holds the context) at f16, and the compute buffer, the runtime's working memory. At 8k the KV cache is small for all three; the section on long context explains why.

Against those needs, an 8GB card such as the RTX 4060 can use 7.4 GB, a 12GB card 11.4 GB, a 16GB card 15.4 GB and a 24GB card 23.4 GB. So the 12B fits fully from 12GB, and the 26B-A4B and the 31B from 24GB.

MoE saves time per token, not memory. The router can pick any expert, so every expert has to be in memory somewhere, and the 26B-A4B needs not much less than the 31B. What MoE changes is where that memory can be.

The 12B on an 8GB card is the close case. Its need is more than the RTX 4060 can use, so it does not fit fully, and in CanRun's calculation part of it runs from system RAM.

What runs on 8GB to 32GB cards

The grid crosses the three models with eight NVIDIA cards of 8, 12, 16, 24 and 32GB. Every verdict assumes an 8k context, an f16 KV cache, 32GB of DDR5-5600 dual-channel system RAM and Windows. Each cell shows an estimated speed band and a confidence label. Gemma 4 gets the stricter thresholds for thinking models, as the section on thinking mode explains.

8GB: RTX 4060

At Q4_K_M, the 12B keeps at least 90% of itself on the GPU and Runs well. The smaller IQ4_XS file needs 7.1 GB, so it fits fully on the GPU with little margin; check ollama ps after loading it. It still Runs well, because its estimate sits just under the stricter Runs great line for thinking models.

The 26B-A4B keeps most of its experts in system RAM on this card and Runs well, a theoretical estimate. Its estimate only just clears the stricter line for Runs well, so a small difference in real speed could move it a grade. The 31B Runs slowly, with much of it in system RAM.

12GB: RTX 3060 12GB, RTX 4070 and RTX 5070

All three 12GB cards hold the 12B fully on the GPU, and it Runs great on each. The 8-bit Q8_0 file needs more than the RTX 3060 12GB can use, so it does not fit fully there and Runs slowly.

The 26B-A4B Runs well on all three cards with part of its experts in system RAM, all theoretical estimates. The 31B Runs slowly on all three, with part of it in system RAM.

16GB: RTX 5060 Ti 16GB

The 12B Runs great on the RTX 5060 Ti 16GB, and even its Q8_0 file fits fully there and Runs well. The 12B Q4_K_M also Runs great on the Radeon RX 9060 XT 16GB, an AMD card outside the grid, and public benchmarks back that result. The two runs below come from llama.cpp's public benchmark threads, one through Vulkan and one through ROCm; CanRun did not run them.

For the 26B-A4B, 16GB is the edge. The Q4_K_M file needs 17.7 GB against the 15.4 GB this card can use, so in CanRun's calculation part of its experts stay in system RAM. It Runs well, a theoretical estimate. The IQ4_XS file sits in the same Good balance quality group as Q4_K_M; for this model it is unsloth's UD-IQ4_XS, at 13.6 GB. It needs 14.3 GB, fits fully on the GPU with about a gigabyte to spare, and Runs great. The ladder shows each quant of the 26B-A4B on this card.

The 31B Runs slowly on this card. Its IQ4_XS file needs 17.6 GB, still more than the card can use, so it still Runs slowly. The UD-Q2_K_XL file fits and Runs well, but it is a 2-bit quant in the Heavy loss group; GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download explains the groups.

To run the 26B-A4B IQ4_XS file on Windows, open PowerShell in your llama.cpp folder:

.\llama-server.exe -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-IQ4_XS -c 8192 -ngl all -fa on

On macOS or Linux, use llama-server instead. With Ollama, the same file is one command:

ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-IQ4_XS

24GB and 32GB: RTX 3090, RTX 4090 and RTX 5090

On the RTX 3090, RTX 4090 and RTX 5090, all three models fit fully on the GPU and Run great.

Public benchmarks back the 26B-A4B cells on the RTX 3090 and the RTX 5090. The RTX 3090 result is an April 2026 user report on Ollama's GitHub: Windows 11, Ollama 0.20.3, a 32k context and 100% GPU in ollama ps, with flash attention turned off to work around an early Gemma 4 slowdown. It used Ollama's own Q4_K_M build, not the unsloth file CanRun calculates.

Q8_0 is where the cards differ. The 26B-A4B Q8_0 file keeps part of its experts in system RAM on the RTX 3090 and Runs well, a theoretical estimate, while on the RTX 5090 it fits fully and Runs great. The 31B Q8_0 file needs more than the RTX 5090 can use, so it Runs well with part of it in system RAM; on the RTX 3090 it Runs slowly.

Why 26B-A4B is faster and can use system RAM

Generation speed follows how much each token has to read from memory. The 26B-A4B reads about 4B parameters for each token, while the 31B reads all of its weights. On the RTX 3090, where both fit, both Run great, and the speed bands in the grid show how far apart they are.

On every card below 24GB in the grid, the 26B-A4B gets the better verdict, although it needs nearly as much memory. It Runs well on all five, as theoretical estimates, while the 31B Runs slowly on all five, as calibrated estimates.

The difference lies in what each model does with the part that does not fit. In CanRun's calculation, the non-expert weights of the 26B-A4B, the KV cache and the compute buffer go on the GPU first. The experts of whole layers then fill the remaining VRAM, with one layer kept free as a margin, and the experts of the other layers stay in system RAM. Each token reads only its chosen experts there. CanRun keeps 4GB of RAM aside for Windows and other programs. The bars show the split on the 8GB RTX 4060.

A dense model has no such shortcut. In partial offload, the 31B reads its layers in system RAM in full for every token, and CanRun caps a dense model with less than 90% of itself on the GPU at Runs slowly. The expert split is a theoretical estimate (±30%), because its real speed depends heavily on the RAM, the OS and the backend.

With 16GB of system RAM, the 26B-A4B still fits on the RTX 4060 in CanRun's calculation and Runs well, a theoretical estimate. The dense 31B does not fit on that card with 16GB of RAM. Runtimes make their own split, so check it with ollama ps.

Since build b7410 (December 2025), llama.cpp's --fit option, on by default, makes this split by itself:

.\llama-server.exe -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M -c 8192

To fix the split yourself, use the N that CanRun's calculator prints for your card: --n-cpu-moe N keeps the expert weights of the first N layers in system RAM. For the RTX 5060 Ti 16GB:

.\llama-server.exe -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M -c 8192 -ngl all --n-cpu-moe 6 -fa on

The RTX 4060 takes --n-cpu-moe 23. On macOS or Linux, use llama-server. Ollama 0.30 and later leaves the split to llama.cpp. For more on this mode, see Running MoE models bigger than your GPU: gpt-oss-120b with system RAM.

Long context: 26B-A4B against 31B

Gemma 4 keeps its KV cache small. According to the configuration files, most layers use a 1024-token sliding window, so their cache stops growing once the context passes that length. Only the full-attention layers keep growing with the context: 8 of 48 in the 12B, 5 of 30 in the 26B-A4B and 10 of 60 in the 31B. The 31B has twice as many full-attention layers as the 26B-A4B, and each has twice as many KV heads. Its KV cache per token is therefore four times as large, as the table shows.

At 128k, the 31B's cache takes 10.7 GB, against 2.7 GB for the 26B-A4B. On a 24GB card, that difference decides how far each model stretches.

For the 31B at Q4_K_M on the RTX 3090, 32k still fits fully on the GPU, with a need of 21.8 GB, and Runs great. At 64k with an f16 cache it needs 24.8 GB, more than the 23.4 GB the card can use. Part of it then runs from system RAM, and it Runs well. A q8_0 (8-bit) KV cache brings 64k back fully onto the GPU, where it Runs great. At 128k with an f16 cache it Runs slowly. A q4_0 cache fits 128k with little margin and Runs great; check ollama ps. At 256k with an f16 cache the 31B does not fit.

The 26B-A4B goes further on the same card. At 128k with an f16 cache it needs 21.3 GB, stays fully on the GPU and Runs great. At 256k with f16, part of its experts move to system RAM, and it Runs well, a theoretical estimate. With a q8_0 cache, 256k fits fully on the GPU and Runs great.

Context also depends on the runtime. Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). A 24GB card therefore starts at 32k, where the 31B is still fully on the GPU in CanRun's calculation. For how the cache grows and how to change its type, see KV cache explained: how context length eats VRAM.

Ollama's gemma4 tags and the files CanRun calculates

On Ollama's library page (checked October 2026), ollama run gemma4 pulls gemma4:latest, which is the same build as gemma4:e4b. That is the E4B edge model, not one of the three sizes above. Google's model card says the E stands for effective parameters: E4B has 4.5B of them and 8B in total, E2B 2.3B and 5.1B. The gap is the per-layer embeddings, a large table that gives every layer its own small embedding for each token and is used only for lookups. llama.cpp counts that table as part of the input layer, which it always keeps on the CPU, so the table stays in system RAM even with every layer on the GPU, and each token reads just its own rows. The bars show the split for E4B on the RTX 4060.

Of the 5.0 GB Q4_K_M file, 3.0 GB goes on the graphics card. In CanRun's calculation the rest, almost all of it the per-layer table, needs room in system RAM instead. With the KV cache and the compute buffer, E4B needs 3.7 GB of VRAM at 8k and E2B 2.1 GB. Their KV cache is small too: according to the configuration files, the last 18 of E4B's 42 layers and the last 20 of E2B's 35 reuse the KV cache of earlier layers.

On the RTX 4060, where the 12B spills over, both edge models fit fully and Run great, as theoretical estimates. E4B stays fully on that card even at its full 128k context with an f16 KV cache, needing 6.9 GB, with little margin; check ollama ps. Its 8-bit Q8_0 file fits fully at 8k too, and 16GB of system RAM leaves enough room for the table. The 8GB laptop chips, the RTX 4060 Laptop and the RTX 4070 Laptop, get the same verdicts in CanRun's calculation, again as theoretical estimates.

On a Mac, CanRun counts the whole file, table included, because the GPU and the CPU share one pool of memory. E4B needs 5.7 GB there, fits fully on a 16GB M4 and Runs well, a theoretical estimate. Its estimate falls under the stricter Runs great line for thinking models, explained below. E2B Runs great on the same Mac, also a theoretical estimate.

A public llama.cpp benchmark through Vulkan backs the RTX 5090 results for both models. Such runs come out slower than memory bandwidth alone predicts, by a fixed delay per token, and CanRun adds that delay to its E2B and E4B estimates. With public benchmarks from only a few cards, it keeps the other E2B and E4B speeds as theoretical estimates (±30%). Ollama's gemma4:e4b and gemma4:e2b are its own builds and larger downloads than the unsloth files CanRun calculates, so check ollama ps after loading one.

For the three sizes here, name the tag: ollama run gemma4:12b, ollama run gemma4:26b or ollama run gemma4:31b. Ollama's page labels the 26B a "Mixture of Experts model with 4B active parameters" and the 31B "Dense". These tags are Ollama's own builds, not the unsloth GGUF files CanRun calculates. A tag can also point to a new build over time. The April 2026 RTX 3090 report above used build 5571076f3d70, which Ollama now lists as gemma4:26b-a4b-it-q4_K_M, while gemma4:26b now points to a different build.

To run the exact files CanRun calculates, pull them from Hugging Face with one of these commands:

ollama run hf.co/unsloth/gemma-4-E2B-it-GGUF:Q4_K_M
ollama run hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M
ollama run hf.co/unsloth/gemma-4-12b-it-GGUF:Q4_K_M
ollama run hf.co/unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M
ollama run hf.co/unsloth/gemma-4-31B-it-GGUF:Q4_K_M

While the model is loaded, the PROCESSOR column of ollama ps shows how it is split between CPU and GPU. Ollama also picks the context length on its own; to set the 8k this guide assumes, see Ollama slow or forgetting your prompt? Context length defaults.

Thinking mode: off in the model, on by default in Ollama

All three sizes have a thinking mode, and so do E2B and E4B. According to the model card, it is on only when the system prompt starts with the <|think|> token, and the chat template sets enable_thinking to false by default. CanRun therefore records Gemma 4 as a model with thinking off by default.

CanRun still holds Gemma 4 to the 1.5× stricter speed thresholds for thinking models, which the legend does not show. The reason is the runtimes. Ollama's local default keeps thinking on for models that can switch it, and its own tests include the case "local gemma default on". llama-server's --reasoning option defaults to auto, which turns thinking on when the chat template supports it.

With thinking off, a Gemma 4 cell can be one grade better than the grid shows. The 12B IQ4_XS file on the RTX 4060 is an example: it is fully on the GPU, yet it Runs well, because its estimate sits just under the stricter Runs great line.

To turn thinking off, run ollama run gemma4:26b --think=false in Ollama, or start llama-server with --reasoning off. According to the model card, the 12B, 26B and 31B still print an empty thought block when thinking is off; E2B and E4B do not. For the speed cost of thinking, see Thinking models on local hardware: the speed cost and how to turn thinking off.

How sure are these numbers?

Memory figures come by formula from the real GGUF file sizes, each model's configuration file, and the sizes in the GGUF headers: the expert and non-expert weights, and for E2B and E4B the per-layer embedding table. Whether a model fits is a calculation, not an estimate.

Speeds are estimates with a band and a label. Measured means a public benchmark for that card, model and quant: here, the 26B-A4B on the RTX 3090 and the RTX 5090, the 12B on the Radeon RX 9060 XT 16GB, and E2B and E4B on the RTX 5090. The RTX 3090 run used Ollama's own build of the Q4_K_M quant. Calibrated means ±12% for dense models fully on an NVIDIA or AMD card, such as the 12B and the 31B, and ±20% for MoE models fully on the GPU and for partial offload. Theoretical means ±30%, used when experts sit in system RAM and for every E2B and E4B cell without a matching benchmark.

All verdicts assume Windows with the card driving the display, 32GB of DDR5-5600 dual-channel system RAM, an 8k context, an f16 KV cache, one model and one request at a time. Ollama and llama.cpp estimate memory their own way, so check ollama ps when a model fits with little margin.

Models with a thinking mode, every Gemma 4 model among them, face 1.5× stricter speed thresholds, which the legend below does not show. The thresholds:

Where to go next

• Running MoE models bigger than your GPU: gpt-oss-120b with system RAM: how experts end up in system RAM, and how much RAM each card needs.
• KV cache explained: how context length eats VRAM: why some models' KV cache grows faster than others.
• Thinking models on local hardware: the speed cost and how to turn thinking off: the speed cost of thinking, and how to turn it off.
• GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download: quant levels and quality groups, including the 2-bit files above.
• How much VRAM do local LLMs need? 8–32 GB tiers (2026): the memory formula and what fits on each VRAM size.
• Model pages for Gemma 4 E2B, Gemma 4 E4B, Gemma 4 12B, Gemma 4 26B-A4B and Gemma 4 31B list every card.
• Card pages such as the GeForce RTX 5060 Ti 16GB and the GeForce RTX 3090 list every model.

FAQ

What is the difference between Gemma 4 26B-A4B and 31B?

The 26B-A4B is an MoE model that uses about 4B of its parameters per token, while the 31B is dense; at Q4_K_M, both need 24GB to sit fully on the GPU. On the 16GB RTX 5060 Ti, the 31B Runs slowly, while the 26B-A4B's IQ4_XS file fits fully and Runs great. On the RTX 3090 both Run great, and a public benchmark backs the 26B-A4B result.

How much VRAM does Gemma 4 need?

At Q4_K_M and an 8k context, aim for 12GB for the 12B and 24GB for the 26B-A4B and the 31B to keep them fully on the GPU, while the edge models E2B and E4B fit fully on 8GB. The three larger sizes need 7.8 GB, 17.7 GB and 19.5 GB. The 12B Runs great on the RTX 3060 12GB, and the larger two fit fully on the RTX 3090. On the 8GB RTX 4060, the 12B keeps at least 90% of itself on the GPU and Runs well. On the 16GB RTX 5060 Ti, the 26B-A4B fits fully as IQ4_XS.

Can Gemma 4 26B-A4B run with only 16GB of system RAM?

In CanRun's calculation, yes, even on the 8GB RTX 4060, by keeping most of its experts in system RAM. The dense 31B does not fit on that card with 16GB of RAM. Speed with experts in system RAM is a theoretical estimate; the grid above shows its band on each card with 32GB of RAM. After loading, check the split with ollama ps.

Which model does ollama run gemma4 download?

The E4B edge model: the default gemma4:latest is the same build as gemma4:e4b (checked October 2026). In CanRun's calculation, E4B at Q4_K_M fits fully even on an 8GB card such as the RTX 4060, with its per-layer embedding table in system RAM. Ollama's build is its own, larger file, so check ollama ps after loading it. For the three larger sizes, use gemma4:12b, gemma4:26b or gemma4:31b, or the hf.co commands above for the exact files CanRun calculates.

Does Gemma 4 think before it answers?

It can, but only when the system prompt starts with <|think|>; otherwise the model keeps thinking off. Ollama and llama-server switch it on by default, so CanRun uses the stricter thresholds for thinking models. To turn it off, add --think=false to ollama run, or start llama-server with --reasoning off.

Sources

• Gemma 4 12B model card: listed as "12B Unified" among the dense models, text, image and audio input, 256K context, Apache 2.0, and thinking switched on with <|think|> at the start of the system prompt.
• Gemma 4 26B-A4B model card: total and active parameters, 8 of 128 experts per token plus 1 shared expert, 30 layers, text and image input, and "almost as fast as a 4B-parameter model".
• Gemma 4 31B model card: a dense model with 60 layers, 256K context and text and image input.
• Gemma 4 E4B model card: what the E in E2B and E4B means, their effective and total parameters, per-layer embedding tables that are "only used for quick lookups", and no empty thought block from E2B and E4B when thinking is off.
• Gemma 4 E2B model card: the smaller edge model, with text, image and audio input and a 128K context.
• Gemma 4 12B configuration, 26B-A4B configuration and 31B configuration: layer counts, the 1024-token sliding window, full-attention layers and their KV heads, and for the 26B-A4B the expert count and experts per token.
• Gemma 4 E4B configuration and E2B configuration: hidden_size_per_layer_input for the per-layer embeddings, and num_kv_shared_layers, the 18 of 42 and 20 of 35 layers that reuse earlier KV caches.
• llama.cpp source: llama-arch.cpp, line 1031 and llama-model.cpp, lines 1684 to 1686: the per-layer embedding table is an input-layer tensor read by row lookup, and llama.cpp "always" keeps the input layer on the CPU.
• Gemma 4 26B-A4B chat template: enable_thinking defaults to false.
• Google: Gemma 4 launch post: the April 2026 launch of E2B, E4B, 26B-A4B and 31B under Apache 2.0, with quantized versions for consumer GPUs.
• Google: introducing Gemma 4 12B: the June 2026 release, no multimodal encoders, and small enough for 16GB of VRAM or unified memory.
• Ollama library: gemma4 and its tags: latest is the same build as e4b, the e2b, e4b, 12b, 26b and 31b tags with their descriptions and download sizes, and the build IDs (checked October 2026).
• Hugging Face: using Ollama with GGUF models on the Hub: the ollama run hf.co/user/repo:quant form.
• unsloth Gemma 4 E2B GGUF, unsloth Gemma 4 E4B GGUF, unsloth Gemma 4 12B GGUF, unsloth Gemma 4 26B-A4B GGUF and unsloth Gemma 4 31B GGUF: the files CanRun calculates.
• llama.cpp: server README: --cpu-moe, --n-cpu-moe N (MoE weights of the first N layers on the CPU), --fit (on by default) and --reasoning (auto by default).
• llama.cpp release b7410: a December 2025 build that includes --fit.
• Ollama v0.30.0 release notes: Ollama runs models with llama.cpp.
• Ollama source: server/model_thinking.go and its tests: the local endpoint keeps thinking on by default, including the test case "local gemma default on".
• Ollama docs: context length: the default context by GPU memory.
• Ollama FAQ: the PROCESSOR column of ollama ps.
• Ollama issue #15237, RTX 3090 report: the measured Gemma 4 26B-A4B result on the RTX 3090, with Ollama 0.20.3 on Windows 11, flash attention off and a 32k context.
• Ollama issue #15353 and llama.cpp PR #23615: the measured Gemma 4 26B-A4B results on the RTX 5090, one through Ollama with flash attention off and one through llama-bench.
• llama.cpp PR #28618: the measured E2B and E4B results on the RTX 5090, through Vulkan with the unsloth Q4_K_M files.
• llama.cpp Vulkan benchmark thread and llama.cpp ROCm benchmark thread: the measured Gemma 4 12B results on the Radeon RX 9060 XT 16GB.