Running MoE models bigger than your GPU: gpt-oss-120b with system RAM

A mixture-of-experts (MoE) model needs memory for its whole file, but each token reads only a small part of it, so the expert weights that do not fit in VRAM can sit in system RAM. In CanRun's calculation, the 63.4 GB gpt-oss-120b file fits on a 12GB RTX 3060 with 64GB of system RAM but not with 32GB. An 8GB RTX 4060 misses 64GB by a hair in CanRun's calculation, so CanRun lists 96GB. The speed then depends mostly on your RAM, and CanRun labels it a theoretical estimate.

How MoE models differ from dense models

In a dense model such as Qwen3 32B, every token passes through every weight, so the speed of generation follows the size of the whole file. In an MoE model, most of each layer is split into many small feed-forward blocks called experts, and a router picks a few of them for each token. According to their configuration files, gpt-oss-120b has 128 experts per layer and uses 4 for each token, while Qwen3.6 35B-A3B has 256 and uses 8 plus one shared expert. The Parameters column below gives both counts: the total, which sets the memory need, and the active part, which sets most of the work per token.

Two models of similar size show what this means in practice. At 8k, Qwen3 32B needs 22.0 GB and Qwen3.6 35B-A3B needs 21.8 GB. Both are more than the 15.4 GB a 16GB RTX 5060 Ti can use. The dense model has to read the layers left in system RAM in full for every token, so on that card it Runs slowly, a calibrated estimate. The MoE model reads only the experts the router picked, and it Runs well, a theoretical estimate.

The router can pick any expert, so every expert has to be in memory somewhere. An MoE model therefore needs as much memory as a dense model with the same total, as How much VRAM do local LLMs need? 8–32 GB tiers (2026) explains. What MoE changes is where that memory can be. Nearly all of an MoE file is expert weights: in the gpt-oss-120b file, the non-expert (shared) weights, such as attention, come to only a few percent. That small part and the KV cache (the memory that holds the context) should stay on the GPU for speed. The experts can sit in system RAM, and only the few that each token uses are read from there.

How the experts end up in system RAM

llama.cpp, the engine behind Ollama 0.30 and later and many other local tools, has two options for this. --cpu-moe keeps all MoE weights in system RAM. --n-cpu-moe N keeps only the expert weights of the first N layers there, so the experts of the other layers stay on the GPU. Since build b7410 (December 2025), llama.cpp also makes this split by default: its --fit option puts the non-expert weights ("dense weights" in the PR) on the GPU first and moves expert weights to system RAM only as far as it has to.

CanRun's calculation follows the same order. It puts the non-expert weights, the KV cache and the compute buffer (the runtime's working memory) on the GPU first. It then fills the remaining VRAM with the experts of whole layers, keeps one layer's worth free as a margin, and leaves the experts of the remaining layers in system RAM. The expert and non-expert sizes come from each GGUF file's header, not from a formula. The N that results is the number our combo pages print in the llama.cpp command.

Qwen3.6 35B-A3B on the RTX 3060 12GB shows the split. It needs 21.8 GB at 8k, against the 11.4 GB the card can use. The GPU bar below is full except for the one layer kept free, and the system RAM bar holds the remaining experts. CanRun keeps 4GB of RAM aside for Windows and other programs, so a 32GB PC counts as 28GB usable.

In this setup it Runs well, a theoretical estimate. The verdict legend further down applies a stricter rule to this mode: Runs well is the best it can get, and it takes the speed that Runs great needs on a full GPU. Qwen3.6 35B-A3B thinks by default, so the stricter thresholds for thinking models apply on top. The Qwen3.6 35B-A3B page compares every card.

A longer context costs speed first

In this mode the KV cache stays on the GPU. A longer context therefore moves more experts to system RAM instead of knocking the model off the card, and the speed drops step by step. For Qwen3.6 35B-A3B on the RTX 3060 12GB, 64k and 128k with an f16 KV cache still Run well, while 256k with an f16 cache falls to Runs slowly, all theoretical estimates. A q8_0 (8-bit) KV cache brings 256k back to Runs well, also a theoretical estimate. According to its configuration file, only one layer in four of Qwen3.6 35B-A3B uses full attention, and the others keep a fixed-size state, so its KV cache grows slowly. Models with full attention on every layer start to slow down at shorter contexts (see KV cache explained: how context length eats VRAM). Once system RAM runs out, the model stops fitting: gpt-oss-120b on the RTX 3060 12GB with 64GB does not fit at 128k with an f16 cache.

How much system RAM each card needs

The grid crosses six MoE models with five NVIDIA cards from 8GB to 32GB. Every verdict assumes an 8k context, an f16 KV cache, 32GB of DDR5 system RAM and Windows. When 32GB is not enough, the line under the badge gives the smallest RAM size that works in CanRun's calculation and the verdict it would get there. CanRun checks 64GB, 96GB and 128GB.

The hint uses the same split as above: the expert layers that do not fit beside the non-expert weights, the KV cache and one free layer go to RAM, and they must fit after the 4GB kept for the system. A 32GB RTX 5090 can use 31.4 GB, so more than half of the 63.4 GB gpt-oss-120b file still has to sit in RAM. That is more than a 32GB PC has left, and 64GB is enough.

gpt-oss-120b

In CanRun's calculation, every card in the grid from 12GB up needs 64GB of system RAM for gpt-oss-120b. The 8GB RTX 4060 misses 64GB by a hair, so CanRun lists 96GB. That gap comes from the margins CanRun keeps, 4GB of RAM for the system and one free layer of VRAM; if you try 64GB, check the split with ollama ps or the llama.cpp log.

With 64GB, gpt-oss-120b Runs slowly on the RTX 3060 12GB, the RTX 5060 Ti 16GB and the RTX 3090, and Runs well on the RTX 5090. These are theoretical estimates, and gpt-oss always thinks, so the stricter thresholds apply. The RTX 3090 estimate sits close to the Runs well line, so a small difference in real speed could put it on either side.

More RAM than the model needs does not make it faster in this calculation. How many expert layers stay on the GPU depends on VRAM, so the RTX 3060 12GB gets the same estimate with 128GB as with 64GB. Our page for gpt-oss-120b on the GeForce RTX 3060 12GB shows the command and the memory split at 64GB.

Qwen3.5 122B-A10B and Solar Open 100B

Qwen3.5 122B-A10B is a little larger and has about twice the active parameters of gpt-oss-120b, so each token reads roughly twice as much. On the 8GB to 16GB cards it needs 96GB of RAM, and on the 24GB and 32GB cards 64GB. With that much RAM it Runs slowly on every card in the grid, even the RTX 5090, as a theoretical estimate.

On the RTX 3090 and RTX 5090, the grid cell suggests a different file. With 32GB of RAM only the UD-Q2_K_XL file fits, and CanRun rates it Runs well on the RTX 3090, a theoretical estimate. It is a 2-bit quant in the Heavy loss group, and CanRun recommends one only when every other quant is rated Won't run. For what the quality groups mean, see GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download.

Solar Open 100B, Upstage's Korean model in the memory table, is not in the grid, but on its five cards it needs 64GB of RAM. That includes the 8GB RTX 4060, where CanRun's calculation leaves only about a gigabyte of RAM to spare. With 64GB it Runs slowly on all five cards, a theoretical estimate.

30B-class MoE models and 16GB of RAM

With the grid's 32GB of RAM, the four smaller models fit on every card in it, down to the 8GB RTX 4060. On the 24GB RTX 3090, Qwen3 30B-A3B and Qwen3.6 35B-A3B fit fully on the GPU and Run great. On the 8GB to 16GB cards they keep part of their experts in RAM, so 16GB of RAM can be too little. With 16GB, neither fits on the RTX 4060, only Qwen3 30B-A3B fits on the RTX 3060 12GB, and both fit on the RTX 5060 Ti 16GB, which holds more experts in VRAM.

Size alone does not set the speed. Gemma 4 26B-A4B is the smaller file, but each token reads more of it than Qwen3.6 35B-A3B (4B active parameters against 3B, and a larger shared part). On the 8GB RTX 3050 it Runs slowly while Qwen3.6 35B-A3B Runs well; both are theoretical estimates under the stricter thresholds for thinking models.

RAM speed sets the pace

In this mode each token reads two things. The chosen experts of the layers in system RAM come from RAM, and everything on the GPU, including its experts and the KV cache, comes from VRAM. RAM is far slower. DDR5-5600 in dual channel moves 90 GB/s, against 360 GB/s on the RTX 3060 12GB. CanRun's formula counts on only half of the RAM figure for this part, a factor fitted to public measurements. The more experts sit in RAM, the more the RAM sets the speed.

Bandwidth is the transfer rate times eight bytes per channel times the number of channels. DDR4-3200 in dual channel gives 51 GB/s, a little over half of DDR5-5600. In CanRun's calculation, moving gpt-oss-120b from DDR5-5600 to DDR4-3200 lowers the estimate by a third to two-fifths on the 12GB to 32GB cards, where most of its experts are in RAM. For Qwen3.6 35B-A3B on the RTX 3060 12GB, where a larger share of each token's work happens on the GPU, it drops by about a fifth. The calculator's RAM type field (DDR4-3200, DDR5-5600 or DDR5-6000) shows the effect on your own setup.

Mainstream desktop processors have two memory channels. Intel's specification page for the Core Ultra 9 285K and AMD's for the Ryzen 7 9700X both list two. More sticks add capacity, not channels, and they can lower the rated speed: AMD lists the Ryzen 7 9700X at DDR5-5600 with two sticks but DDR5-3600 with four, about two-thirds of the bandwidth. When you add RAM for a large MoE model, check what your processor and motherboard support with four sticks before you count on the full speed.

Running it: llama.cpp and Ollama commands

On Windows, llama.cpp ships ready-made builds on its GitHub releases page; for an NVIDIA card, take the CUDA build together with the matching cudart file. Unzip both into one folder and open PowerShell in that folder; on macOS or Linux, run llama-server instead of .\llama-server.exe. With a build from December 2025 or later, the simplest command leaves the split to --fit:

.\llama-server.exe -hf ggml-org/gpt-oss-120b-GGUF:MXFP4 -c 8192

To fix the split yourself, set -ngl and --n-cpu-moe; --fit leaves options you set alone. This is the command our page prints for gpt-oss-120b on the RTX 3060 12GB with 64GB of RAM (the N for other cards is on each combo page):

.\llama-server.exe -hf ggml-org/gpt-oss-120b-GGUF:MXFP4 -c 8192 -ngl all --n-cpu-moe 32 -fa on

-ngl all puts every layer on the GPU, and --n-cpu-moe 32 then sends the experts of the first 32 layers back to system RAM. If loading fails for lack of VRAM, for example after you raise the context, raise N by one or two: each step moves one more layer's experts to RAM.

With Ollama, quit the Ollama app from the taskbar, then start the server with the context you want and run the model from Hugging Face in a second PowerShell window:

$env:OLLAMA_CONTEXT_LENGTH="8192"; ollama serve

ollama run hf.co/ggml-org/gpt-oss-120b-GGUF:MXFP4

On macOS or Linux the first line is OLLAMA_CONTEXT_LENGTH=8192 ollama serve. Ollama decides the split on its own: unless you set num_gpu, it starts llama-server without a GPU layer count, and llama.cpp's --fit places the experts. For how to set the context permanently, see Ollama slow or forgetting your prompt? Context length defaults. In LM Studio, the "Force Model Expert Weights onto CPU" load setting keeps all expert weights in RAM, the same as --cpu-moe.

Check where the model landed

Run ollama ps while the model is loaded: the PROCESSOR column shows the split between CPU and GPU. With llama.cpp, the startup log reports how much of the model went to each device. A model whose experts sit partly in RAM always shows a split; what matters is that the GPU share is as large as your VRAM allows.

On Windows, NVIDIA's driver can borrow shared system memory when VRAM runs out, so with N set too low llama.cpp may run far below the estimate without an error. NVIDIA's support article shows how to turn this off for one program: in the NVIDIA Control Panel, under Manage 3D settings and Program Settings, add llama-server.exe and set "CUDA - Sysmem Fallback Policy" to "Prefer No Sysmem Fallback". A split that is too tight then fails with an out-of-memory error instead.

How sure are these numbers?

Fit is calculated by formula. CanRun reads each MoE file's expert and non-expert sizes from its GGUF header, adds the KV cache and the compute buffer, and compares the parts with the VRAM and system RAM the PC can use. Whether a model fits with a given amount of RAM is therefore a calculation, not an estimate. Results right at a RAM step, such as gpt-oss-120b on an 8GB card with 64GB, depend on the margins CanRun keeps: 4GB of RAM for the system and one free layer of VRAM.

All speeds here are generation speeds. With experts in system RAM they are theoretical estimates (±30%). Against 42 public measurements, the formula's average error was about 11%; on the 35 measured on NVIDIA cards it was about 10%, and 89% of them landed within ±20%. Some setups were off by close to 40%, depending on the OS, the backend and the RAM, so the label stays theoretical. Setups the formula does not describe well were left out of that check: the Vulkan backend, 8-channel server memory and very long contexts of 32k and beyond.

All of this assumes Windows with the card driving the display, DDR5-5600 dual-channel RAM, an 8k context, an f16 KV cache, one model loaded and one request at a time. llama.cpp's automatic split keeps its own margin of free VRAM on each GPU, so the N it picks can differ from CanRun's by a layer or so.

Models with a thinking mode face 1.5× stricter speed thresholds, which the legend below does not show: with thinking on, each answer costs many more tokens. Among the models here, gpt-oss-20b and gpt-oss-120b always think. Qwen3.6 35B-A3B, Qwen3.5 122B-A10B, Solar Open 100B and Qwen3 32B think by default, and Gemma 4 26B-A4B can think but has it off by default. Qwen3 30B-A3B (2507) has no thinking mode. The thresholds:

Where to go next

• How much VRAM do local LLMs need? 8–32 GB tiers (2026): what fits fully on 8GB to 32GB cards, and the memory formula.
• KV cache explained: how context length eats VRAM: why some models' KV cache grows faster than others.
• GGUF quantization: Q4_K_M vs IQ4_XS vs Q8_0 — which to download: quant levels and quality groups, including the 2-bit files above.
• Ollama slow or forgetting your prompt? Context length defaults: Ollama's default context and how to change it.
• Model pages such as gpt-oss-120b and Qwen3 30B-A3B (2507) list every card, with the RAM each one needs.

FAQ

What hardware do I need to run gpt-oss-120b?

In CanRun's calculation at an 8k context, a graphics card with 12GB or more plus 64GB of system RAM. An 8GB card misses 64GB by a hair, so CanRun lists 96GB for it. With 32GB of RAM it does not fit even on a 32GB RTX 5090. The file is 63.4 GB, and its model card targets a single 80GB GPU. Unified-memory machines with 128GB, such as the Ryzen AI Max+ 395 (Strix Halo) and the NVIDIA DGX Spark, hold the whole file in memory, and public measurements put it at Runs great on both.

Can I run gpt-oss-120b without a graphics card?

In CanRun's calculation, yes, with 96GB of RAM. On a CPU with DDR5-5600 dual-channel memory, CanRun rates it Runs slowly, a calibrated estimate; CPU-only setups are capped at that verdict. With 64GB of RAM the file does not fit after the share CanRun keeps for the system. An 8GB card lowers the need, but only to just over what 64GB leaves after the 4GB CanRun keeps for the system; if you try 64GB, check the split with ollama ps or the llama.cpp log. From 12GB of VRAM up, 64GB of RAM is enough.

Should I add RAM or get a card with more VRAM?

Add RAM first if the model does not fit at all; once it fits, more RAM does not raise CanRun's estimate. After that, more VRAM keeps more expert layers on the GPU and faster RAM speeds up the rest; the grid above shows each card. When you plan new RAM, aim for the smallest size at which your model fits, in a stick count your processor supports at full speed.

Why is my MoE model slow even though it fits?

Usually because more of it sits in system RAM than needed, or the RAM is slower than you think. Check the split with ollama ps or the llama.cpp log; if Ollama's GPU share is far below your VRAM, look for a leftover num_gpu setting or try llama.cpp with --n-cpu-moe. --cpu-moe puts every expert in RAM even when some would fit, four sticks of RAM may run at a lower rated speed than two, and a long context moves more experts to RAM. On Windows, a split that is too tight can spill into shared GPU memory, as the commands section explains.

Should I use --cpu-moe or --n-cpu-moe?

Use --n-cpu-moe N when you set the split yourself, because it keeps as many expert layers on the GPU as fit. --cpu-moe is the same as putting every layer's experts in RAM, which is slower whenever the card has room for some of them. With a llama.cpp build from December 2025 or later you can leave both out and let --fit choose, but keep -c: without it, --fit may shorten the context before it moves any weights. Our combo pages print the N CanRun calculated for each card.

Sources

• llama.cpp: server README: --cpu-moe (all MoE weights on the CPU), --n-cpu-moe N (MoE weights of the first N layers on the CPU), -ngl (auto or all), --fit (on by default), --fit-target (free memory kept per device) and --fit-ctx (the shortest context --fit may set, 4096 by default), -fa, -c.
• llama.cpp PR #16653: automatic fitting: --fit adjusts options the user did not set to fit device memory, keeps options the user set, lowers an unset context before it moves weights, and for MoE models places dense weights in VRAM first. Merged in December 2025.
• llama.cpp release b7410: a December 2025 build that includes that change, with Windows CUDA builds and cudart files among its downloads.
• Ollama v0.30.0 release notes: Ollama runs models with llama.cpp.
• Ollama source: llm/llama_server.go: Ollama starts llama-server without -ngl unless num_gpu is set, leaving GPU placement to llama.cpp.
• Ollama FAQ: the PROCESSOR column of ollama ps.
• Hugging Face: using Ollama with GGUF models on the Hub: the ollama run hf.co/user/repo:quant form.
• LM Studio 0.3.23 release notes: Force Model Expert Weights onto CPU, which keeps all MoE expert weights in system RAM.
• gpt-oss-120b model card and configuration: total and active parameters, MXFP4 MoE weights, fits a single 80GB GPU; 36 layers, 128 experts, 4 per token.
• Qwen3.6-35B-A3B configuration: 40 layers, 256 experts, 8 per token, a shared expert, and full attention on one layer in four (full_attention_interval).
• Qwen3.5-122B-A10B configuration: 48 layers, 256 experts, 8 per token, a shared expert.
• Qwen3-30B-A3B-Instruct-2507 configuration: 48 layers, 128 experts, 8 per token.
• Gemma 4 26B-A4B configuration: 30 layers, 128 experts, 8 per token.
• Solar Open 100B configuration: 48 layers, 128 routed experts, 8 per token, one shared expert.
• AMD Ryzen 7 9700X specifications: two memory channels; DDR5-5600 with two sticks, DDR5-3600 with four.
• Intel Core Ultra 9 285K specifications: two memory channels.
• NVIDIA support: CUDA Sysmem Fallback Policy: on Windows the driver can use shared system memory when VRAM runs out, so programs keep running at lower speed, and the "Prefer No Sysmem Fallback" setting turns that off per program.
• llama.cpp DGX Spark results file, February 2026 and quozul.dev: gpt-oss-120b on the Ryzen AI Max+ 395: the measured gpt-oss-120b results on 128GB unified-memory machines.