Can I run Qwen3.5 122B-A10B on the GeForce RTX 4060?
Won't runQ4_K_M needs 76.6 GB, 69.2 GB more than the 7.4 GB of usable memory. With 96 GB of system RAM: Runs slowly, est. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%).
8 GB VRAM · 272.0 GB/s · FP16 15.1 TFLOPS. Assumes 32 GB of DDR5-5600 system RAM, Windows with this GPU driving the display, 8k context and an f16 KV cache.
Verdict by system RAM
This page assumes 32 GB of system RAM. For this pair, the verdict depends on how much system RAM you have.
- 16 GBWon't runQ4_K_M doesn't fit
- 32 GBWon't runQ4_K_M doesn't fit(used on this page)
- 64 GBWon't runTry IQ4_XS: Runs slowlyest. 13.9 tok/s (9.8–18.1, theoretical estimate ±30%)
- 96 GBRuns slowlyest. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%)(used for the commands below)
- 128 GBRuns slowlyest. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%)
Every quant of Qwen3.5 122B-A10B on the GeForce RTX 4060
| Quant | File size | Verdict | Speed | Memory | Runs as | Notes |
|---|---|---|---|---|---|---|
| UD-Q2_K_XL | 42.85 GB | Won't run | — | Needs about 42.1 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free | — | — |
| IQ4_XS | 61.93 GB | Won't run | — | Needs about 60.6 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free | — | — |
| Q4_K_Mbaseline | 78.26 GB | Won't run | — | Needs about 76.6 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free | — | — |
| Q8_0 | 132.55 GB | Won't run | — | Needs about 129.8 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free | — | — |
Models with a thinking mode spend extra tokens thinking, so their speed thresholds are 1.5× stricter (30 / 12 / 3 tok/s). With thinking turned off, a model may do one grade better.
Where the memory goes at Q4_K_M
The memory, context and speed figures below assume 96 GB of system RAM (the configuration from the RAM hint above); with 32 GB nothing fits.
- Weights
- 5.7 GB
- KV cache
- 0.2 GB
- Compute buffer
- 0.6 GB
- OS reserve
- 0.6 GB
- Free
- 0.9 GB
- Weights in system RAM
- 72.6 GB
The system RAM share above also counts the 1.57 GB MTP (multi-token prediction) block, which llama.cpp does not load by default, so it errs on the safe side. Turning on MTP speculative decoding (llama.cpp --spec-type draft-mtp) loads that block and needs KV cache for it too.
System RAM, context length and KV cache
| Context | KV f16 | KV q8_0 | KV q4_0 |
|---|---|---|---|
| 4k | 76.5 GBRuns slowly est. 12.1 tok/s8.5–15.8theoretical estimate ±30% | 76.5 GBRuns slowly est. 12.2 tok/s8.5–15.8theoretical estimate ±30% | 76.4 GBRuns slowly est. 12.2 tok/s8.5–15.9theoretical estimate ±30% |
| 8k | 76.6 GBRuns slowly est. 12.1 tok/s8.4–15.7theoretical estimate ±30% | 76.6 GBRuns slowly est. 12.1 tok/s8.5–15.8theoretical estimate ±30% | 76.5 GBRuns slowly est. 12.2 tok/s8.5–15.8theoretical estimate ±30% |
| 16k | 76.9 GBRuns slowly est. 11.9 tok/s8.3–15.5theoretical estimate ±30% | 76.7 GBRuns slowly est. 12.1 tok/s8.4–15.7theoretical estimate ±30% | 76.6 GBRuns slowly est. 12.1 tok/s8.5–15.8theoretical estimate ±30% |
| 32k | 77.5 GBRuns slowly est. 11.6 tok/s8.1–15.1theoretical estimate ±30% | 77.1 GBRuns slowly est. 11.9 tok/s8.3–15.5theoretical estimate ±30% | 76.9 GBRuns slowly est. 12.0 tok/s8.4–15.7theoretical estimate ±30% |
| 64k | 79.9 GBRuns slowly est. 4.9 tok/s3.9–5.9calibrated estimate ±20% | 79.1 GBRuns slowly est. 5.4 tok/s4.3–6.5calibrated estimate ±20% | 77.4 GBRuns slowly est. 11.9 tok/s8.3–15.4theoretical estimate ±30% |
| 128k | 81.5 GBRuns slowly est. 4.1 tok/s3.3–4.9calibrated estimate ±20% | 80.0 GBRuns slowly est. 4.8 tok/s3.9–5.8calibrated estimate ±20% | 79.2 GBRuns slowly est. 5.4 tok/s4.3–6.4calibrated estimate ±20% |
| 256k | 84.7 GBRuns slowly est. 3.0 tok/s2.4–3.6calibrated estimate ±20% | 81.7 GBRuns slowly est. 4.0 tok/s3.2–4.8calibrated estimate ±20% | 80.1 GBRuns slowly est. 4.8 tok/s3.8–5.7calibrated estimate ±20% |
S = Runs great · A = Runs well · B = Runs slowly · F = Won't run
With 96 GB of system RAM: At Q4_K_M the verdict stays Runs slowly up to 256k context with an f16 KV cache, and up to 256k with q8_0.
Estimated speed
- Generation
- With 96 GB of system RAM: est. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%)
- Prompt processing
- Prompt-processing estimates only apply when the whole model runs on the GPU.
Why this verdict: Experts run from system RAM — Runs well needs 30 tok/s or more on this path
Measured on this exact combination
No public measurement for this exact combination yet; the numbers above are estimates.
If this is not enough
A smaller model that runs well on the GeForce RTX 4060
Runs wellQwen3.5 35B-A3B: Runs well, est. 39.3 tok/s (27.5–51.1, theoretical estimate ±30%)
Same GPU with 96 GB of system RAM
Runs slowlyQ4_K_M: Runs slowly, est. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%)
How to run it
Commands for the UD-Q4_K_M file. Both tools download from Hugging Face on first run.
These commands assume 96 GB of system RAM (the configuration from the RAM hint above). For a different amount of RAM, pick it under “System RAM, context length and KV cache” above to see the commands for it.
File: UD-Q4_K_M/Qwen3.5-122B-A10B-UD-Q4_K_M-00001-of-00003.gguf and 2 more shards (78.26 GB total) from unsloth/Qwen3.5-122B-A10B-MTP-GGUF.
llama-server -hf unsloth/Qwen3.5-122B-A10B-MTP-GGUF:UD-Q4_K_M -c 8192 -ngl all --n-cpu-moe 48 -fa on- -ngl all loads every layer on the GPU.
- --n-cpu-moe 48 keeps the experts of all 48 layers in system RAM, which is what this estimate assumes.
- llama.cpp b7410 (December 2025) and later split the model in a similar way on their own if you leave out -ngl and --n-cpu-moe.
- -fa on enables flash attention (needed for KV cache quantization).
- --cache-type-k q8_0 --cache-type-v q8_0 shrinks the KV cache to 0.11 GB at 8k context.
Ollama cannot pull multi-file GGUFs yet, so use llama.cpp for this file.
Related pages
Guide: How much VRAM do local LLMs need? GPU requirements for 8–32 GB (2026)
Other models on the GeForce RTX 4060
Qwen3.5 122B-A10B on other hardware
Frequently asked questions
Can I run Qwen3.5 122B-A10B on the GeForce RTX 4060?
Q4_K_M needs 76.6 GB, 69.2 GB more than the 7.4 GB of usable memory. With 96 GB of system RAM: Runs slowly, est. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%). Q4_K_M needs 76.6 GB at 8k context; this setup has 7.4 GB of usable memory and 28.0 GB of free system RAM.
How much context can Qwen3.5 122B-A10B use on the GeForce RTX 4060?
With 96 GB of system RAM: At Q4_K_M the verdict stays Runs slowly up to 256k context with an f16 KV cache (6.4 GB of KV) and up to 256k with a q8_0 KV cache (3.4 GB). The model supports up to 256k.
Which quant should I use, and how fast is it?
No tracked quant runs with 32 GB of RAM; with 96 GB, Q4_K_M reaches Runs slowly, est. 12.1 tok/s (8.4–15.7, theoretical estimate ±30%).
Speeds are estimates from memory bandwidth, calibrated against public benchmarks, and each one comes with an error band and a confidence label. Real results vary with drivers, backend, context length and thermals.