Which local LLMs can the GeForce RTX 4060 run?

Assumes 32 GB of DDR5-5600 system RAM, Windows with this GPU driving the display, 8k context and an f16 KV cache.

What to expect from the GeForce RTX 4060

The RTX 4060 has 8GB of VRAM, and with Windows drawing the desktop on the same card, CanRun counts about 7.4 GB of it as free for a model. That is enough to hold Llama 3.1 8B, Qwen3 8B and Qwen3.5 9B at Q4_K_M with an 8k context, and all three run great here. Long context is where 8GB runs out first: Qwen3 8B at 32k needs 10.3 GB with an f16 KV cache.

Dense 12B–14B models sit right at the edge. Gemma 4 12B at Q4_K_M needs 7.8 GB, so part of it goes to system RAM, while its IQ4_XS file stays fully on the GPU. Qwen3 14B spills further and runs slowly. Mixture-of-experts models are the better way to go bigger: with their experts in system RAM, CanRun expects gpt-oss-20b and Qwen3 30B-A3B to run well. Those two are theoretical estimates (±30%) and lean on your system RAM speed.

The GeForce RTX 4060 Laptop has the same 8GB, so the same models fit, but slightly less memory bandwidth: 256 GB/s against 272 GB/s on this card. Its speed estimates come out a little lower, but gpt-oss-20b still runs well on it with part of its experts in system RAM (a theoretical estimate, ±30%). If 8GB keeps getting in the way, more VRAM changes which models fit; a faster 8GB card does not.

Specs

VRAM: 8 GB
Memory bandwidth: 272.0 GB/s
FP16 compute: 15.1 TFLOPS
Launch year: 2023
Price: $299 launch MSRP
Also written as: 4060, RTX4060

Verdicts at a glance

At Q4_K_M (or the closest available quant) with 8k context.

Runs great19 modelsRuns well11 modelsRuns slowly13 modelsWon't run16 models7 more models run at a lower quant.

How to read the verdicts

Runs great: Fully on the GPU at 20 tok/s or more
Runs well: Fully on the GPU at 8–20 tok/s; MoE experts in system RAM at 20 tok/s or more; or at least 90% on the GPU at 8 tok/s or more
Runs slowly: 2–8 tok/s; CPU-only; less than 90% on the GPU; or MoE experts in system RAM below 20 tok/s
Won't run: Does not fit, or under 2 tok/s

Every model on the GeForce RTX 4060

Dense models split between GPU and CPU slow down sharply — the CPU side sets the pace. MoE models that keep only their experts in system RAM degrade far more gently.

Scroll sideways to see every column.

59 models sorted by verdict and speed
Model	Verdict	Speed	Quant	Memory	Runs as	Context	Notes
MiniCPM5 1B	Runs great	est. 213.6 tok/s188.0–239.2calibrated estimate ±12%	Q4_K_M	1.9 / 8.0 GB	Full GPU	up to 128k	thinking mode
Ling-3.0 Tiny	Runs great	est. 176.6 tok/s123.6–229.6theoretical estimate ±30%	Q4_K_M	5.9 / 8.0 GB	Full GPU	up to 128k	thinking mode
Llama 3.2 1B	Runs great	est. 176.6 tok/s155.4–197.7calibrated estimate ±12%	Q4_K_M	2.2 / 8.0 GB	Full GPU	up to 128k	—
EXAONE 4.0 1.2B	Runs great	est. 145.0 tok/s127.6–162.4calibrated estimate ±12%	Q4_K_M	2.5 / 8.0 GB	Full GPU	up to 64k	thinking mode
LFM2.5 2.6B	Runs great	est. 105.5 tok/s73.9–137.2theoretical estimate ±30%	Q4_K_M	3.0 / 8.0 GB	Full GPU	up to 128k	thinking mode
Gemma 4 E2B	Runs great	est. 105.3 tok/s73.7–137.0theoretical estimate ±30%	Q4_K_M	2.7 / 8.0 GB	Full GPU	up to 128k	thinking mode
HyperCLOVA X SEED 1.5B	Runs great	est. 104.9 tok/s92.3–117.5calibrated estimate ±12%	Q4_K_M	3.0 / 8.0 GB	Full GPU	up to 16k	—
MiniCPM5 2B	Runs great	est. 99.6 tok/s87.6–111.5calibrated estimate ±12%	Q4_K_M	2.9 / 8.0 GB	Full GPU	up to 64k	thinking mode
Spark-X2.5 4B	Runs great	est. 65.6 tok/s57.7–73.5calibrated estimate ±12%	Q4_K_M	4.1 / 8.0 GB	Full GPU	up to 64k	thinking mode
Llama 3.2 3B	Runs great	est. 64.3 tok/s56.6–72.1calibrated estimate ±12%	Q4_K_M	4.1 / 8.0 GB	Full GPU	up to 32k	—
Qwen3.5 4B	Runs great	est. 61.5 tok/s54.1–68.8calibrated estimate ±12%	Q4_K_M	4.2 / 8.0 GB	Full GPU	up to 64k	thinking mode
Gemma 4 E4B	Runs great	est. 55.5 tok/s38.8–72.1theoretical estimate ±30%	Q4_K_M	4.3 / 8.0 GB	Full GPU	up to 128k	thinking mode
Llama 3.1 8B	Runs greatLlama 3.1 8B verdict on the GeForce RTX 4060	38.1 tok/s8k estimate 31.8 tok/s (28.0–35.6, calibrated estimate ±12%)measured (1 run, 4k context)	Q4_K_M	6.9 / 8.0 GB	Full GPU	up to 8k	—
DeepSeek R1 Distill Llama 8B	Runs great	est. 31.8 tok/s28.0–35.6calibrated estimate ±12%	Q4_K_M	6.9 / 8.0 GB	Full GPU	up to 8k	thinking mode
Kanana 1.5 8B	Runs great	est. 31.8 tok/s28.0–35.6calibrated estimate ±12%	Q4_K_M	6.9 / 8.0 GB	Full GPU	up to 8k	—
Nanbeige4.2 3B	Runs great	est. 31.1 tok/s21.8–40.4theoretical estimate ±30%	Q4_K_M	5.0 / 8.0 GB	Full GPU	up to 8k	thinking mode
Qwen3.5 9B	Runs greatQwen3.5 9B verdict on the GeForce RTX 4060	est. 31.0 tok/s27.3–34.7calibrated estimate ±12%	Q4_K_M	6.6 / 8.0 GB	Full GPU	up to 8k	thinking mode
DeepSeek R1 0528 Qwen3 8B	Runs great	est. 30.5 tok/s26.9–34.2calibrated estimate ±12%	Q4_K_M	7.1 / 8.0 GB	Full GPU	up to 8k	thinking mode
Qwen3 8B	Runs greatQwen3 8B verdict on the GeForce RTX 4060	est. 30.5 tok/s26.9–34.2calibrated estimate ±12%	Q4_K_M	7.1 / 8.0 GB	Full GPU	up to 8k	thinking mode
Mellum2.1 12B-A2.5B Thinking	Runs well	est. 74.5 tok/s52.2–96.9theoretical estimate ±30%	Q4_K_M	7.6 / 8.0 GB + 1.7 GB RAM	MoE experts in RAM	up to 128k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
MiniCPM-V 4.7 35B-A3B	Runs well	est. 45.9 tok/s32.1–59.6theoretical estimate ±30%	Q4_K_M	7.1 / 8.0 GB + 15.4 GB RAM	MoE experts in RAM	up to 64k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
Qwen3.6 35B-A3B	Runs wellQwen3.6 35B-A3B verdict on the GeForce RTX 4060	est. 43.6 tok/s30.5–56.6theoretical estimate ±30%	Q4_K_M	7.3 / 8.0 GB + 16.3 GB RAM	MoE experts in RAM	up to 64k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
Qwen3.5 35B-A3B	Runs wellQwen3.5 35B-A3B verdict on the GeForce RTX 4060	est. 39.3 tok/s27.5–51.1theoretical estimate ±30%	Q4_K_M	7.3 / 8.0 GB + 16.7 GB RAM	MoE experts in RAM	up to 64k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
gpt-oss-20b	Runs wellgpt-oss-20b verdict on the GeForce RTX 4060	est. 36.9 tok/s25.9–48.0theoretical estimate ±30%	MXFP4	7.3 / 8.0 GB + 6.1 GB RAM	MoE experts in RAM	up to 32k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
Kanana 1.5 15.7B-A3B	Runs well	est. 35.5 tok/s24.9–46.2theoretical estimate ±30%	Q4_K_M	7.4 / 8.0 GB + 5.3 GB RAM	MoE experts in RAM	up to 16k	Not fully on the GPU — Runs great needs the whole model in GPU memory
Nemotron 3.5 Lightning 30B-A3B	Runs well	est. 32.6 tok/s22.8–42.3theoretical estimate ±30%	Q4_K_M	6.8 / 8.0 GB + 19.9 GB RAM	MoE experts in RAM	up to 32k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
Qwen3 30B-A3B (2507)	Runs wellQwen3 30B-A3B (2507) verdict on the GeForce RTX 4060	est. 32.4 tok/s22.6–42.1theoretical estimate ±30%	Q4_K_M	7.6 / 8.0 GB + 13.0 GB RAM	MoE experts in RAM	up to 32k	Not fully on the GPU — Runs great needs the whole model in GPU memory
Gemma 4 26B-A4B	Runs wellGemma 4 26B-A4B verdict on the GeForce RTX 4060	est. 30.6 tok/s21.4–39.7theoretical estimate ±30%	Q4_K_M	7.3 / 8.0 GB + 11.0 GB RAM	MoE experts in RAM	up to 8k	Not fully on the GPU — Runs great needs the whole model in GPU memory thinking mode
K2-Horizon 7B	Runs well	est. 28.0 tok/s24.6–31.4calibrated estimate ±12%	Q4_K_M	7.4 / 8.0 GB	Full GPU	up to 8k	thinking mode
Gemma 4 12B	Runs wellGemma 4 12B verdict on the GeForce RTX 4060	est. 21.9 tok/s17.6–26.3calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 0.4 GB RAM	Partial offload	up to 16k	UD-Q2_K_XL (heavy quality loss): Runs great thinking mode
Gemma 3 12B	Runs slowlyGemma 3 12B verdict on the GeForce RTX 4060	est. 16.7 tok/s13.4–20.1calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 1.0 GB RAM	Partial offload	up to 64k	Less than 90% on the GPU — a GPU/CPU split is capped at Runs slowly Try IQ4_XS: Runs well Q2_K (heavy quality loss): Runs great
K2-Horizon MoVA 36B-A4B	Runs slowly	est. 9.6 tok/s6.7–12.5theoretical estimate ±30%	Q4_K_M	24.0 GB RAM	CPU only	up to 16k	thinking mode
HyperCLOVA X SEED Think 14B	Runs slowly	est. 8.4 tok/s5.9–10.9theoretical estimate ±30%	Q4_K_M	8.0 / 8.0 GB + 3.4 GB RAM	Partial offload	up to 32k	thinking mode
Qwen3 14B	Runs slowlyQwen3 14B verdict on the GeForce RTX 4060	est. 8.1 tok/s6.5–9.7calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 3.5 GB RAM	Partial offload	up to 32k	Q2_K (heavy quality loss): Runs well thinking mode
DeepSeek R1 Distill Qwen 14B	Runs slowly	est. 7.6 tok/s6.1–9.1calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 3.8 GB RAM	Partial offload	up to 16k	Q2_K (heavy quality loss): Runs well thinking mode
Phi-4	Runs slowly	est. 7.4 tok/s5.9–8.9calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 3.9 GB RAM	Partial offload	up to 16k	—
Mistral Small 3.2 24B	Runs slowly	est. 4.0 tok/s3.2–4.8calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 8.8 GB RAM	Partial offload	up to 32k	—
Muse Glimmer 30B	Runs slowly	est. 3.7 tok/s2.9–4.4calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 10.6 GB RAM	Partial offload	up to 64k	thinking mode
Gemma 3 27B	Runs slowly	est. 3.6 tok/s2.9–4.4calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 10.4 GB RAM	Partial offload	up to 64k	—
Qwen3.6 27B	Runs slowly	est. 3.6 tok/s2.9–4.3calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 10.5 GB RAM	Partial offload	up to 16k	thinking mode
Qwen3.5 27B	Runs slowlyQwen3.5 27B verdict on the GeForce RTX 4060	est. 3.5 tok/s2.8–4.2calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 10.8 GB RAM	Partial offload	up to 16k	thinking mode
Qwen3.8 27B	Runs slowlyQwen3.8 27B verdict on the GeForce RTX 4060	est. 3.4 tok/s2.8–4.1calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 11.2 GB RAM	Partial offload	up to 16k	thinking mode
Gemma 4 31B	Runs slowlyGemma 4 31B verdict on the GeForce RTX 4060	est. 3.2 tok/s2.5–3.8calibrated estimate ±20%	Q4_K_M	8.0 / 8.0 GB + 12.2 GB RAM	Partial offload	up to 8k	thinking mode
Kolibri 1	Won't runTry Q2_K (heavy quality loss): Runs well	est. 43.5 tok/s30.5–56.6theoretical estimate ±30%	Q2_K	7.3 / 8.0 GB + 23.9 GB RAM	MoE experts in RAM	up to 64k	Not fully on the GPU — Runs great needs the whole model in GPU memory With 64 GB RAM: Runs well thinking mode
Granite 4.2 30B	Won't runTry Q2_K (heavy quality loss): Runs slowly	est. 5.1 tok/s4.1–6.2calibrated estimate ±20%	Q2_K	8.0 / 8.0 GB + 6.2 GB RAM	Partial offload	up to 16k	thinking mode
DeepSeek R1 Distill Qwen 32B	Won't runTry Q2_K (heavy quality loss): Runs slowly	est. 4.4 tok/s3.5–5.2calibrated estimate ±20%	Q2_K	8.0 / 8.0 GB + 7.6 GB RAM	Partial offload	up to 16k	thinking mode
Qwen3 32B	Won't runTry Q2_K (heavy quality loss): Runs slowlyQwen3 32B verdict on the GeForce RTX 4060	est. 4.4 tok/s3.5–5.2calibrated estimate ±20%	Q2_K	8.0 / 8.0 GB + 7.7 GB RAM	Partial offload	up to 16k	thinking mode
Nanbeige4.3 33B	Won't runTry Q2_K (heavy quality loss): Runs slowly	est. 4.1 tok/s3.3–4.9calibrated estimate ±20%	Q2_K	8.0 / 8.0 GB + 8.4 GB RAM	Partial offload	up to 16k	thinking mode
EXAONE 4.0 32B	Won't runTry IQ4_XS: Runs slowly	est. 3.5 tok/s2.8–4.1calibrated estimate ±20%	IQ4_XS	8.0 / 8.0 GB + 11.1 GB RAM	Partial offload	up to 16k	thinking mode
EXAONE 4.5 33B	Won't runTry IQ4_XS: Runs slowly	est. 3.3 tok/s2.6–3.9calibrated estimate ±20%	IQ4_XS	8.0 / 8.0 GB + 11.7 GB RAM	Partial offload	up to 16k	thinking mode
DeepSeek V4 Flash 0731	Won't run	—	MXFP4	needs 155.1 GB	—	—	Needs about 155.1 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free thinking mode
GLM-5.3 Flash	Won't run	—	Q4_K_M	needs 181.4 GB	—	—	Needs about 181.4 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free thinking mode
gpt-oss-120b	Won't rungpt-oss-120b verdict on the GeForce RTX 4060	—	MXFP4	needs 63.6 GB	—	—	Needs about 63.6 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free With 96 GB RAM: Runs slowly thinking mode
Laguna S 2.1	Won't run	—	Q4_K_M	needs 68.5 GB	—	—	Needs about 68.5 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free With 96 GB RAM: Runs slowly thinking mode
Ling-3.0 Flash	Won't run	—	Q4_K_M	needs 75.4 GB	—	—	Needs about 75.4 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free With 96 GB RAM: Runs slowly thinking mode
Llama 3.3 70B	Won't runLlama 3.3 70B verdict on the GeForce RTX 4060	—	Q4_K_M	needs 45.2 GB	—	—	Needs about 45.2 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free
Qwen3.5 122B-A10B	Won't runQwen3.5 122B-A10B verdict on the GeForce RTX 4060	—	Q4_K_M	needs 76.6 GB	—	—	Needs about 76.6 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free With 96 GB RAM: Runs slowly thinking mode
Solar Open 100B	Won't run	—	Q4_K_M	needs 64.0 GB	—	—	Needs about 64.0 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free With 64 GB RAM: Runs slowly thinking mode
Solar Open 2 250B	Won't run	—	IQ4_XS	needs 136.5 GB	—	—	Needs about 136.5 GB; 7.4 GB of VRAM and 28.0 GB of RAM are free thinking mode

Models with a thinking mode spend extra tokens thinking, so their speed thresholds are 1.5× stricter (30 / 12 / 3 tok/s). With thinking turned off, a model may do one grade better.

Measured results on the GeForce RTX 4060

Public benchmarks we calibrate against. Their conditions (context, backend, flags) can differ from the estimates above.

Measured results on the GeForce RTX 4060
Model	Quant	Backend	Context	Prompt (tok/s)	Generation (tok/s)	Flags	Source	Measured
Llama 3.1 8B	Q4_K_M	llama.cpp	4k	1,528.0	38.1	ctx-weighted	localscore.ai	2026-08-09

Frequently asked questions

What is the largest model that runs entirely on the GeForce RTX 4060?

Gemma 4 12B at IQ4_XS (a 6.38 GB file) fits entirely in 8 GB with 8k context, at est. 29.2 tok/s (25.7–32.7, calibrated estimate ±12%).

How many local LLMs run well on the GeForce RTX 4060?

At Q4_K_M with 8k context, out of 59 tracked models: 19 run great, 11 run well, 13 run slowly and 16 won't run.

Can the GeForce RTX 4060 run a 70B model like Llama 3.3 70B?

No. Llama 3.3 70B at Q4_K_M needs about 45.2 GB, while this setup offers 7.4 GB of GPU memory and 28.0 GB of free system RAM.

Speeds are estimates from memory bandwidth, calibrated against public benchmarks, and each one comes with an error band and a confidence label. Real results vary with drivers, backend, context length and thermals.