Can I run Llama 3.3 70B on the Apple M4?
Won't runQ4_K_M needs 45.8 GB, 23.4 GB more than the 22.4 GB of usable memory. IQ2_M (heavy quality loss) works instead: Runs slowly, est. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%).
32 GB unified memory · 120.0 GB/s · FP16 3.8 TFLOPS. Assumes 8k context and an f16 KV cache. Verdicts are shown for each memory size.
Verdict by memory size
The page uses the 32 GB configuration; here is every size.
- 16 GBWon't runQ4_K_M doesn't fit
- 24 GBWon't runQ4_K_M doesn't fit
- 32 GBWon't runTry IQ2_M (heavy quality loss): Runs slowlyest. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%)(used on this page)
Every quant of Llama 3.3 70B on the Apple M4
| Quant | File size | Verdict | Speed | Memory | Runs as | Notes |
|---|---|---|---|---|---|---|
| IQ2_M | 24.12 GB | Runs slowly | est. 2.2 tok/s1.6–2.9theoretical estimate ±30% | 26.8 GB RAM | CPU only | — |
| Q2_K | 26.38 GB | Won't run | — | Needs about 29.6 GB; 28.0 GB of memory is free (22.4 GB usable by the GPU) | — | — |
| IQ4_XS | 37.90 GB | Won't run | — | Needs about 41.2 GB; 28.0 GB of memory is free (22.4 GB usable by the GPU) | — | — |
| Q4_K_Mbaseline | 42.52 GB | Won't run | — | Needs about 45.8 GB; 28.0 GB of memory is free (22.4 GB usable by the GPU) | — | — |
| Q8_0 | 74.98 GB | Won't run | — | Needs about 78.2 GB; 28.0 GB of memory is free (22.4 GB usable by the GPU) | — | — |
Where the memory goes at IQ2_M
- Weights in system RAM
- 26.8 GB
Context length vs. KV cache
| Context | KV f16 | KV q8_0 | KV q4_0 |
|---|---|---|---|
| 4k | 25.5 GBRuns slowly est. 2.4 tok/s1.6–3.1theoretical estimate ±30% | 24.8 GBRuns slowly est. 2.4 tok/s1.7–3.1theoretical estimate ±30% | 24.5 GBRuns slowly est. 2.4 tok/s1.7–3.2theoretical estimate ±30% |
| 8k | 26.8 GBRuns slowly est. 2.2 tok/s1.6–2.9theoretical estimate ±30% | 25.6 GBRuns slowly est. 2.3 tok/s1.6–3.1theoretical estimate ±30% | 24.9 GBRuns slowly est. 2.4 tok/s1.7–3.1theoretical estimate ±30% |
| 16k | 30.1 GBdoes not fit | 27.0 GBRuns slowly est. 2.2 tok/s1.6–2.9theoretical estimate ±30% | 25.7 GBRuns slowly est. 2.3 tok/s1.6–3.0theoretical estimate ±30% |
| 32k | 35.7 GBdoes not fit | 30.7 GBdoes not fit | 27.2 GBRuns slowly est. 2.2 tok/s1.5–2.9theoretical estimate ±30% |
| 64k | 46.7 GBdoes not fit | 36.7 GBdoes not fit | 31.3 GBdoes not fit |
| 128k | 68.8 GBdoes not fit | 48.8 GBdoes not fit | 38.1 GBdoes not fit |
S = Runs great · A = Runs well · B = Runs slowly · F = Won't run
At IQ2_M the verdict stays Runs slowly up to 8k context with an f16 KV cache, and up to 16k with q8_0.
Estimated speed
- Generation
- est. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%)
- Prompt processing
- Prompt-processing estimates only apply when the whole model runs on the GPU.
Measured on this exact combination
No public measurement for this exact combination yet; the numbers above are estimates.
If this is not enough
A smaller model that runs well on the Apple M4
Runs greatMiniCPM-V 4.7 35B-A3B: Runs great, est. 37.5 tok/s (30.0–45.0, calibrated estimate ±20%)
A lower quant of Llama 3.3 70B
Runs slowlyIQ2_M (24.12 GB, heavy quality loss): Runs slowly, est. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%)
Llama 3.3 70B on the Apple M4 Pro
Runs slowlyQ4_K_M: Runs slowly, est. 3.0 tok/s (2.1–3.9, theoretical estimate ±30%)
How to run it
Commands for the IQ2_M file. Both tools download from Hugging Face on first run.
File: Llama-3.3-70B-Instruct-IQ2_M.gguf (24.12 GB) from bartowski/Llama-3.3-70B-Instruct-GGUF.
llama-server -hf bartowski/Llama-3.3-70B-Instruct-GGUF:IQ2_M -c 8192 -ngl 0 -fa on- -ngl 0 keeps everything on the CPU.
- -fa on enables flash attention (needed for KV cache quantization).
- --cache-type-k q8_0 --cache-type-v q8_0 shrinks the KV cache to 1.44 GB at 8k context.
Terminal:
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
ollama run hf.co/bartowski/Llama-3.3-70B-Instruct-GGUF:IQ2_M- Ollama defaults to 4k context on GPUs under 24 GB. Set OLLAMA_CONTEXT_LENGTH when starting the server, or /set parameter num_ctx inside the chat.
/set parameter num_ctx 8192
Related pages
Guide: Running local LLMs on a Mac (M4, M5): how much unified memory?
Other models on the Apple M4
Llama 3.3 70B on other hardware
Frequently asked questions
Can I run Llama 3.3 70B on the Apple M4?
Q4_K_M needs 45.8 GB, 23.4 GB more than the 22.4 GB of usable memory. IQ2_M (heavy quality loss) works instead: Runs slowly, est. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%). IQ2_M needs 26.8 GB at 8k context; the GPU can use 22.4 GB of the unified memory.
How much context can Llama 3.3 70B use on the Apple M4?
At IQ2_M the verdict stays Runs slowly up to 8k context with an f16 KV cache (2.7 GB of KV) and up to 16k with a q8_0 KV cache (2.9 GB). The model supports up to 128k.
Which quant should I use, and how fast is it?
IQ2_M (heavy quality loss) (24.12 GB) is the recommended balance: Runs slowly, est. 2.2 tok/s (1.6–2.9, theoretical estimate ±30%). It is also the largest tracked file that gets this verdict.
Speeds are estimates from memory bandwidth, calibrated against public benchmarks, and each one comes with an error band and a confidence label. Real results vary with drivers, backend, context length and thermals.