Two GPUs for local LLMs: memory adds up, speed doesn't

A second graphics card adds memory, not memory bandwidth: a model that already fits on one card gets no faster, but one that spilled into system RAM can get several times faster once it fits. With llama.cpp's default split, two RTX 3060 12GB cards hold about what one 24GB RTX 3090 holds, but each token is generated at roughly one RTX 3060's speed.

What splitting a model across GPUs does

llama.cpp, which Ollama 0.30 and later also runs on, splits a model across cards by layers unless you tell it otherwise (--split-mode layer). Each card holds a contiguous block of layers, and the KV cache (the memory that holds the context) for those layers stays on the same card. To generate a token, the first card runs its layers, then hands a small intermediate result to the next card, which runs the rest. The cards take turns, so for a single chat the time per token is the sum of the times the cards need to read their shares of the weights. Memory adds up across the cards; memory bandwidth, which sets the speed, does not.

In CanRun's calculation, a model that already fits on one card gets the same estimated speed on two identical cards. Public results point the same way. In one 2024 benchmark set, Llama 3 8B at Q4_K_M (a 4-bit quant) generated a few percent slower on two RTX 3090 cards than on one, and a little slower again on four and six. What the extra cards changed was what fit: Llama 3 70B at Q4_K_M ran out of memory on one RTX 3090 and ran on two.

Llama 3.1 8B on an RTX 3060 12GB has a public benchmark. It fits with room to spare and Runs great, measured on real hardware. For two of these cards, the calculator shows the 8k estimate printed under the measurement, not the measured speed, because public results exist only for single cards. In CanRun's calculation the second card does not raise it.

Prompt processing is the exception. Layer split works on many prompt tokens at once, so the cards can be busy at the same time. In the same 2024 benchmark set, an 8k-token prompt was processed faster at each step from one to six RTX 3090 cards. llama.cpp's multi-GPU guide recommends the layer mode when fast prompt processing matters. CanRun does not estimate prompt speed for several cards (only single-card combo pages show it), so this guide sticks to generation speed. A newer, experimental mode splits every layer across the cards instead; it is covered in the setup section below.

When a second card pays off: models that do not fit on one

A second card earns its place when a model does not fit in one card's VRAM. Without it, part of a dense model's weights sits in system RAM (partial offload) and generation slows down sharply; a mixture-of-experts (MoE) model keeps some of its expert weights in RAM instead. The table lists what common models need on one graphics card at Q4_K_M (MXFP4 for gpt-oss), an 8k context and an f16 KV cache. Each extra card adds a little on top, as the section on memory costs explains.

Gemma 3 27B shows the difference. It needs 17.8 GB, more than the 11.4 GB an RTX 3060 12GB can use, so on one card part of it sits in system RAM and it Runs slowly. The bar shows how much has to leave the card.

Two of these cards have 24GB between them. Only the first card drives the display, so the pair can use 23.4 GB, the same as one RTX 3090. Gemma 3 27B fits there with several gigabytes to spare and nothing goes to system RAM. In CanRun's calculator the pair runs it close to three times as fast as one card does and rates it Runs well, a theoretical estimate (±30%) like every multi-card result.

MoE models gain in a different way. On one RTX 3060 12GB, Qwen3 30B-A3B keeps the experts that do not fit in system RAM and Runs well, a theoretical estimate. Our page for Qwen3 30B-A3B (2507) on the GeForce RTX 3060 12GB shows that split. It needs 19.8 GB on one card, so on two cards it fits entirely in VRAM, and the calculator moves it up to Runs great, also theoretical.

When the model already fits, a second card changes nothing you would notice. Qwen3 14B fits on one RTX 3060 12GB and Runs well there; on two, CanRun's estimate stays the same.

The largest MoE models lean on system RAM as well. With 32GB of RAM, gpt-oss-120b does not fit with one RTX 3090, because the experts that do not fit in VRAM need more RAM than is left. With 64GB of RAM one RTX 3090 runs it, keeping most experts in RAM, and it Runs slowly, a theoretical estimate. A second RTX 3090 holds more of the experts, so in CanRun's calculator the rest fits in 32GB of RAM and the pair rates it Runs well (theoretical). For the single-card case, see our page for gpt-oss-120b on the GeForce RTX 3090 and the guide Running MoE models bigger than your GPU: gpt-oss-120b with system RAM.

Two RTX 3060 12GB or one RTX 3090? Two RTX 5060 Ti 16GB or one RTX 5090?

These two comparisons come up often because the totals match: two 12GB cards make 24GB, like an RTX 3090, and two 16GB cards make 32GB, like an RTX 5090. The grid shows each card on its own at 8k with an f16 KV cache, 32GB of DDR5 system RAM and Windows. Pairs are not in the grid; the text below gives what CanRun's calculator shows for them.

On memory, the pairs come close: in CanRun's calculation a pair fits almost everything the matching single card fits, minus one extra compute buffer (the runtime's working memory), as the memory section below explains.

On speed, they do not. The RTX 3060 12GB reads its memory at 360 GB/s and the RTX 3090 at 936 GB/s, about two and a half times as fast. The RTX 5060 Ti 16GB has 448 GB/s and the RTX 5090 1,792 GB/s, four times as much. Because the cards in a pair take turns, the pair generates at about one smaller card's speed, and these ratios carry over to the estimates.

24GB: two RTX 3060 12GB against one RTX 3090

Gemma 3 27B Runs great on the RTX 3090 and Runs well on the pair, as the previous section showed. Qwen3 32B fits on the RTX 3090 and Runs well. On the pair it also fits, with less than a gigabyte to spare, because each card needs its own compute buffer. At the RTX 3060's speed it lands just under the higher line that thinking models must clear, so the calculator shows Runs slowly, a theoretical estimate that could fall on either side of that line. On one RTX 3060 12GB, Qwen3 32B runs partly from system RAM and Runs slowly.

32GB: two RTX 5060 Ti 16GB against one RTX 5090

Qwen3 32B Runs great on the RTX 5090. One RTX 5060 Ti 16GB has to put part of it in system RAM, and it Runs slowly there. Two of them hold it fully, and the calculator rates it Runs well (theoretical).

A dense 70B model needs more than either. Llama 3.3 70B needs 45.2 GB at Q4_K_M, so on the RTX 5090 part of it sits in system RAM and it Runs slowly; two RTX 5060 Ti 16GB cards have the same 32GB and get the same partial offload. Two RTX 3090 cards, with 48GB between them, hold it at 8k with less than two gigabytes to spare, and the calculator rates it Runs well (theoretical); at 16k part of it spills into system RAM. One RTX 3090 is rated Won't run at Q4_K_M, because too much of the model sits in system RAM. In the 2024 benchmark set above, two RTX 3090 cards generated Llama 3 70B, the same size, somewhat faster than CanRun's estimate.

What the pairs cost

Our data lists the RTX 3060 12GB at $250 street (as of 2026-08-09), the RTX 3090 at $1,050 street (as of 2026-08-09), the RTX 5060 Ti 16GB at $569 street (as of 2026-08-09) and the RTX 5090 at $4,200 street (as of 2026-08-09). At those listed prices, two RTX 3060 12GB cards cost a little under half an RTX 3090, and two RTX 5060 Ti 16GB cards a little over a quarter of an RTX 5090. The listing does not say whether each price is for a new or a used card. A pair buys the memory; the single card buys the memory and the speed. Two cards also need two free PCIe slots with room between them, a case that fits both and a power supply sized for both. For single cards by budget, see Recommended GPUs for local LLMs in 2026: what each budget runs.

Mixing different cards

Cards do not have to match. Without other instructions, llama.cpp splits the layers in proportion to each card's free memory. An RTX 3090 with an RTX 3060 12GB gets about two thirds of the layers on the RTX 3090 and one third on the RTX 3060, out of 36GB in total, minus the display share. Each token still passes through both cards, so the third on the RTX 3060 is read at the RTX 3060's speed, and that share slows the whole run.

For a model that fits on the RTX 3090 alone, that is a loss. Split across both cards, CanRun's estimate for such models drops by about a third. Qwen3 8B Runs great on the RTX 3090 either way, but Gemma 4 31B Runs great on the RTX 3090 alone and drops to Runs well when split with the RTX 3060 (theoretical). The calculator adds a note when one card can hold the model alone and would be faster that way. Ollama's FAQ says it loads a model onto a single GPU when the model fits there, but it judges that at its default context, which grows with the cards' total memory (see the Ollama section). llama.cpp uses every card it can see, so pick one with --device (see the next section).

The gain is memory. Nemotron 3.5 Lightning 30B-A3B needs 25.1 GB, more than the 23.4 GB an RTX 3090 can use. On the RTX 3090 alone it keeps some experts in system RAM and Runs well, a theoretical estimate; with the RTX 3060 12GB added, it fits on the GPUs. Its speed estimate barely changes; what changes is that no experts sit in system RAM. When this guide was written, that was the only model in CanRun's catalog that the extra RTX 3060 12GB moved fully onto the GPUs at 8k, while several models that fit on the RTX 3090 alone dropped from Runs great once split. Check your own pair in the calculator before you count on a mixed setup.

The calculator mixes NVIDIA desktop cards with each other or AMD desktop cards with each other, up to four in total; Intel Arc and older cards such as the GTX 10 series are left out. NVIDIA and AMD cards in one machine need a llama.cpp build that drives both, which the FAQ below covers.

What each extra card costs in memory

Adding cards adds VRAM, but not all of it holds the model.

• A compute buffer per card. In CanRun's calculation each card holds its own compute buffer, about half a gigabyte at 8k, growing slowly with the context. Two cards pay it twice, four cards four times.
• The display share, once. Windows keeps some VRAM on the card that drives your monitor (the note under the memory table gives the amount). With several cards, CanRun takes it from the first card only, which then gets slightly fewer layers in llama.cpp's free-memory split.
• A margin on every card. llama.cpp's automatic fitting, on by default since December 2025, keeps some VRAM free on each card (--fit-target) and adjusts the settings you leave unset, such as the context size and the number of GPU layers. The margins add up with the number of cards, so a model that only just fits in CanRun's numbers, such as Qwen3 32B on two RTX 3060 12GB cards, may start with a shorter context or with a layer in system RAM. Check the llama.cpp log or ollama ps.
• Whole layers. llama.cpp gives each layer to one card and puts the output layer on the last card. A card's share is rounded to whole layers, so some VRAM on each card can stay unused. CanRun counts the cards' memory as one pool and does not model this.

Room for context grows with the cards too, because each card keeps the KV cache for its own layers. Qwen3 32B at 32k with an f16 KV cache needs 28.7 GB, more than the 23.4 GB one 24GB card can use, and comfortably within two RTX 3090 cards. How the KV cache grows is explained in KV cache explained: how context length eats VRAM.

Setting it up in llama.cpp and Ollama

llama.cpp

llama.cpp spreads a model across every GPU it can see, so for two cards you usually change nothing. The options below come from llama.cpp's multi-GPU guide. The commands are written for PowerShell on Windows; on Linux, write the model path as ./Qwen3-32B-Q4_K_M.gguf.

llama-server --list-devices
llama-server -m .\Qwen3-32B-Q4_K_M.gguf -c 8192
llama-server -m .\Qwen3-32B-Q4_K_M.gguf -c 8192 -dev CUDA0,CUDA1 -ts 3,1
llama-server -m .\Qwen3-8B-Q4_K_M.gguf -c 8192 -dev CUDA0

The first command lists the cards with their names, such as CUDA0 and CUDA1, and their memory. The second uses the default: layers split across all visible cards by free memory. -c 8192 matches the 8k context CanRun assumes; without it, llama-server starts from the model's full context and its automatic fitting shrinks it. The third command sets a custom split, three parts to the first card and one to the second; -ts (--tensor-split) takes proportions in the order of --device. Use it when the automatic split leaves one card too full, for example the card that drives the display, and give that card the smaller number. With -ts set, the automatic fitting no longer moves layers for you. The last command runs on one card only.

For CUDA builds, the environment variable CUDA_VISIBLE_DEVICES hides cards from llama.cpp altogether: $env:CUDA_VISIBLE_DEVICES="0"; llama-server ... in PowerShell, or CUDA_VISIBLE_DEVICES=0 llama-server ... on Linux. --device picks among the cards llama.cpp can see and works with every backend.

Ollama

Ollama decides on its own. According to its FAQ, it loads a model onto one GPU when the model fits on any single GPU, and spreads it across all GPUs only when it does not. Whether a model fits depends on the context Ollama starts with. Ollama picks its default context length from the GPU memory it finds: 4k tokens below 24 GiB, 32k tokens from 24 to 48 GiB, and 256k tokens at 48 GiB or more (Ollama docs, checked 2026-09-29). To pick that default, Ollama adds up the memory of every card it finds, and its server log records the result on a vram-based default context line. Two RTX 3060 12GB cards therefore start at 32k, where one card alone gets 4k. At 32k Qwen3 14B needs 14.7 GB, more than one RTX 3060 12GB can use, so by the FAQ's rule Ollama spreads it across both cards. An RTX 3090 with an RTX 3060 12GB also starts at 32k, where Qwen3 32B needs 28.7 GB, more than the RTX 3090 alone holds. To match CanRun's 8k basis, set OLLAMA_CONTEXT_LENGTH as below; other ways to set the context are in Ollama slow or forgetting your prompt? Context length defaults.

The environment variable OLLAMA_SCHED_SPREAD makes Ollama always spread across all GPUs. To limit Ollama to certain NVIDIA cards, set CUDA_VISIBLE_DEVICES; Ollama's GPU documentation recommends the card UUIDs that nvidia-smi -L prints, because numeric order can vary. AMD cards use ROCR_VISIBLE_DEVICES instead.

nvidia-smi -L
$env:CUDA_VISIBLE_DEVICES="GPU-your-uuid-1,GPU-your-uuid-2"; $env:OLLAMA_CONTEXT_LENGTH="8192"; ollama serve

Quit Ollama from its taskbar icon before running the second line; the settings apply only to the server started from that PowerShell window. For a lasting setting on Windows, add the variables under "Edit environment variables for your account" and restart Ollama from the Start menu. After loading a model, ollama ps shows whether it sits fully on the GPUs (PROCESSOR reads 100% GPU), and nvidia-smi shows how much memory each card uses.

Tensor split: faster generation, experimental

Since April 2026, llama.cpp has an experimental --split-mode tensor that splits each layer across the cards, so they work on the same token together. In the tests in the pull request that added it, run on two RTX 4090 cards, dense models generated faster than with layer split, while some MoE models generated slower and prompt processing was often slower. It needs flash attention and an unquantized KV cache, does not work with automatic fitting or every model architecture, and depends far more on the link between the cards. For good speed it also needs NVIDIA's NCCL library, which llama.cpp's guide says is not shipped with CUDA and may need installing separately; NVIDIA provides it for Linux. llama.cpp's guide says its performance should be good for NVIDIA cards with CUDA, with no guarantees for other setups. CanRun does not model this mode; all estimates here assume layer split. The older --split-mode row, which older guides mention, was removed from the CUDA backend in July 2026.

How sure are these numbers?

Multi-GPU results are a beta shown only in the calculator. GPU pages, model pages, the combo pages and the grid above all assume one graphics card. Multi-card results are never matched to public benchmarks, so their speeds are labeled theoretical (±30%), and the verdicts this guide gives for pairs come from the calculator, not from the grid.

For identical cards, CanRun checked its formula against ten public results with two to four cards: nine NVIDIA setups and one pair of AMD RX 7900 XTX cards. Every measured speed was equal to or faster than CanRun's estimate, by up to 28% (CanRun's estimate was at most 22% low). All ten used dense models at short context. None used an MoE model or kept experts in system RAM, so the MoE pair results above (Qwen3 30B-A3B, gpt-oss-120b) have no public check, and with experts in RAM CanRun may place too many experts on the GPUs. For mixed cards there are three checks. The only one with cards from the catalog, an RTX 5070 Ti with an RTX 3060 12GB, measured about 7% above the estimate. The other two used an older datacenter card and checked only the split formula. CanRun has no checks for MoE models on mixed cards, for mixed AMD cards or for more than two mixed cards.

The memory figures (weights on the GPU, KV cache, compute buffer per card, display share) are calculated by formula from real GGUF files, as on every CanRun page. What the formula leaves out is listed in the memory section above: the fitting margin on each card, whole-layer rounding and the output layer on the last card. Results assume Windows with the first card driving the display, 32GB of DDR5 system RAM, an 8k context, an f16 KV cache, one request at a time and llama.cpp's default layer split. Verdicts for single cards assume that setup. Models with a thinking mode, such as Qwen3 32B, face speed thresholds 1.5× higher than the legend shows.

FAQ

Does a second GPU make a model generate faster?

Not if the model already fits on one card. With llama.cpp's default layer split the cards take turns on each token, so two identical cards generate at about one card's speed. With the default split, a second card speeds things up only when it keeps a model out of system RAM: Gemma 3 27B on one RTX 3060 12GB Runs slowly, and two of those cards hold it entirely in VRAM.

Are two 12GB cards the same as one 24GB card?

For memory, almost: both have 24GB and lose the same display share, but the pair holds one more compute buffer, so it has about half a gigabyte less for the model at 8k. For speed, no: the pair generates at about the speed of one 12GB card. Against an RTX 3090, two RTX 3060 12GB cards read memory about two and a half times slower, and the estimates follow that.

Do the cards need NVLink or a fast PCIe slot?

Not for llama.cpp's default layer split. Only a small intermediate result moves between the cards for each token, and llama.cpp's multi-GPU guide says this mode can tolerate slow links between GPUs. The experimental tensor split is much more sensitive, and the same guide suggests more PCIe lanes or NVLink when several cards run slower than one.

Can I mix NVIDIA and AMD cards?

Only with a different setup, and CanRun does not estimate it. NVIDIA cards run through CUDA and AMD cards through ROCm, so one llama.cpp process needs a backend that drives both. The simplest route is llama.cpp's prebuilt Vulkan release for Windows (win-vulkan-x64), which sees both cards. llama.cpp's build guide also notes that you can build several backends together and choose devices with --device. CanRun's calculator only mixes cards from the same maker, because too few public results exist to estimate the other setups.

Sources

• llama.cpp: multi-GPU guide: split modes (layer as the default pipeline split with the KV cache on the card that owns each layer, experimental tensor, deprecated row), --tensor-split, --device, --list-devices, CUDA_VISIBLE_DEVICES, tolerance of slow links in layer mode, tensor-mode requirements and NCCL not being shipped with CUDA.
• llama.cpp release b11630: prebuilt Windows builds, including Vulkan.
• llama.cpp source: layer placement: the default split by each device's free memory, whole layers per device, the input layer on the CPU and the output layer on the last device.
• llama.cpp: server README: -c (default: the model's own context), --split-mode, --tensor-split, --main-gpu, --fit, --fit-target (margin per device) and --fit-ctx.
• llama.cpp PR #16653: automatic fitting: --fit, on by default, sets unset options to fit device memory and leaves the allocation alone when a tensor split is set. Merged in December 2025.
• llama.cpp PR #19378: tensor parallelism: experimental --split-mode tensor, merged in April 2026, with test results on two RTX 4090 cards and its limits (flash attention, unquantized KV cache, no --fit, unsupported architectures).
• llama.cpp PR #24216: remove -sm row: row split removed from the CUDA backend in July 2026.
• llama.cpp: build guide: building several backends together, such as CUDA and Vulkan, and choosing devices with --device.
• Ollama FAQ: how Ollama loads models on multiple GPUs (one GPU when the model fits on any single one, otherwise all GPUs) and the PROCESSOR column of ollama ps.
• Ollama docs: GPU: CUDA_VISIBLE_DEVICES with UUIDs from nvidia-smi -L, ROCR_VISIBLE_DEVICES for AMD and GGML_VK_VISIBLE_DEVICES for Vulkan.
• Ollama source: server/routes.go: the default context set from the memory of all GPUs added together, logged as vram-based default context.
• Ollama docs: context length: the VRAM-based default context and OLLAMA_CONTEXT_LENGTH.
• Ollama source: envconfig/config.go: OLLAMA_SCHED_SPREAD, "Always schedule model across all GPUs".
• Ollama v0.30.0 release notes: models run with llama.cpp.
• GPU-Benchmarks-on-LLM-Inference: 2024 llama.cpp results for Llama 3 8B and 70B on one to six RTX 3090 cards and one to eight RTX 4090 cards, used here for the generation and prompt-processing trends and as part of CanRun's multi-GPU check.
• llama.cpp discussion #21112: the RTX 5070 Ti + RTX 3060 12GB result used for CanRun's mixed-card check.