Running a local LLM on multiple GPUs: what you gain and what you do not

Putting two graphics cards in one machine is the oldest trick for running a model that does not fit on one. It works, and it works differently from what most people expect: a second card gives you more memory and almost none of the extra speed. Knowing why saves you from buying the wrong pair.

How the split works

By default llama.cpp divides a model by layers. The first half of the layers sit on the first card and the second half on the second. For every token the work flows through card one, then card two, and only one card is busy at any moment. Between them travels a small activation vector per token, a few kilobytes, which is why a normal PCIe slot is plenty and the lane count hardly matters. The options are -sm layer, which is the default, and -ts for the proportions, for example -ts 1,1 for an even split or -ts 3,2 to put more on the larger card. Our flags guide lists the related settings.

The consequence is simple. Memory adds up: 2 cards with 12 GB each hold what a 24 GB card holds, minus the runtime overhead that each one carries. Bandwidth does not add up. Each token still reads every weight once, and each weight is read at the speed of the card that holds it, so the model runs at roughly the average of the 2 cards' speeds, not the sum.

There are other modes. -sm row and tensor splitting in recent builds split each layer across cards so both work at once. They can help on cards joined by a fast link, and in practice the gain on consumer hardware is modest and the setup is touchier. Serving frameworks such as vLLM use tensor parallelism properly and it scales well, but that is for serving many users, not for one person chatting.

Two cheap cards or one good one

Take the case people ask about most: two used RTX 3060 12 GB cards for 24 GB in total, against one used RTX 3090. The pair costs less. It also reads memory at 360 GB/s. On Qwen3.5 27B at 4 bits our engine estimates about 17 tokens per second for the pair, and about 33.0 on the 3090 for the same model. The pair works and it is usable, and the 3090 is nearly 3 times faster for a modest price difference.

The pairing only makes real sense in three cases. You already own one card and a second is the cheapest way to add memory. You need a total beyond what any single card offers, such as 48 GB from two used 24 GB cards, which fits a 70B model at 4 bits. Or the card you want does not exist at your price, so memory per dollar decides.

Matching helps. Two identical cards split cleanly. A fast card with a slow one works, but the slow card limits its layers and you will see it in the speed.

What it costs besides the cards

A second card brings a list of practical bills. The power supply has to feed both, and a pair of 3090s asks for 700 W of graphics power before the rest of the machine. The case has to fit two thick cards with air between them, since a card that breathes the other's exhaust throttles. The motherboard needs two full-size slots, and the second one often runs at a lower lane count, which is fine for layer splitting. Heat and noise double. And the running cost doubles too, which the cost guide puts into dollars.

A rule we use: add a second card to extend a card you have, and when you are starting from nothing, buy the biggest single card you can afford.

Setting it up

Both Ollama and llama.cpp use every GPU they can see, and most of the work is making sure they see them. Run nvidia-smi and check that both cards appear and that the driver version is the same for both. In llama.cpp the split happens on its own, and -ts is for the case where you want to control it. Ollama spreads a model across cards when it does not fit on one, and its OLLAMA_SCHED_SPREAD setting asks it to spread a model that would fit on one card as well, which is rarely worth it. CUDA_VISIBLE_DEVICES hides a card from a program, which is how you keep one reserved for your display or for another job.

Mixed generations work and are worth thinking through. A newer card and an older one with the same amount of memory split cleanly, and the older one sets the pace for its share of the layers. Different amounts of memory are fine as long as you set the split in proportion. What you cannot do is combine an NVIDIA and an AMD card in one split without using a backend that supports both, and Vulkan builds of llama.cpp are the route people take for it. Expect each card to hold about a gigabyte of overhead of its own, so a pair of 12 GB cards holds a little less than a single 24 GB card.

Pairs that make sense

Two 16 GB cards give 32 GB and run a 30B class model at a good quality with a long context. Two 24 GB cards give 48 GB, the amount that holds 70B models at 4 bits and the sweet spot for local use of large dense models. A 3090 and a 3060 12 GB give 36 GB, a cheap upgrade path from a single 3090. In every case, check the total against the model in the calculator, allow a gigabyte or so per card for overhead, and keep a little headroom for the context window.