The best GPUs for running AI models locally (2026)

If you want to run large language models on your own computer, the graphics card matters more than anything else in the box. Two numbers decide almost everything. The amount of memory sets the largest model you can load, and the speed of that memory sets how quickly the model writes. Buy the most VRAM you can afford first, then the most bandwidth, and only then look at the rest of the spec sheet.

That makes 24GB the sweet spot in 2026. The strongest open models that fit on one card, such as Qwen3.8 27B and Gemma 4 31B, need about 18GB to 23GB at 4-bit, and the cheapest fast way to get 24GB is still a used RTX 3090. Below that, 16GB cards handle models up to about 20B parameters. Above it, you are paying for speed or for 32GB to 96GB of memory, and prices climb steeply.

How we picked

We rank cards on memory first, because a model that doesn’t fit in VRAM either won’t run or spills into system RAM that is many times slower. Bandwidth comes second: generating each token means reading the model’s active weights from memory, so a card with twice the bandwidth writes roughly twice as fast. Then price, measured as cost per gigabyte of reasonably fast memory. Software support breaks ties. NVIDIA’s CUDA runs everything, AMD’s ROCm and Vulkan run most popular tools, and Intel’s stack works but needs more setup.

We have not lab-tested these cards. Speeds in this guide come from our estimate model, which predicts single-user generation speed from memory bandwidth and model size and is calibrated against published llama.cpp results. The methodology page explains how it works and where it can be wrong. Estimates carry a tilde (~) and assume a llama.cpp-style runtime, 8K of context and one user. They cover generation speed only, not how fast a card reads a long prompt. Measured results live on our benchmarks page.

We judged every card on the open models people actually run in October 2026. For dense models that means Qwen3.8 27B and Gemma 4 31B. For fast mixture-of-experts (MoE) models, which only read a small slice of their weights per token, we used Qwen3.6 35B-A3B and gpt-oss-20b. For the big ones, gpt-oss-120b and Qwen3.5 122B-A10B. The chart shows why bandwidth matters so much: the same model at the same quantization runs about twice as fast on an RTX 5090 as on an RTX 3090, and more than three times as fast on a 3090 as on a 128GB mini PC.

Generation speed

Qwen3.8 27B at Q4_K_M: bandwidth sets the pace

Estimated tokens per second, single user, higher is better.

1. RTX 5090

   32GB · 1,792 GB/s

   ~80tok/s

2. RTX 4090

   24GB · 1,008 GB/s

   ~47tok/s

3. RTX 3090

   24GB · 936 GB/s

   ~39tok/s

4. RX 7900 XTX

   24GB · 960 GB/s

   ~36tok/s

5. R9700

   32GB · 640 GB/s

   ~24tok/s

6. M5 Max 128GB

   128GB · 614 GB/s

   ~23tok/s

7. Arc Pro B70

   32GB · 608 GB/s

   ~20tok/s

8. Arc Pro B60

   24GB · 456 GB/s

   ~15tok/s

9. DGX Spark

   128GB · 273 GB/s

   ~12tok/s

10. Ryzen AI Max+ 395

    128GB · 256 GB/s

    ~11tok/s

Estimates from our speed model. Measured results are on the benchmarks page.

Best overall: a used RTX 3090

The RTX 3090 came out in 2020, and it is still the card we would buy first. It has 24GB of GDDR6X at 936 GB/s, within 8 percent of the RTX 4090’s bandwidth, and used cards sold for about $1,050 to $1,450 in October 2026. At about $44 per gigabyte, it is the cheapest way to get fast memory in a card that drops into a normal desktop and runs current CUDA releases. Buy it if you want the best models that fit on one card and you are comfortable buying used.

It runs Qwen3.8 27B at Q5_K_M (about 21GB) at ~34 tok/s, estimated, or at Q4_K_M with 64K of context in about 22GB. Gemma 4 31B fits at Q4_K_M (about 22.5GB) at ~33 tok/s, though only with a modest context of around 16K tokens. MoE models are much quicker: Gemma 4 26B A4B at Q5_K_M runs at ~109 tok/s and gpt-oss-20b at ~119 tok/s. With 64GB of system RAM holding the rest of the model, gpt-oss-120b runs at ~21 tok/s. The best models for the RTX 3090 page lists everything else.

The downsides come with age. Many cards spent years in mining rigs, the memory chips on the back of the board run hot and often need new thermal pads, and the 350W rating hides sharp power spikes, so don’t skimp on the power supply. Newer cards also accelerate FP8 and FP4 formats in hardware, which the 3090 can’t. That matters for some vLLM deployments and very little for llama.cpp’s GGUF files.

If you prefer AMD, the RX 7900 XTX sells for about the same $1,050 used with 24GB at 960 GB/s, but prompt processing is slower and some tools are CUDA-only (RTX 3090 vs RX 7900 XTX). The RTX 4090 is nearly 20 percent faster at generation (~40 tok/s on Qwen3.8 27B at Q5_K_M, estimated) and much faster at reading long prompts, but used cards cost about $2,500 (RTX 3090 vs RTX 4090). If you won’t buy used, the RTX 5070 Ti costs about $1,200 new with 896 GB/s, but only 16GB (RTX 3090 vs RTX 5070 Ti). Our RTX 3090 guide covers buying and setting one up, and the Sleeper 3090 shows a complete single-card build.

Best budget: RTX 5060 Ti 16GB

The RTX 5060 Ti 16GB is the cheapest new card we’d buy for local AI. About $620 gets you 16GB of GDDR7 at 448 GB/s, full CUDA support and a 180W board power that works with modest power supplies. It’s the card for adding local AI to a gaming PC or building a first rig on a budget.

Sixteen gigabytes covers models up to about 20B parameters. gpt-oss-20b fits entirely on the card (about 15GB) and runs at ~74 tok/s, estimated, and Qwen3.5 9B runs at practically lossless Q8_0 at ~32 tok/s. MoE models hold up well when they spill over: Qwen3.6 35B-A3B at Q4_K_M runs at ~42 tok/s with part of it in system RAM. Dense 27B to 31B models are the limit. Qwen3.8 27B squeezes in only at Q3_K_M (about 15GB), with little room left for context.

The weak spot is the 128-bit memory bus, which leaves it roughly 40 percent slower than a 3090 on models both cards can hold. It also uses a PCIe 5.0 x8 link, which matters little for one card but can slow a two-card setup on an older PCIe 3.0 motherboard. Check that the listing says 16GB, because an 8GB card shares the name.

On a tighter budget, a used RTX 3060 12GB costs about $270 and runs Qwen3.5 9B at Q6_K at ~30 tok/s (RTX 3060 12GB vs RTX 5060 Ti 16GB). For about $650 new, the Intel Arc Pro B60 gives you 24GB, enough for Qwen3.8 27B at Q5_K_M, but its 456 GB/s bus and less mature software hold it to ~13 tok/s on that model (RTX 3090 vs Arc Pro B60). The RX 9060 XT 16GB costs about $470 but has nearly 30 percent less bandwidth (RTX 5060 Ti vs RX 9060 XT), and a used RTX 4060 Ti 16GB at about $380 is slower still (RTX 4060 Ti vs RTX 5060 Ti). If you can stretch to about $1,200, the RTX 5070 Ti doubles the bandwidth and runs gpt-oss-20b at ~125 tok/s. Two 5060 Tis make a 32GB pair for about $1,240, and the GPUter build shows a complete single-card system.

Fastest consumer GPU: RTX 5090

The RTX 5090 is the fastest card you can put in a normal desktop. Its 1,792 GB/s of bandwidth is almost double a 3090’s, and its 32GB holds the best single-card models at higher quality with room for long context. Buy it if speed is worth more to you than money, for example for coding agents that generate thousands of tokens per task.

Estimated, it runs Qwen3.8 27B at Q6_K (about 24GB) at ~60 tok/s, or at Q4_K_M at ~80 tok/s, about twice the 3090’s ~39 tok/s. Gemma 4 31B runs at Q5_K_M at ~59 tok/s, and Qwen3.6 35B-A3B fits at Q5_K_M with room to spare and runs at ~196 tok/s.

The catch is price. Launch MSRP was $1,999; in October 2026 new cards listed for about $4,300 to $5,000 because of the GDDR7 shortage. It also draws up to 575W through a single 12V-2x6 connector, so plan on a quality 1,000W or larger power supply and good case airflow. And 32GB still isn’t enough for gpt-oss-120b. With the rest of the model in system RAM, it runs at ~25 tok/s, barely ahead of a 3090 doing the same (~21 tok/s), because the slow system memory sets the pace.

Two used 3090s cost about $2,100 for 48GB: more memory, but not more speed. The RTX 4090 at about $2,500 used sits in between (RTX 4090 vs RTX 5090, RTX 3090 vs RTX 5090). The RTX 5080 is not a cheaper route to the same thing, since it has only 16GB (RTX 5080 vs RTX 5090).

Best for multi-GPU builds: Radeon AI PRO R9700

The Radeon AI PRO R9700 is the cheapest well-supported way to put a lot of new VRAM in one tower. Each card has 32GB at 640 GB/s, a blower cooler that exhausts out the back and a 300W rating, so four fit in a workstation case for 128GB. At about $1,800 each (up from a $1,299 MSRP), that’s about $56 per gigabyte, against about $134 for an RTX 5090.

One card runs Qwen3.6 35B-A3B at Q5_K_M at ~78 tok/s and Qwen3.8 27B at Q6_K at ~18 tok/s, estimated. Four cards hold gpt-oss-120b entirely in VRAM at ~37 tok/s and Qwen3.5 122B-A10B at Q6_K at ~20 tok/s.

Be clear about what you are buying: capacity, not speed. On dense models one R9700 is slower than a used 3090 (~24 vs ~39 tok/s on Qwen3.8 27B at Q4_K_M, estimated), and tools that only support CUDA won’t run. ROCm officially supports the card, and both llama.cpp and vLLM work on it, so the common local stack is covered. Plan on Linux, where ROCm support is most complete.

The Arc Pro B70 is the cheapest new 32GB card at about $1,300, with similar bandwidth (608 GB/s) but software that still needs more setup (R9700 vs Arc Pro B70). Four used 3090s give you 96GB for about $4,200 and run gpt-oss-120b faster (~56 tok/s), but they draw about 1,400W on their own and usually need risers or an open frame, like the FJÄLLBO build. If you want CUDA at 32GB, see RTX 5090 vs R9700.

Most memory on one card: RTX PRO 6000 Blackwell

The RTX PRO 6000 Blackwell puts 96GB on one card at the same 1,792 GB/s as the RTX 5090. That is enough to run 100B-class models entirely in VRAM, without the splitting, risers and power planning that multi-GPU rigs need. It is a card for professionals and well-funded enthusiasts. At about $16,000 in October 2026, close to double its 2025 street price, one card costs more than a complete four-3090 workstation.

The speed difference is large. Estimated, gpt-oss-120b runs at ~172 tok/s, Qwen3.5 122B-A10B at Q4_K_M at ~105 tok/s and GLM-4.5-Air at Q5_K_M at ~80 tok/s. A 128GB mini PC manages ~35 tok/s on gpt-oss-120b. Two cards (192GB) can run DeepSeek V4 Flash at its native FP4, at an estimated ~72 tok/s.

NVIDIA sells three versions with the same memory: a 600W Workstation Edition, a 300W Max-Q blower card that stacks four to a tower (as in the a16z workstation), and a passively cooled Server Edition. If 72GB is enough, the RTX PRO 5000 Blackwell 72GB holds gpt-oss-120b for about $9,100, the most recent street price we found (RTX 5090 vs RTX PRO 6000).

Best for 100B-class models: Ryzen AI Max+ 395

Sometimes the best GPU for local AI isn’t a graphics card. The Ryzen AI Max+ 395, sold in mini PCs and known as Strix Halo, shares up to 128GB of memory between its CPU and its integrated GPU. Windows lets the GPU use up to 96GB, and Linux about 120GB after a kernel setting change. That’s as much GPU memory as the $16,000 RTX PRO 6000, in a quiet box whose chip runs at 45 to 120W.

Bandwidth is only 256 GB/s, which suits MoE models that read a small slice of their weights per token. gpt-oss-120b runs at ~35 tok/s, estimated, and Qwen3.5 122B-A10B at Q4_K_M at ~18 tok/s. A 24GB card with 64GB of system RAM manages ~21 tok/s on gpt-oss-120b, so the mini PC wins on the big models before you even count noise and power.

Dense models are another story. Gemma 4 31B at Q4_K_M runs at ~10 tok/s here against ~33 tok/s on a 3090, and prompt processing is slower than on NVIDIA hardware, so long documents take a while to read in. 128GB systems sold for about $3,300 to $4,350 in October 2026, up from about $2,000 in 2025. Our guide to the best Macs and mini PCs for local LLMs compares it with the Mac Studio and NVIDIA’s DGX Spark (DGX Spark vs Ryzen AI Max+ 395).

Our picks compared

The table puts our picks next to the main alternatives. The 120B column shows where the lines fall for a big model: 16GB cards can’t hold gpt-oss-120b even with 64GB of system RAM, 24GB and 32GB cards run it with part of the model offloaded (amber), and only the 96GB card and the mini PC run it entirely in fast memory.

GPU	9B speed	27B speed	120B MoE speed	Price
RTX 309024GB · 936 GB/s	~98tok/s	~39tok/s	~21tok/s	~$1,050 usedCheck Amazon ↗
RTX 5060 Ti 16GB16GB · 448 GB/s	~55tok/s	~10tok/s	Won't fit	~$620Check Amazon ↗
RTX 509032GB · 1,792 GB/s	~190tok/s	~80tok/s	~25tok/s	~$4,300Check Amazon ↗
R970032GB · 640 GB/s	~61tok/s	~24tok/s	~23tok/s	~$1,800Check Amazon ↗
RTX PRO 600096GB · 1,792 GB/s	~190tok/s	~80tok/s	~170tok/s	~$16,000Check Amazon ↗
Ryzen AI Max+ 395128GB unified · 256 GB/s	~30tok/s	~11tok/s	~35tok/s	~$3,300Check Amazon ↗
RTX 409024GB · 1,008 GB/s	~115tok/s	~47tok/s	~21tok/s	~$2,500 usedCheck Amazon ↗
RX 7900 XTX24GB · 960 GB/s	~88tok/s	~36tok/s	~20tok/s	~$1,050 usedCheck Amazon ↗
RTX 5070 Ti16GB · 896 GB/s	~105tok/s	~13tok/s	Won't fit	~$1,200Check Amazon ↗
Arc Pro B6024GB · 456 GB/s	~38tok/s	~15tok/s	~18tok/s	~$650Check Amazon ↗
Arc Pro B7032GB · 608 GB/s	~49tok/s	~20tok/s	~21tok/s	~$1,300Check Amazon ↗

How much VRAM do you need?

A model needs memory for its weights, for the context cache that grows with every token in the conversation, and for some runtime overhead. At 4-bit (Q4_K_M) with 8K of context, this is roughly where each memory size lands:

Model	Size	Q4_K_M	Q6_K	Q8_0
Qwen3.5 9B	9B	7.2 GB	9.5 GB	12 GB
gpt-oss-20b	20.9B (3.6B active)	15 GBMXFP4	—	—
Gemma 4 26B A4B	26B (3.8B active)	17 GB	23 GB	30 GB
Qwen3.8 27B	27B	18 GB	25 GB	32 GB
Gemma 4 31B	31B	22 GB	30 GB	36 GB
Qwen3.6 35B-A3B	35B (3B active)	24 GB	32 GB	41 GB
gpt-oss-120b	117B (5.1B active)	68 GBMXFP4	—	—
Qwen3.5 122B-A10B	122B (10B active)	77 GB	104 GB	134 GB

Context adds up. Qwen3.8 27B at Q4_K_M needs about 18GB at 8K tokens, about 22GB at 64K and about 26GB at 128K. Gemma 4 31B grows faster, from about 22.5GB at 8K to about 27GB at 64K. Quantization is the other lever: Q4_K_M keeps most of a model’s quality at about 30 percent of its 16-bit size, and Q8_0 is close to lossless at about half. Our guides to how much VRAM you need and quantization go deeper, and the VRAM calculator checks any model, quantization and context length against your card.

New or used?

Used is where the value is in 2026. The shortage hit new cards hardest, so a used RTX 3090 at about $1,050 gets you more memory than a new $1,200 RTX 5070 Ti. When you buy used, pick a seller or marketplace that accepts returns, and run a long generation job as soon as the card arrives while you watch temperatures. Hot memory or throttling usually points to tired thermal pads, which are a cheap fix, but you want to find out inside the return window.

Old datacenter cards are the other used route, and they ask more of you. The Tesla P40 is the cheapest 24GB card at about $230 to $330, but it has no tensor cores, very slow FP16, passive cooling that needs a fan shroud, and no support in CUDA 13 (RTX 3090 vs Tesla P40). The Tesla V100 32GB has 900 GB/s for about $700, but most units are SXM2 modules that need an adapter board and custom cooling, as in this V100 workstation, and CUDA 13 dropped it too. The Instinct MI50 32GB was a bargain at about $150 before the shortage; at $600 or more, and without support in current ROCm releases, it no longer is (RTX 3090 vs MI50).

Used workstation cards hold their value. A 48GB RTX A6000 sold for about $4,400 and an RTX 6000 Ada for $4,750 or more, which is RTX 5090 money for more memory but much less bandwidth.

What about two or more GPUs?

Adding cards adds memory, not speed. llama.cpp and the apps built on it split a model’s layers across GPUs, and each token passes through the cards one after another, so two cards run a model at about the speed of one. What you gain is room for bigger models and better quantizations. Estimated, two 3090s (48GB) run Qwen3.8 27B at Q8_0 at ~22 tok/s and cut gpt-oss-120b’s offloading enough to reach ~32 tok/s. Four (96GB) hold gpt-oss-120b entirely, at ~56 tok/s.

Tensor parallelism in vLLM and similar runtimes can make several cards faster than one, but it works best with matched cards and a fast link between them. The 3090 is one of the last GeForce cards with NVLink, which is why dual-3090 rigs such as the Hardigg 3090 still use it.

Plan the platform before you buy the cards. Most consumer motherboards have one full-speed x16 slot and at best split it into two x8 slots, so two GPUs is the practical limit. Three or more call for a workstation or server platform with more PCIe lanes (Threadripper, Xeon or EPYC), as in The Machine with twelve 3090s or PewDiePie’s seven-card RTX 4000 Ada rig. Size the power supply for the cards’ combined rating plus 20 to 30 percent headroom. If you are building a Linux server, our guide to NVIDIA GPUs on Linux covers drivers and setup.

What to avoid

8GB cards. The RTX 5060 costs about $450, and models like Qwen3.5 9B at Q4_K_M (about 7GB) are its ceiling. Spend about $170 more on the 16GB RTX 5060 Ti.

12GB cards at new-card prices. The RTX 5070 has fast memory (672 GB/s), but at about $800 for 12GB it costs more than the 16GB 5060 Ti and holds smaller models.

The RTX 5080 for AI alone. At about $1,600 it has the same 16GB as the RTX 5070 Ti and only about 7 percent more bandwidth.

A used RTX 4090 at full price. At $2,500 to $3,000 it costs more than twice as much as a 3090 for the same 24GB. It makes sense only if you need its faster prompt processing.

Old hardware at shortage prices. Pascal cards like the P40 have lost CUDA 13 support, and the MI50 has lost current ROCm support. They still run llama.cpp, but they are tinkerer’s cards, not safe buys.

Low-memory lookalikes. The RTX 5060 Ti, RTX 3060 and RX 9060 XT all come in 8GB versions with the same name. The 8GB RTX 3060 also has a narrower bus. Check the memory size before you pay.