How much VRAM do you need to run LLMs locally?

For most people the number to remember is about 0.6GB of VRAM per billion parameters. That is what a model’s weights take at Q4_K_M, the 4-bit format most local tools default to, and you add the KV cache for your context plus roughly a gigabyte of runtime overhead on top. In practice that comes to about 7GB for Qwen3.5 9B, about 18GB for the dense Qwen3.8 27B, and about 68GB for gpt-oss-120b at 8K context.

Those numbers line up with hardware tiers. An 8GB card handles small models, 16GB covers 9B to 20B models, 24GB is the practical floor for 27B to 35B models, 32GB buys room for higher quality and long context, and you need 96GB or more before 120B-class models fit without spilling into system RAM. The rest of this guide shows where the numbers come from so you can work them out for any model. If you’d rather skip the math, the VRAM calculator does it for every model and GPU in our database.

What fills your VRAM

Three things take up memory when a model runs on a GPU.

Weights. These are the model’s parameters, and their size is the parameter count times the bits stored per weight, divided by eight. Q4_K_M averages about 4.85 bits per weight in our estimates, so a 27B model needs 27 × 4.85 ÷ 8, or about 16.4GB. llama.cpp’s own measurements agree: its quantization docs list Llama 3.1 8B at 4.89 bits per weight and 4.58GiB for Q4_K_M, 7.95GiB for Q8_0, and 14.96GiB at 16-bit. When we know a model’s real file size, we use that instead of the formula.

KV cache. For every token in the context window, the model stores attention keys and values so it doesn’t have to recompute them for each new token. This grows in a straight line with context length, and the cost per token depends on the model’s architecture. Qwen3.8 27B stores 64KiB per token, so 8K tokens take about 0.5GB and 128K tokens take about 8.6GB.

Overhead. The CUDA or Metal runtime, compute buffers and scratch space need room too. We budget 0.6GB plus 3% of the weights. If a monitor is plugged into the same GPU, the desktop takes some VRAM as well.

VRAM by model and quantization

The table below comes from the same estimator that powers our GPU and model pages. It assumes 8K of context, a reasonable working size for chat and short coding sessions, with a 16-bit KV cache. The gpt-oss models ship in their own 4-bit format called MXFP4, so they show a single number.

Model	Size	Q4_K_M	Q6_K	Q8_0
Qwen3.5 9B	9B	7.2 GB	9.5 GB	12 GB
gpt-oss-20b	20.9B (3.6B active)	15 GBMXFP4	—	—
Gemma 4 26B A4B	26B (3.8B active)	17 GB	23 GB	30 GB
Qwen3.8 27B	27B	18 GB	25 GB	32 GB
Gemma 4 31B	31B	22 GB	30 GB	36 GB
Qwen3.6 35B-A3B	35B (3B active)	24 GB	32 GB	41 GB
gpt-oss-120b	117B (5.1B active)	68 GBMXFP4	—	—
Qwen3.5 122B-A10B	122B (10B active)	77 GB	104 GB	134 GB

For models that aren’t in our database, the weights alone scale with parameter count. Add the KV cache and about 1GB of overhead to these.

Real files land close to these figures. The Q4_K_M file for Llama 3.3 70B is 42.5GB, for example. If you want to understand what Q4_K_M and Q8_0 actually do to a model, our quantization guide covers the formats and the measured quality loss at each level.

Context length is the hidden cost

Weights are a fixed cost. The KV cache grows with every token you keep in context, and long-context work can make it the largest item on the bill. Here is the 16-bit KV cache alone for a few models:

The spread comes from architecture: how many layers keep a cache for the full context, how many key and value heads each layer has, and whether some layers only look at a sliding window of recent tokens. Llama 3.3 70B keeps a full-length cache on all 80 of its layers, which is why 128K of context costs more than its 4-bit weights. Most of the 2026 models in our database are far cheaper per token, but the differences still matter. Qwen3.8 27B at Q4_K_M needs about 18GB at 8K and about 22GB at 64K, both fine on a 24GB card, but about 26GB at 128K.

Model	Size	Q4_K_M	Q8_0
Qwen3.8 27B	27B	26 GB	40 GB
Gemma 4 31B	31B	33 GB	46 GB
gpt-oss-120b	117B (5.1B active)	73 GBMXFP4	—
GLM-4.5-Air	106B (12B active)	91 GB	141 GB

Shrinking the KV cache

You can store the cache at lower precision. In llama.cpp the flags are -ctk and -ctv (cache type for keys and values), which default to f16 and accept q8_0 or q4_0, according to the llama-server docs. Ollama uses the OLLAMA_KV_CACHE_TYPE environment variable and needs flash attention, which it turns on automatically when the GPU supports it. Ollama’s FAQ describes q8_0 as about half the memory of f16 with usually no noticeable quality impact, and q4_0 as a quarter of the memory with a loss that can become more noticeable at long context.

With an 8-bit cache, Qwen3.8 27B at Q4_K_M and 128K of context drops from about 26GB to about 22GB, which brings it back inside a 24GB card.

Dense vs mixture-of-experts models

A dense model reads every weight for every token. A mixture-of-experts (MoE) model routes each token through a small subset of its weights, so only the “active” parameters are read per token. The whole model still has to sit in memory, though.

The short version is that memory follows total parameters and speed follows active parameters. Qwen3.6 35B-A3B has 35B parameters in total but about 3B active per token. At Q4_K_M it needs about 24GB, more than the dense Qwen3.8 27B’s 18GB, yet on an RTX 3090 it generates at about 138 tok/s by our estimate against about 39 tok/s for the 27B. OpenAI’s gpt-oss-120b has 117B total parameters and 5.1B active, according to Hugging Face’s gpt-oss announcement.

This changes how you shop. For MoE models, memory capacity matters more than raw bandwidth, which is why large unified-memory machines handle them well.

When the model doesn’t fit: offloading to system RAM

llama.cpp, Ollama and LM Studio can keep part of a model in system RAM and run it from there. How well that works depends on the model type.

For dense models, it’s painful. Every token has to read the CPU-side weights through system memory, which is many times slower than GDDR6X or GDDR7. Qwen3.8 27B at Q4_K_M on an 8GB RTX 5060 runs at about 4 tok/s by our estimate.

For MoE models, it works surprisingly well. You keep the attention layers and shared weights on the GPU and push the expert weights, which are read sparsely, into RAM. llama.cpp’s --cpu-moe and --n-cpu-moe flags do exactly this, and recent llama-server builds try to fit the model to your available memory automatically with --fit, which is on by default. By our estimate, gpt-oss-120b runs at about 21 tok/s on a single 24GB card with 64GB of RAM, and Qwen3.6 35B-A3B runs at about 28 tok/s on an 8GB card.

Our offload estimates assume 64GB of system RAM. If you plan to offload, system RAM capacity and speed become part of your GPU budget. In Ollama, ollama ps shows the split; anything other than “100% GPU” means part of the model is running from system memory.

Unified memory: Macs, Ryzen AI Max and DGX Spark

Apple silicon Macs, AMD’s Ryzen AI Max chips and NVIDIA’s DGX Spark share one pool of memory between the CPU and GPU. That makes big models fit cheaply. We count about 75% of a 128GB machine, or about 96GB, as usable for the model.

The catch is bandwidth. Generation speed depends on how fast the chip can read the weights, and unified memory is slower than high-end GDDR7. By our estimates, gpt-oss-120b runs at about 64 tok/s on a Mac Studio M5 Max 128GB (614 GB/s), about 35 tok/s on a Ryzen AI Max+ 395 (256 GB/s), and about 172 tok/s on an RTX PRO 6000 Blackwell (1,792 GB/s). Dense models suffer more: Qwen3.8 27B at Q8_0 runs at about 7 tok/s on the Ryzen AI Max+ 395. Our Mac and mini PC guide goes deeper on these machines.

What fits at each VRAM tier

All speeds below are our estimates for one user generating text at 8K context. “Tight” means the model fits but uses more than 88% of memory, leaving little room for longer prompts.

Two things to keep in mind with multiple GPUs. First, we count 97% of the combined memory as usable, because splitting a model duplicates some buffers, so two 24GB cards give about 46.6GB. Second, splitting by layer adds capacity but not speed. The cards take turns working through the layers, so two 3090s are no faster than one on a model that fits on one card, and usually a little slower.

Above 96GB you’re into server territory. DeepSeek V4 Flash needs about 168GB at its native FP4, and GLM-5.3 needs about 466GB at Q4_K_M. Those call for multi-GPU servers or 256GB to 512GB Mac Studios.

Checking what you’re actually using

Estimates get you to the right card; real usage varies with the runtime version, batch size and how many requests you serve at once. On NVIDIA cards, nvidia-smi shows memory in use per GPU. llama.cpp prints the size of each buffer when it loads a model, and Ollama’s ollama ps shows the total size and the CPU and GPU split.

If you’re buying hardware, size it for the model you want next year rather than the one you run today. Models keep getting more capable at the same size, but context windows and agent workloads keep pushing memory use up. Our GPU buying guide turns these tiers into specific picks, and each GPU page has a best-models list, like this one for the RTX 3090.