NVIDIA · Ampere · 2020
NVIDIA RTX A6000 for local AI
A proven 48GB Ampere workstation card with NVLink, now priced near its launch MSRP.
Updated · Speeds marked ~ are estimates; seemethodology
- VRAM
- 48GB
- Bandwidth
- 768 GB/s
- Power
- 300W
- Runs well
- 8 models
RTX A6000
Typically about $4,400 used
Can the RTX A6000 run LLMs?
The RTX A6000 has 48GB of GDDR6 ECC and 768 GB/s of memory bandwidth, which decides how big a model fits and how fast it generates text. It comfortably runs models up to about 35B parameters: the largest in our catalog is Qwen3.6 35B-A3B at Q8_0 (41GB with 8K context, ~79 tokens/sec). Our top all-round pick for it is Qwen3.8 27B (Q8_0, ~18 tokens/sec). It can even run Llama 3.3 70B Instruct at Q3_K_M (~16 tokens/sec).
Recommendations
Best models for the RTX A6000
Best overall · also coding, reasoning, vision
Qwen3.8 27BQ8_0 · 32GB · ~18 tokens/sec
Runs wellRunner-up · also largest that fits
Qwen3.6 35B-A3BQ8_0 · 41GB · ~79 tokens/sec
Runs wellFastest good model
gpt-oss-20bMXFP4 · 15GB · ~105 tokens/sec
Runs well
Compatibility
What runs on the RTX A6000
Every current model in our catalog, at the best quantization that fits in 48GB with 8K tokens of context. "Partial offload" means part of the model runs from system RAM (we assume 64GB), which is much slower.
| Model | Parameters | Best quant | Memory | Speed | Verdict |
|---|---|---|---|---|---|
| Gemma 4 E4B | 8B | Q8_0 | 10 GB | ~55 tok/s | Runs well |
| Qwen3.5 9B | 9B | Q8_0 | 12 GB | ~49 tok/s | Runs well |
| Phi-4 | 14.7B | Q8_0 | 18 GB | ~34 tok/s | Runs well |
| gpt-oss-20b | 20.9B (3.6B active) | MXFP4 | 15 GB | ~105 tok/s | Runs well |
| Gemma 4 26B A4B | 26B (3.8B active) | Q8_0 | 30 GB | ~68 tok/s | Runs well |
| Qwen3.8 27B | 27B | Q8_0 | 32 GB | ~18 tok/s | Runs well |
| Gemma 4 31B | 31B | Q8_0 | 36 GB | ~17 tok/s | Runs well |
| Qwen3.6 35B-A3B | 35B (3B active) | Q8_0 | 41 GB | ~79 tok/s | Runs well |
| GLM-4.5-Air | 106B (12B active) | Q4_K_M | 68 GB | ~14 tok/s | Partial offload |
| Llama 4 Scout | 109B (17B active) | Q4_K_M | 70 GB | ~10 tok/s | Partial offload |
| gpt-oss-120b | 117B (5.1B active) | MXFP4 | 68 GB | ~35 tok/s | Partial offload |
| Qwen3.5 122B-A10B | 122B (10B active) | Q4_K_M | 77 GB | ~15 tok/s | Partial offload |
| DeepSeek V4 Flash | 284B (13B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3-Flash | 320B (18B active) | — | — | — | Won't fitWhat it needs |
| GLM-5.3 | 744B (40B active) | — | — | — | Won't fitWhat it needs |
Performance
RTX A6000 vs. alternatives on Qwen3.8 27B
Generation speed in tokens per second for a single user. Token generation is limited mostly by memory bandwidth, which is why bandwidth predicts these results so well.
RTX 3090
Q4_K_M · 24GB
~39tok/s
RTX 6000 Ada
Q8_0 · 48GB
~25tok/s
RTX A6000
Q8_0 · 48GB
~18tok/s
R9700
Q6_K · 32GB
~18tok/s
Measured
RTX A6000 benchmark results
| Model | Quant | Runtime | Generation | Prompt | Source |
|---|---|---|---|---|---|
| Llama 2 7B Chat | Q4_0 | llama.cpp (CUDA) | 140 tok/s | 4915 tok/s | llama.cpp discussion #15013: Performance of llama.cpp on NVIDIA CUDA · October 2026 |
| Llama 3.1 8B Instruct | Q4_K_M | llama.cpp (CUDA) | 105 tok/s | 3780 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| Llama 3.1 8B Instruct | F16 | llama.cpp (CUDA) | 41 tok/s | 4510 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| Llama 3.3 70B Instruct | Q4_K_M | llama.cpp (CUDA) | 15 tok/s | 480 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
| 4× · Llama 3.3 70B Instruct | Q4_K_M | llama.cpp (CUDA) | 14 tok/s | 495 tok/s | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) · May 2024 |
Our take
Should you buy the RTX A6000 for AI?
The RTX A6000 has 48GB of ECC GDDR6 on a 384-bit bus for 768 GB/s in a 300W dual-slot blower card. It supports two-way NVLink bridges, and two cards hold 70B models at 8 bit. Ampere still has full CUDA support, including FlashAttention 2. Used cards sold for about $4,400 in October 2026, close to the original $4,650 MSRP, so check the RTX 6000 Ada and newer 32GB cards before buying.
What we like
- 48GB per card with ECC
- Two-way NVLink support
- 300W blower design for multi-GPU towers
Watch out for
- 768 GB/s is slower than GeForce cards with less memory
- Used prices close to launch MSRP
Best for: 48GB per slot for less than newer workstation cards · Dual-card 96GB workstations
Specifications
RTX A6000 specs that matter for AI
- Memory
- 48GB GDDR6 ECC
- Memory bandwidth
- 768 GB/s
- Memory bus
- 384-bit
- Architecture
- Ampere
- Power
- 300W
- Interface
- PCIe 4.0 x16
- Released
- 2020
- Launch price
- $4,650
- Street price (used)
- ~$4,400 (2026-10)
- Software
- CUDA, Vulkan
Sources: NVIDIA RTX A6000 product page
Cross-shop
Alternatives to the RTX A6000
Frequently asked questions
How much VRAM does the RTX A6000 have?
The NVIDIA RTX A6000 has 48GB of GDDR6 ECC with 768 GB/s of memory bandwidth on a 384-bit bus.
What is the largest LLM the RTX A6000 can run?
Fully in GPU memory, the largest model in our catalog that runs comfortably is Qwen3.6 35B-A3B (35B (3B active)) at Q8_0, using about 41GB with 8K context at ~79 tokens/sec.
Can the RTX A6000 run Llama 3.3 70B?
Yes. At Q3_K_M it needs about 39GB with 8K context and runs at ~16 tokens/sec.
What is the best LLM to run on the RTX A6000?
Our top pick is Qwen3.8 27B at Q8_0, which fits in 48GB with room for 8K of context and runs at ~18 tokens/sec. See our full list of the best models for the RTX A6000 for coding, reasoning, and vision picks.
Is the RTX A6000 good for local AI in 2026?
A proven 48GB Ampere workstation card with NVLink, now priced near its launch MSRP.