NVIDIA GPUs on Linux for local AI: which cards to buy and how to set them up

If you’re going to run models on Linux for hours at a time, buy a card whose fans the NVIDIA driver can actually control. In my experience that means NVIDIA’s own Founders Edition cards, MSI and PNY. I learned this the hard way with two Gigabyte RTX 4090s whose fans sat at idle under Ubuntu until the cards got hot enough to shut the whole system down.

The software side is much easier than it used to be. On Ubuntu, one ubuntu-drivers command installs a working driver, NVIDIA’s open kernel modules are now the default for RTX 20 series and newer cards, and a few more commands get you to a model running in Ollama or llama.cpp. The second half of this guide walks through that setup step by step, with links to the official docs for each part.

Why Linux for local AI

Almost every GPU server running AI workloads runs Linux, so a Linux machine at home matches the environment that models, drivers and tools are built and tested on: the same software stack, CUDA libraries and performance assumptions.

When I first wrote this post I recommended Ubuntu 22.04 LTS, which nearly everyone running serious AI work used, from a single RTX 3060 12GB up to racks of B200s. It’s still supported, but for a new build in 2026 I’d start with 24.04 LTS. NVIDIA’s CUDA installation guide lists Ubuntu 22.04, 24.04 and 26.04 LTS as supported for CUDA 13.4.

I also said back then that I only used NVIDIA’s closed-source drivers. NVIDIA now ships open-source kernel modules as the default for modern cards (more on that below), while the user-space libraries that CUDA uses are still NVIDIA’s own. The community nouveau driver still isn’t an option for AI, because CUDA doesn’t run on it.

AI workloads are hard on cards

Gaming is a burst workload: GPU utilization swings between about 25% and 90%, with idle moments where temperatures spike and then cool off. Inference and training are sustained workloads:

• Utilization sits around 80 to 100%, with very little idle time
• Temperatures stay high for hours
• The whole card heat-soaks over long runs, which is why blower cards suit server-style builds

LLM inference also works the memory harder than most games, because every generated token reads the model’s weights out of VRAM. Memory cooling matters more than the “it’s just a chatbot” load suggests.

The fan-control problem

This is the issue that started this post. I bought two Gigabyte RTX 4090s that worked fine on Windows. On every version of Ubuntu I tried, I could use the GPUs but couldn’t control their fans. The fans sat at idle, and once the temperature climbed high enough, the system turned itself off. nvidia-smi saw both cards, but the fans seemed to be run by a secondary controller that needed Windows-only software.

One correction to what I wrote back then: nvidia-smi reports fan speed but doesn’t set it. Manual fan control on Linux goes through NVIDIA’s management library, NVML, which can set fan speed on Maxwell and newer cards with fans (NVML reference), or through nvidia-settings on a desktop. On a reference-design card, the driver’s automatic fan curve just works. On some partner cards, the driver can’t reach the fans at all, and no setting fixes that.

Before trusting a new card with long jobs, run a model for 20 to 30 minutes and watch nvidia-smi. If the fan percentage rises along with the temperature, you’re fine.

Which brands to buy

This list comes from my own experience, not a test of every model each brand makes, and partner cards can change between production runs. For anything not listed, search for reports of Linux fan control working on that exact model.

NVIDIA Founders Edition

NVIDIA’s own reference cards have the best driver support and the most reliable fan control on Linux: the RTX 3090, RTX 4090 and RTX 5090 Founders Editions. If you can get one, start here.

MSI, air-cooled and liquid-cooled

MSI is the partner brand I’d pick first: on the air-cooled side, the RTX 3090 Suprim X, RTX 4090 Gaming Slim and RTX 5090 Suprim SOC, and on the liquid-cooled side, the Suprim Liquid RTX 4090 and RTX 5090 if you have room for the radiator. I’d still avoid a used liquid-cooled 3090, because its pump is several years old by now.

PNY consumer cards

PNY’s GeForce cards make the list too: the XLR8 RTX 3090, the Verto RTX 4090 and the Epic-X ARGB RTX 5090.

PNY professional cards

This is where the original post was cut off. PNY builds and sells NVIDIA’s professional RTX cards in North America. They use NVIDIA’s own board and cooler designs, so they get the same first-party driver support as Founders Edition cards, and they’re built for sustained loads. The ones you’ll see in local AI builds:

• RTX A6000: 48GB on Ampere with a blower cooler, the workstation sibling of the 3090.
• RTX 6000 Ada: 48GB on Ada, also a blower.
• RTX 4000 Ada: 20GB in a single slot. PewDiePie’s rig uses a stack of them.
• RTX PRO 6000 Blackwell: 96GB. The 300W Max-Q version uses a blower and is what a16z’s four-GPU workstation runs.

They cost a lot more per gigabyte than GeForce cards. An RTX PRO 6000 Blackwell sold for about $16,000 in October 2026, and street prices for every card here have been volatile because of the memory shortage.

What I avoid

• Gigabyte, after my 4090 experience. Your card may work fine, but I haven’t gone back.
• OEM pulls of consumer cards, like the Dell version of the RTX 3090, which are built to the PC maker’s spec and don’t always behave like retail cards.
• Used cards with an AIO cooler that has years of pump wear on it.

Blower, open-air or liquid

Power and connectors

AI loads hold a card near its power limit for hours.

• RTX 3090: 350W. The Founders Edition uses NVIDIA’s 12-pin connector with an adapter; most partner cards use two or three 8-pin PCIe plugs. It’s known for sharp transient spikes.
• RTX 4090: 450W through a 16-pin 12VHPWR connector.
• RTX 5090: 575W through the updated 16-pin 12V-2x6 connector.

With the 16-pin connectors, push the plug all the way in and avoid bending the cable sharply at the connector. Size the power supply for every GPU’s full rating plus the CPU, with headroom for spikes; big rigs like The Machine, with twelve 3090s, need more than one supply. A power limit, covered in the last setup step, takes pressure off all of this.

Ubuntu setup, step by step

This works the same on a fresh Ubuntu LTS install with one card or several.

1. Install the driver

Ubuntu’s NVIDIA driver docs recommend the ubuntu-drivers tool, partly because it installs prebuilt, signed kernel modules that work with Secure Boot. Update and reboot first, then:

sudo ubuntu-drivers list
sudo ubuntu-drivers install
sudo reboot

That installs the driver Ubuntu considers the best match for your card. On a headless server, add --gpgpu to both commands to get the -server branch, and install the matching utilities package, for example nvidia-utils-580-server. cat /proc/driver/nvidia/version shows the running driver, and a “driver/library version mismatch” error after an update goes away with a reboot.

2. Open or proprietary kernel modules

NVIDIA made its open-source kernel modules the default starting with the R560 driver. In short:

• Blackwell cards (RTX 50 series, RTX PRO Blackwell) require the open modules.
• Turing, Ampere, Ada and Hopper (RTX 20, 30 and 40 series and their workstation siblings) should use the open modules.
• Maxwell, Pascal and Volta cards, like the Tesla P40 or a V100, need the proprietary modules. If you mix one of those with a newer card in the same machine, you’re on the proprietary modules too. NVIDIA has said R580 is the last driver branch for these older architectures.

In the ubuntu-drivers list output, packages ending in -open, like nvidia-driver-580-open, are the open modules, and the first line of /proc/driver/nvidia/version mentions “Open Kernel Module” when they’re loaded. NVIDIA’s own apt repository calls the open driver package nvidia-open. Pick one source for drivers and stick with it, because mixing the two causes conflicts.

3. Check it with nvidia-smi

Run nvidia-smi. You should see every card with its memory, temperature, power draw and fan speed. nvidia-smi -L lists the GPUs with their UUIDs.

The “CUDA Version” in the header confuses a lot of people. According to NVIDIA’s nvidia-smi docs, it’s the newest CUDA version your driver supports, not a toolkit you’ve installed. For long runs, this one-liner prints the numbers that matter every five seconds:

nvidia-smi --query-gpu=index,name,temperature.gpu,fan.speed,power.draw,memory.used --format=csv -l 5

4. Decide whether you need the CUDA toolkit

The driver is what runs models. The CUDA toolkit is the compiler (nvcc), headers and developer libraries, and you only need it to compile CUDA code, like building llama.cpp from source. Ollama’s Linux docs list the CUDA install as optional.

As of CUDA 13.4, the toolkit packages no longer install or depend on the driver on Linux, so the two are managed separately. CUDA 13.x needs a driver from the R580 branch or newer, per the CUDA release notes. The installation guide uses NVIDIA’s network repository; on Ubuntu 24.04 that looks like this:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit
echo 'export PATH=/usr/local/cuda-13.4/bin${PATH:+:${PATH}}' >> ~/.bashrc

Change ubuntu2404 to match your release and the version in the path to match the toolkit you installed, then open a new shell and check with nvcc --version.

5. Run a model

The quickest route is Ollama, using its official install script:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:20b

gpt-oss-20b needs about 15GB at 8K context, so it fits tightly on a 16GB card and comfortably on a 24GB one. In a second terminal, ollama ps should show “100% GPU”; anything else means part of the model is running from system RAM. Ollama defaults to a 4,096-token context. To raise it for the background service, run sudo systemctl edit ollama and add a line like Environment="OLLAMA_CONTEXT_LENGTH=32768" under [Service], as described in Ollama’s FAQ.

For the latest features and full control, build llama.cpp with CUDA, following the llama.cpp build docs. This needs the CUDA toolkit from step 4:

sudo apt install build-essential cmake git libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 32768 --port 8080

Then open http://localhost:8080 for the built-in chat page. Per the llama-server docs, it listens on 127.0.0.1 by default, works out GPU layers automatically, and splits the model by layer across all your cards. Set CUDA_VISIBLE_DEVICES=0,1 to choose which cards it uses, and check the VRAM calculator to see which models fit.

6. Set a power limit

This is the step I’d never skip on a machine that runs for hours. The nvidia-smi docs say the -pl power limit is set in watts, needs root, and has to fall between the card’s minimum and maximum limits:

nvidia-smi -q -d POWER
sudo nvidia-smi -pm 1
sudo nvidia-smi -i 0 -pl 280

The first command shows the current, default, minimum and maximum limits. The second turns on persistence mode, which keeps the driver loaded between jobs. The third caps GPU 0 at 280W; leave out -i 0 to apply it to every card. For a 3090, 280 to 300W is a sensible starting point; the dual 3090 build holds its cards at 300W in a 25-liter case. Generation is mostly limited by memory bandwidth, so a moderate cap usually costs less speed than you’d expect, but measure it with your own model.

Neither setting survives a reboot (NVIDIA’s docs note persistence mode resets to disabled every time), so put them in a small systemd service at /etc/systemd/system/nvidia-power.service:

[Unit]
Description=Set NVIDIA persistence mode and power limit

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 280

[Install]
WantedBy=multi-user.target

Enable it with sudo systemctl enable --now nvidia-power.service.

Where to go from here

If you’re still choosing hardware, my RTX 3090 guide covers the card I’d start most people on, and the GPU buying guide covers everything else. For finished Linux builds, the sleeper RTX 3090 and the GPUter RTX 5060 Ti are good single-card examples.