Used Server GPUs for Local AI
The short version
Cheapest that works: the Tesla P100 at $150 for bandwidth, or the P40 at $345 for 24GB in one pool.
For 32GB: the V100 32GB at $910. The cheapest single card with 32GB, at a bandwidth nothing below it matches.
48GB on a budget: the Quadro RTX 8000 at $1,875. It is the cheapest 48GB card by a wide margin — $2,585 less than an A6000 for the same capacity.
48GB that just works: the A6000 at $4,460 if you need 48GB on CUDA with modern driver support. The MI210 gives you more for the money if you run ROCm.
Why price per GB stops being the right measure
The $150 P100 wins on cost per gigabyte every time, and at the bottom of the range that is the number to buy on: those cards are all old datacenter silicon, so the only question is how much memory you get per dollar.
Above about 32GB the number misleads you. Cost per GB rewards capacity you could have bought more cheaply somewhere else, and past 32GB there is nowhere else: two 16GB cards do not make a 32GB card, because a single model sees 16GB unless you shard it across both yourself, and most local inference stacks will not. The expensive tiers buy something the cheap ones cannot supply at any price.
The bands below are approximate. Used prices move, and every table is live.
Under $300
One model up to 14B, or 30B quantized hard.
| Card | VRAM | Price, Nov 3 | $/GB | Gen t/s |
|---|---|---|---|---|
| Tesla P100 | 16GB | $150 | $9.38 | ~55 |
| Tesla M40 | 24GB | $191 | $7.96 | ~22 |
| Tesla P40 | 24GB | $345 | $14.38 | ~26 |
$300 to $900
32GB in one pool, or HBM2 bandwidth at half the price of a 3090.
| Card | VRAM | Price, Nov 3 | $/GB | Gen t/s |
|---|---|---|---|---|
| Tesla V100 32GB | 32GB | $910 | $28.44 | ~68 |
| Quadro RTX 6000 | 24GB | $925 | $38.54 | ~50 |
| Titan RTX | 24GB | $1,000 | $41.67 | ~50 |
| Instinct MI100 | 32GB | $1,299 | $40.59 | ~92 |
$900 to $2,000
Modern drivers, display output, and the first affordable 48GB.
| Card | VRAM | Price, Nov 3 | $/GB | Gen t/s |
|---|---|---|---|---|
| RTX 3090 | 24GB | $1,500 | $62.50 | ~70 |
| Radeon Pro W7800 | 32GB | $1,699 | $53.09 | ~43 |
| Quadro RTX 8000 | 48GB | $1,875 | $39.06 | ~50 |
Over $2,000
48GB or more, on hardware current software expects.
| Card | VRAM | Price, Nov 3 | $/GB | Gen t/s |
|---|---|---|---|---|
| Radeon Pro W7900 | 48GB | $3,200 | $66.67 | ~65 |
| A40 | 48GB | $4,000 | $83.33 | ~52 |
| Instinct MI210 | 64GB | $4,299 | $67.17 | ~123 |
| RTX A6000 | 48GB | $4,460 | $92.92 | ~58 |
Choosing between the cheap three
The P100 at $150 is the most card per dollar anywhere on this page. 732 GB/s of HBM2 is two to three times what the GDDR5 cards manage, and bandwidth is what you feel while a model generates. The limit is 16GB: comfortable for 14B, tight for 30B.
The P40 at $345 trades that bandwidth for 24GB in one contiguous pool, on Pascal, which llama.cpp and most CUDA stacks still handle without argument. It is the default first 24GB card.
The M40 at $191 is the same 24GB for less. Maxwell is the catch: no fast native FP16 path and a generation further from current software. Right card if the budget is fixed and 24GB is not negotiable.
Not the K80
The K80 is the cheapest 24GB card you will find at $55, and you should not buy one. Its 24GB is two separate 12GB GPUs on one board, so a single model sees 12GB — $4.58 per usable GB, which is no longer remarkable. Kepler has also been dropped from current CUDA builds, so you inherit old drivers and old toolchains. The full argument is in Tesla K80 vs P40.
Getting to 32GB
The V100 32GB at $910 is the cheapest single card with 32GB in one pool, at 900 GB/s. It fits a 30B model at full weight, or a 70B quantized to Q2, without splitting anything across cards. The V100 16GB at $299 is the same speed for half the memory, and the better buy if your models fit.
The MI100 at $1,299 is faster than either on paper — 1,229 GB/s — and the cheapest 32GB card here. It needs ROCm, and on a card this old that means checking your exact framework before buying, not after.
Titan RTX and Quadro RTX 6000 are the same Turing silicon at about the same price, 24GB each. Both are actively cooled with display output, which is worth real money if you do not want to solve the cooling problem below.
Paying for drivers that already work
The RTX 3090 at $1,500 is the first card here that is simply a normal desktop GPU: 24GB, 936 GB/s, Ampere, active cooling, display output, no adapters. You pay roughly six times the P40's price for the same capacity, and what you get is everything working on the day it arrives.
The Quadro RTX 8000 at $1,875 is 48GB, the same as an A6000, for $39.06 per GB against the A6000's $92.92. Turing is two generations old and has no fast FP16 path worth using. If 48GB in one pool is the requirement and the A6000 is out of reach, this is the card.
The 48GB decision
The A6000 at $4,460 is the default recommendation at this level, and the one most buyers land on. 48GB, Ampere, CUDA, a workstation form factor with display outputs and active cooling. Nothing about it needs a workaround.
On paper the MI210 beats it: 64GB against 48GB, 1,638 GB/s against 768, for $4,299 — about the same money. The catch is ROCm. If your stack is CUDA-first, and most local inference stacks are, the A6000 is the card that works on arrival. Buy the MI210 if you already run ROCm and know your frameworks support it.
The W7900 at $3,200 sits between them: 48GB on RDNA3 with display outputs, cheaper than the A6000, and carrying the same ROCm question. The A40 at $4,000 is an A6000 without display output, built for a rack — worth it only if you are putting it in one.
What a server GPU costs beyond the price
Every Tesla card here is passively cooled and will throttle or shut down in a desktop case without help: they were built for a rack pushing air through them. A 3D-printed shroud and a 92mm fan runs about $15. The workstation cards — Titan RTX, Quadro, A6000, W7900 — bring their own fans, which is part of what the extra money buys.
- No display output on any Tesla card, the A40 or the MI210. You need a second GPU or integrated graphics to get a picture.
- EPS, not PCIe, power. Most datacenter cards use an 8-pin EPS connector rather than the PCIe 8-pin your power supply provides. The adapter is cheap but not optional, and plugging PCIe into EPS damages the card.
- Case depth and airflow. Full-length double-width cards in a case that was not planning for one. Measure first.
Our guide to buying used GPUs on eBay covers seller checks and what a fair listing looks like.
What to skip
SXM and SXM2 modules. You will see P100s, V100s and A100s listed at a fraction of the prices above. Those are mezzanine modules needing a proprietary server baseboard; they cannot go in a PCIe slot. Check the listing says PCIe.
Anything under 12GB. The T4 at $440 is here for its 70W draw, not its capacity. Below 12GB you are limited to 7B models quantized, and a used consumer card does that more easily.
Cards sold untested or for parts. Passive datacenter cards are often pulled from failed servers, and the discount rarely covers the risk.
Where these prices come from
Every figure here is the lowest used price we track on eBay US, refreshed twice daily. They move, sometimes sharply — the AI boom did unpleasant things to used GPU prices and has not finished. The comparison table carries the current numbers for every card.