2× RTX PRO 6000 Blackwell Self-hosted LLM Inference: A Complete performance and economics overview

We recently started offering servers with 2× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. 96 GB of GDDR7 per GPU, 192 GB in total, in an ordinary air-cooled rack server with no need for liquid cooling. Before you put one in production (or in your closet), you probably want to know what it can actually do with modern LLMs.

We tested 12 model families, from 9B to 122B parameters, across more than 85 configurations and 435 benchmark runs. That covered dense and mixture-of-experts (MoE) models, BF16, FP8, NVFP4 and INT4 quantizations, tensor-parallel, data-parallel and expert-parallel deployments, with and without speculative decoding, all through a standardized vLLM benchmark suite. Every run used the same server hardware configuration that’s available from us.

This post summarizes what we learned. The headline numbers, the surprises, the traps… and what any of it means if you're deciding to run it yourself.

Goals (and non-goals)

We wanted to compare single- and dual-GPU performance and understand the economics of self-hosting.

Goals:

Learn what to expect when it comes to LLM performance on the RTX PRO 6000 in single- and dual-GPU configurations, across various model sizes, architectures, quantizations and configurations.
Get a sense of the economics of self-hosting an LLM.

Explicit non-goals:

This is neither a deployment nor a configuration guide. vLLM, SGLang, llama.cpp and friends are each their own rabbit hole. As we said, we used vLLM because the author knows it best; your mileage with other engines may differ.
Nor is this article an attempt to tune every model to its limit. The joy of squeezing out maximum performance is left to the reader.

Token economics and self-hosted LLM inference

In order to make an informed economic decision, you should have at least a rough idea of the following numbers:

Prompt processing (prefill) is how quickly the model processes the input, measured in tokens per second. (Before the model can output anything, it has to chew through the text you sent it.)
Token generation (decode) is how quickly the model spits out new tokens.
KV cache capacity – whatever GPU memory is left over after the model weights are loaded, is used to hold the context of every active conversation. It determines how many concurrent users it can hold and how much context per user it can hold. We'll come back to this after the throughput numbers (See How many users fit? section).
Ratio of useful tokens – can the given model help solve your tasks. Generating tokens at light speed is useless if the model is brain-dead. At the same time, big, powerful, slow models might be unnecessary, with some effort on your side (harness, prompts).

You should also have an idea of how they relate to each other when selecting suitable hardware. The model we choose to run on this hardware needs to process / generate enough tokens per month to bring the per-token cost down.

Whether running a big frontier model at $100 per 1M output tokens is worth it depends on your task, but probably isn’t.

The challenge of achieving high token throughput, as you will see in the benchmarks, is concurrency:

Single session / user won’t saturate the GPUs - to achieve the high throughput numbers, concurrency is needed.
To serve many concurrent sessions, a lot of KV cache is needed.
To have KV cache headroom for many sessions, enough GPU memory must be available after the model weights are loaded.

All this to say a suitable model that can be operated at reasonable per-token cost will require more hardware than you might expect. It might even be a model that fits on your gaming GPU at home, but now scalable.

To make matters even worse, prompt processing and token generation steps have very different performance profiles and interrupt each other - GPU can process only so many tokens in a batch, and prompts, processed in parallel, take most or all of the batch as compared to token-by-token generation across multiple sequences. This manifests as a “choppy” experience to the users - each user’s new prompt or agent’s file read interrupts everybody else’s token generation, bringing overall throughput numbers down.

The “solution” to this problem is to deploy the thing twice - P/D disaggregation and is outside of the scope of this article. Please note however, it is the only way to get close to the reported synthetic benchmark numbers for general (mixed) workloads, such as coding agents.

If your use case is skewed, let’s say “prompt-heavy” – for example, throwing an email at the LLM asking “Is this spam?” with “yes” or “no” answers – you will mostly care about prompt processing speeds.

How we tested

The benchmarks used the vLLM’s own bench sweep tooling random-token prompts (See vLLM benchmarking docs) with following configurations, that allow us to grasp the performance under distinct regimes:

Single-request decode: sequential, 1024 input tokens, 4096 output tokens – What you will see as a single user; affects for example how long a single agentic task will take.
Concurrent decode: 1000 requests, 128 input tokens, 1024 output tokens – Ceiling under concurrent, decode-heavy load, across many users.
Prefill: 100 requests, 4096 input tokens, 1 output token – Affects how long you wait for a first token after sending a long prompt.

The environment:

Linux 6.18.38
CUDA 13.0, driver 580.126.20
vLLM 0.25.1, 0.26.0 for the DeepSeek-V4-Flash and DFlash, 0.28.0 for Qwen3.8-27B, and a qwen38-flash-next dev build for Qwen3.8-Flash-Next

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.20             Driver Version: 580.126.20     CUDA Version: 13.0     |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:01:00.0 Off |                    0 |
| N/A   46C    P0            161W /  600W |           92351MiB /  97887MiB |    100%      |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:E1:00.0 Off |                    0 |
| N/A   45C    P0            160W /  600W |           92365MiB /  97887MiB |    100%      |
+-----------------------------------------+------------------------+----------------------+

Headline chart results

The cross-model picture, feat. every model in its best-performing configuration:

Key observations:

Dense vs. MoE architecture - while dense models are usually smarter, more reliable, the huge speed gains usually make up for the marginal difference in accuracy.
Speculative decoding greatly increases single-request decode speeds, but costs a lot of KV cache, especially larger DFlash or DSpark drafters and adds overhead with higher concurrency.
Expert Parallelism bumps total throughput but eats into KV cache.
Tensor Parallelism over PCIe primarily gives more KV capacity, but might negatively impact throughput if the communication overhead is large enough - depending on the model (activation) size and its architecture

Things this table tells you that the throughput table doesn't

Quantization buys capacity twice. Smaller weights leave more room for KV. On a single GPU, Qwen3.6-35B-A3B in BF16 leaves 19 GB (~1.0 M tokens); in NVFP4 it leaves 60 GB (~2.7 M tokens). On two GPUs it's the difference between 5.2 M and 7.4 M tokens.
Architecture can make a model, regardless of its speed, impractical. DeepSeek-V4-Flash posts respectable throughput, but at ~55 KB/token it holds barely three 128K sessions. On this hardware, it's built for short-context traffic.
Gemma4-31B pays an eye-watering ~140 KB/token, and even with 152 GB of free KV it only reaches 1.2 M.
KV exhaustion is what "falling over" looks like. The Gemma4-31B BF16 single-GPU run failed 36% of our 1,000-request decode benchmark. It needed ~1.2 M tokens of KV and had 206k available. The engine spent the run preempting and recomputing.
Nemotron is the capacity king. ~5.5 KB per token plus small NVFP4 weights add up to 17.8 M tokens: 135 concurrent 128K-context sessions on a single two-GPU node.

The best fit to date: Qwen3.8

*The model does not know much, but is smart enough to go find out and not lie about it.*

The reasoning is not raw throughput, but rather what you get per unit of capability: a dense Qwen3.8-27B is a very capable model for its size, and this node serves it at ~4,000 tok/s aggregate decode, ~15,700 tok/s prefill, 180 tok/s single-stream with MTP, 2.3 M tokens of KV cache.

If you're picking one general-purpose model on one or two RTX PRO 6000s, this is the way.

Qwen3.8-27B

What the matrix says

TP2+NVFP4 is the default recipe. The 2nd GPU doubles aggregate decode from 1,490 → 3,210 tok/s and triples KV capacity. NVFP4 wins outright every column: prefill +46% over BF16, concurrent decode +23%, single-stream +80%. It also leaves the most room for cache. As with the MoE models, there’s no throughput case for BF16 here.

MTP is the interactive switch. Three speculative tokens take single-stream decode from 45 → 112 tok/s in BF16 and from 81 → 180 tok/s in NVFP4 at a ~25% aggregate-throughput cost.

DFlash2 turns the 27B into a coding buddy. With a dedicated draft model and 7 speculative tokens, single-stream hits 243 tok/s in BF16 and 360 tok/s in NVFP4. Aggregate decode roughly halves. Recommended for lower-concurrency deployments.

One GPU is a legitimate deployment. A single RTX PRO 6000 runs the NVFP4 27B at 2,710 tok/s of aggregate decode with 0.96 M tokens of cache, and, amusingly, at higher prefill throughput than TP2 (17,100 vs. 15,700 tok/s), since two-GPU prefill pays the PCIe all-reduce tax. So TP2 wins where it matters for a shared endpoint: decode and capacity.

Qwen3.8-Flash-Next

A next-generation Qwen model with various architecture improvements:

KV cache costs ~26 KB per token instead of the 27B's ~65 KB, so context is 2.5× cheaper per GB.
Knowledge as ngram tables that can be (and were) offloaded to CPU RAM.

It ships FP8-quantized, keeps the 256K context window.

Flash-Next matches the 27B's prefill and lands ~20% below its concurrent decode. MTP, though, is a poor trade on this architecture: aggregate decode halves (3,130 → 1,540 tok/s) for a modest single-stream gain (94 → 127). Unlike on the 27B, we can't recommend it.

How many users fit

KV capacity per configuration as reported by vLLM:

What it costs

The formula = monthly cost ÷ (throughput × seconds per month) – at the illustrative $2,800/month:

The recommended configuration lands at $0.27 per 1M output tokens at full utilization, around $0.90 at a realistic 30% – for a dense, 256K-context, 27B-class model. Prompt tokens cost $0.07 per 1M.
That's roughly 3× the 35B MoE's $0.09 – the price of dense-model quality, and still comfortably below typical API pricing for this model class.
Single-stream configurations are a different purchase: $3–6 per 1M at full utilization. You're not buying throughput, you're buying latency (and idle GPUs).

If we provisioned one of these nodes today: Qwen3.8-27B, TP2, NVFP4 as the shared endpoint; its MTP twin (or a single-GPU deployment, with the second card spared) for interactive work; Flash-Next for long-context and agentic traffic. The 35B MoE remains the answer when tokens-per-second-per-dollar is the only variable.

Reality check: synthetic benchmarks vs. real traffic

The tables above are best-case numbers from synthetic, homogeneous load. Real traffic is not homogeneous, and this is where naive capacity planning falls apart.

As mentioned in the beginning of this article, incoming requests (prefills) will interrupt token generation for all users.

The prefill performance really matters for user experience – for example, you can fit DeepSeek V4 Flash with 1M+ context on the two GPUs, with some manoeuvring, but chewing through 1M token request would take about 1m30s, during which every other user wonders why their agent stopped responding mid-sentence.

Here is what that looks like in practice: 16 concurrent coding sessions running against Qwen3.6-35B-A3B BF16 TP2 with MTP over about two hours of real usage.

Note that the real aggregate decode speeds are far from the corresponding multiple of single-request decode speeds we observed in benchmarks.

The same configuration that sustains 6,300+ tok/s under pure concurrent decode and ~350 tok/s on a single request, averages only ~240 tok/s of aggregate generation across these sessions – peaks barely reach 500 tok/s. Prompt processing interruptions on every request (agent step) bring the averages down dramatically.

The industry-standard solution is P/D (prefill/decode) disaggregation: deploy the model at least twice, one instance dedicated to prompt processing and one to token generation, with the KV cache handed off between them. Then tune the P:D ratio to your traffic – prompt-heavy workloads want more P instances, chat and code generation want more D. vLLM has first-class support for this, with transfer layers such as NIXL to move KV cache between instances. With 2× RTX PRO 6000 nodes, disaggregation pushes you toward smaller and/or quantized models – but, as the tables above show, that's a perfectly livable place to be.

Matching your hardware to use cases

Generalizing from our results, self-hosting strategies fall into three buckets:

1. Run the largest, most capable model possible, regardless of token speed. With llama.cpp's CPU+GPU hybrid inference for MoE models, large models like GLM-5.2 or Kimi K2.6 are possible at ~16 tok/s generation, which amounts to ~$100 per 1M output tokens. Prompt processing speeds, however, are horrendous. This is at best, a focused “deep-research” rig.

2. Run a hardware-adequate model for internal needs. Small (<50 B) or medium (100–300 B) parameter models, serving a few workflows or a handful of power users. This is the 2× RTX PRO 6000's comfort zone: one node, one engine, no disaggregation, Qwen3.6-35B-A3B-class models at 150–400 tok/s per session.

3. Run a hardware-adequate model for customer-facing or heavy agentic workloads. This is where time-to-first-token and token generation experience matters. Disaggregation and a lot of available KV capacity is a must. On 2× RTX PRO 6000 nodes this means smaller and/or quantized models per instance. Multiple nodes, each running a P/D pair of, say, NVFP4-quantized 35B-class MoEs, is a genuinely competitive building block.

The economics

Raw throughput only matters once you convert it to cost per token. The formula is simple:

# cost per 1M tokens = monthly server cost × 1,000,000 / (tokens per second × 2,592,000)

(2,592,000 = seconds in a 30-day month.) Illustratively, at $2,800/month for the 2× RTX PRO 6000 server, at 100% utilization:

Even the least efficient configuration lands in the same ballpark as, or below, typical API pricing for comparable models.

Note that the table shows 100% utilization, which nobody achieves. Real fleets with request gaps, idle periods and prefill/decode interference run at a fraction of synthetic peak – at ~30% effective utilization, so multiply the numbers by ~3.3 (e.g. the flagship NVFP4 setup lands around $0.30 per 1M tokens). Utilization, not hardware price, is the dominant term in self-hosting economics.

Key takeaways

If we had to compress all of the above into a few rules of thumb:

A model that runs also has to earn its keep. Loading the weights is only the first step; you also have to generate enough tokens per month to bring the cost per token down (See the utilization math in The economics section), or the value of the tokens you actually produce has to justify the outlay. Idle GPUs are the real expense.
Serving the fastest model that’s still useful usually beats serving the best one. Bigger and "smarter" only wins when you actually need the extra capability; for simpler, narrower use cases a smaller, faster model delivers the same outcome at a fraction of the per-token cost and latency. The same reasoning points us at hardware-adequate models (See Matching your hardware to use cases section).
You’re paying for task success, not tokens. What matters is the ratio of useful tokens — tokens that advance the task — to tokens generated, which means continuously evaluating the model and the harness around it on your real workloads. Throughput and latency tell you how fast money moves; only evaluation tells you whether it was well spent.