Capacity experiment

Can this deployment carry the headcount?

State the hypothesis, read the prediction, run the load test, then feed the measured efficiencies back in. Method and assumptions

01Hypothesis

Loading the model…

Model

GPU

GPUs 1× H200

Split (DP × TP)

Concurrent users 64 · 2.13 req/s

Headcount optional

Cold request waits at most p95 ≤ 5 s

Generation slowed at most 15% of the time

Per-user decode floor 40 tok/s

More assumptions

Weights, cache & speed

Weights

Deployed-weight overhead

KV cache dtype

Recurrent state dtype

KV cache across TP ranks

MTP decode speedup 2.94×

CPU offload 0 GiB

max_seq_len cap 180k

Workload

Prompt-length distribution

User prompt median 31k

User prompt sigma 0.81

Subagent prompt median 8k

Subagent prompt sigma 0.90

Request mixture & caching

Subagent ratio 0.10 · 1 per 10

Subagents share the user prefix

System / shared prefix 15k

Traffic & latency

Peak active share 100%

Sessions per active person 1.0

Think time 30 s/turn

Warm turn size 2,000 tok

Output per response 400 tok

Cold traffic

Cache-miss (invalidation) rate 1.0%

Cold burst size 32 at once presets

02Prediction

ID	Prediction	Role

03Test

Download the configuration, run it against your endpoint, and compare what it measures with the predictions above.

04Calibrate

Set the prefill and decode efficiency the test measured.

Prefill efficiency (MFU) 0.45

Decode efficiency (MBU) 0.22

Decode pricing

max_num_seqs unset

Verdict at these values

Where the verdict flips

05If it fails

Every configuration on this GPU, by capability and price per seat. Click a dot or a row to load it.

Price per seat against capability

chart axis

binds on cache decode cold wait slowed generation saturation efficient set dominated hollow: max users is a lower bound (decode search censored)

All configurations at your load

—Under the hood

Cache capacity

A · Prompt-length distribution

User Subagent

B · Warm sessions (p5) vs hardware

TP — one shared cache (band to p95) DP — system total (needs sticky routing)

C · Per-user decode speed vs max_num_seqs

p50 per-user p5–p95 capacity zone all warm decoding your load (steady state)

D · Aggregate throughput vs max_num_seqs

aggregate p50 all warm decoding your load (steady state)

Cold traffic and chunk size

E · The max_num_batched_tokens trade

F · Cold-spike tolerance

this configuration MFU 30–55% other topologies absorbs < 1 miss

I · Slowed generation against load

J · Cold request wait against load

Cold request, idle server

Prefill time of one cold prompt with nothing else running. A property of the deployment, not a limit.

Prompt	Prefill time

E2 · Anatomy of one pass — the toll and the math

weight-stream toll (fixed) compute (grows with chunk)

E3 · Which models pay for small chunks

dense MoE

E4 · What a decoder feels — the token stream itself

stream slowed behind a chunk a token arrives the miss's first token

All five ceilings

Max users under each constraint

G · The binding constraint vs cache-miss rate

cache decode cold wait slowed generation saturation your load (red if over)

The bill

Hardware and electricity per month

Electricity price €0.19 / kWh

GPU price €3.80 / GPU·h

Facility PUE

vLLM recipe

This configuration as a vLLM recipe

Method, assumptions and glossary