Which model, on which GPUs, with which parameters — for agentic coding on H200 / B300. The hypothesis, once: a request that hits a warm session prefills only its new turn and is served comfortably; a cold one re-prefills its whole context and briefly stalls everyone. Plan on warm p5.
Memory model calibrated to an FP8 anchor projected from the baseline's measured FP16 H200 pool, validated once against a real TP2 bring-up (within 0.3%); everything prefill-side is analytic and unvalidated. All assumptions, the full model & glossary →
Hardware, model and serving dtypes. These change most numbers on the page.
How many sessions stay warm — and whether the cache or the decode bandwidth runs out first. Memory only: nothing here depends on how often requests arrive.
Who is using it, and what their prompts look like.
A · Prompt-length distribution
Log-normal density, user vs subagent. Dashed line = the max_seq_len truncation cap.
B · Warm sessions (p5) vs hardware
The planning view: p5 warm sessions vs GPU count. Ring = your current configuration.
C · Per-user decode speed vs max_num_seqs
p50 line, p5–p95 band, log axis. Shaded = the GPU-resident warm-capacity zone; dashed = the 40 tok/s floor and 50 tok/s comfortable mark.
D · Aggregate throughput vs max_num_seqs
System p50 tok/s (×replicas for DP).
Adds work arriving: the request rate, the latency budget, and what cold traffic — steady or in a spike — costs.
How much traffic arrives, how cold it is, and how fast it has to be answered. These are the knobs that decide the table at the bottom of the page.
E · The max_num_batched_tokens trade
Why vLLM's prefill chunk is 32k, not 2k: small chunks shrink the ITL spike (bottom) but sink effective MFU (top) and the cold-request ceiling with it (middle). Dashed = the 32,768 default every tile is priced at.
F · Cold-spike tolerance
B* — the largest burst of simultaneous misses still answered in budget — vs the standing miss rate. Band = the MFU 30–60% bracket; each curve reaches zero at its own f*.
Every ceiling above, converted into one unit — max concurrent users — so the binding one is simply the smallest.
The deploy card — this configuration as a recipe
What to run and what to expect, from the current controls. Projections, not measurements — method & caveats.
All four ceilings, side by side
How many concurrent users each constraint allows. The shortest bar is the answer; the marker is your load.
G · The binding constraint vs cache-miss rate
All four ceilings vs the miss rate. The grey halo traces the lowest — what you actually get; the dot is your load (red if over), and the vertical marker is where the binding constraint first changes hands.
The frontier — every model × split on the selected GPU
Which configurations can serve your current load, sorted by max users; yours is highlighted. Each row is priced at its model's own speculative-decode capability (the MTP slider only affects the selected configuration). Figures are system-wide users, per-group in parentheses for DP.
Model, assumptions & glossary · Monte-Carlo model mirrored from scripts/scenario_model.py; constants in the CONFIG block.