Which model, on which GPUs, with which parameters — for agentic coding on H200 / B300. The hypothesis, once: a request that hits a warm session prefills only its new turn and is served comfortably; a cold one re-prefills its whole context and briefly stalls everyone. Plan on warm p5.
Memory model calibrated to an FP8 anchor projected from the baseline's measured FP16 H200 pool, validated once against a real TP2 bring-up (within 0.3%); everything prefill-side is analytic and unvalidated. All assumptions, the full model & glossary →
Hardware, model and serving dtypes. These change most numbers on the page.
How many sessions stay warm — and whether the cache or the decode bandwidth runs out first. Memory only: nothing here depends on how often requests arrive.
Who is using it, and what their prompts look like.
A · Prompt-length distribution
Log-normal density, user vs subagent. Dashed line = the max_seq_len truncation cap.
B · Warm sessions (p5) vs hardware
The planning view: p5 warm sessions vs GPU count. Ring = your current configuration.
C · Per-user decode speed vs max_num_seqs
p50 line, p5–p95 band, log axis. Shaded = the GPU-resident warm-capacity zone; dashed = the 40 tok/s floor and 50 tok/s comfortable mark.
D · Aggregate throughput vs max_num_seqs
System p50 tok/s (×replicas for DP).
Adds work arriving: the request rate, the latency budget, and what cold traffic — steady or in a spike — costs. It also prices act 1's decode curve at the load you actually set — arrivals are open-loop, so far fewer sessions decode at any instant than sit warm, and each one runs much faster than the stress test above.
How much traffic arrives, how cold it is, and how fast it has to be answered. These are the knobs that decide the table at the bottom of the page.
max_num_seqs setting.E · The max_num_batched_tokens trade
A (re)prefilling request's chunks ride the decode batch's fused passes, and the pass duration is everyone's inter-token gap. Bottom: that gap ÷ the normal all-decoding gap. Middle: cold requests one replica group can prefill per second (1 / mean miss machine time). Top: the compute share of a miss pass. Small chunks shrink each freeze but pay the weight stream every pass — sinking MFU and the cold ceiling. Dashed = the study's 32,768 default; the dot marks the chunk every tile, the deploy recipe and the validation script are priced at (the same, unless a share link pins another). E2–E4 below take the three strips apart.
F · Cold-spike tolerance
B* — the largest burst of simultaneous misses still answered in budget — vs the standing miss rate. Band = the MFU 30–55% bracket; each curve reaches zero at its own f*.
E2 · Anatomy of one pass — the toll and the math
One miss-side pass = a fixed weight-stream toll (gray) + compute that grows with the chunk (blue: GEMM plus the chunk's own attention). The gray never changes — the chunk decides how much math amortises it, which is all chart E's top strip measures. Hover chart E or E3 for a live third bar; switch models to watch the gray flip from sliver (dense) to dominant (MoE at 2k).
E3 · Which models pay for small chunks
Chart E's middle strip for every model at once: each cold-request ceiling vs chunk, normalised to its own 32k value, at the selected GPUs and dtypes. Dense models barely care; a MoE streams its whole expert bank every pass, so the toll dwarfs the math at small chunks. Selected model = thick.
E4 · What a decoder feels — the token stream itself
Chart E's bottom strip as lived: one mean-length miss in a live token stream (ticks = tokens) at three chunk sizes, every waiting gap labelled in ms. The green total is identical in every row — the chunk sets the texture of the stall, not its sum. ▲ = the miss's own first token, which small chunks postpone: the decoders' comfort is paid by the cold request. "Worst" here is the miss's last full chunk (the most cache to attend over); chart E's spike prices its midpoint chunk, so the two differ a little at long contexts.
Every ceiling above, converted into one unit — max concurrent users — so the binding one is simply the smallest. Five beats: the verdict, how fragile it is, the recipe and its bill, the script that tests it, and every alternative ranked.
All four ceilings, side by side
How many concurrent users each constraint allows. The shortest bar is the answer; the marker is your load.
G · The binding constraint vs cache-miss rate
All four ceilings vs the miss rate. The grey halo traces the lowest — what you actually get; the dot is your load (red if over), and the vertical marker is where the binding constraint first changes hands.
Each bar sweeps one assumption across its plausible range with every other control held at your settings. Colour = which constraint binds there; the red underline marks where your load no longer fits. The ▼ is your current value, and the dashed tick is the nearest flip: the value where the bar changes colour (a different constraint takes over) or where the red underline starts or stops (the verdict itself changes) — the label on the right names which, in the flip’s own colour. Rows are sorted by how close that tick sits to your value.
The deploy card — this configuration as a recipe
What to run and what to expect, from the current controls. Projections, not measurements — method & caveats.
The bill: hardware and electricity
Two lines. Hardware is GPU-hours at the on-demand rental rate (the default is the cross-provider median for the selected part; owned hardware is whatever your depreciation says). Electricity is wall power priced from the duty cycle the model already computes — prefill bursts near the power cap, bandwidth-bound decode well under it, idle in between — plus the chassis and the facility overhead (PUE). GPU term ±20–25%; B300 figures are extrapolated from Hopper measurements — method & sources.
Every number on this page is a projection — the planning stance defines "comfortable capacity" as measured SLO capacity, which no model produces. This button hands out the configuration on screen as a workingset.toml, and the workingset package turns it into the load test: closed-loop users with the shared prefix and log-normal contexts configured above, a ladder bracketing the predicted limit, and a predicted-vs-measured verdict table. The predictions are not in the file — ws predict recomputes them from it, so a config can never carry a number the code did not produce. Start with --dry-run; probe B* with --burst N.
H · The frontier: what a seat costs against what the model can do
Every configuration that can carry your load as one dot: its Terminal-Bench score (one lab, one harness per version — Artificial Analysis; DeepSeek-V4.1-Flash's is its vendor card's until AA runs it) against what one seat costs a month with the configuration full — the bill at its max users, divided by them (hardware + electricity, at your prices). Priced full rather than at your load because at your load the hardware line of the bill is just the GPU count; at capacity a row that carries twice the users halves the seat. Dots stacked at one score are the same model on different splits — the column is what the topology choice costs. The staircase joins the configurations nothing beats on both counts: for any capability floor, the cheapest seat. Dots off it are dimmed because some other row scores at least as high for no more money — bills within 1% count as the same money, so a dearer dot can win on score alone.
The two Terminal-Bench versions disagree about this field. On 2.1 (89 tasks, Terminus 2 — Artificial Analysis's legacy eval) the eight models sit inside 46 points and the top four inside 7; on 4.0 (66 tasks, mini-SWE-agent v2.4.6 — the version carrying AA's Intelligence Index v4.3) the same eight spread over 42 points, GLM-5.3 leads Qwen3.8-Flash-Next by 17 rather than trailing it by 2, and two models score a measured 0.0%. Each version is its own staircase, so neither can stand in for the other; the table below prints both columns whichever is on the axis.
Which configurations can serve your current load, sorted by max users; yours is highlighted. Each row is priced at its model's own speculative-decode capability (the MTP slider only affects the selected configuration) and at the MBU slider's decode efficiency, one value for every row — measured on one deployment; nothing here measures a dense one. Figures are system-wide users, per-group in parentheses for DP. TB 2.1 and TB 4.0 are the model's Terminal-Bench pass@1 on the two versions as Artificial Analysis measures them (one harness per version for every model — Terminus 2 on 2.1, mini-SWE-agent v2.4.6 on 4.0; vendor cards are not comparable, and DeepSeek-V4.1-Flash's 90.6 / 31.2 are the one pair of vendor figures, carried until AA publishes) — a model property, so every split of a model shares it. Chart H's toggle picks which of the two is its axis; the emphasised column is the one in play. The €/mo column is the whole bill at your load — GPU-hours at your GPU price plus the electricity at your €/kWh — so two rows with the same headroom rarely bill the same; €/seat is that bill with the row full, per user it then carries (chart H's y-axis).
Model, assumptions & glossary · Monte-Carlo model mirrored from scripts/scenario_model.py; constants in the CONFIG block.