Every assumption behind the interactive explorer, in one place. The explorer states each figure; this page says where it comes from and how far to trust it.
A request that hits a warm session — one whose KV cache and recurrent state are still resident — prefills only its new turn, and is served well and comfortably. A cold request re-prefills its whole context on the compute roofline, briefly stealing bandwidth from every active user. Deployments are therefore sized by warm sessions, planning on p5: 95% of Monte-Carlo draws hold at least that many.
That hypothesis is no longer just an assertion. The Max cold req/s and Cold-spike tolerance readouts price it on prefill's own (compute) roofline, in a second pair of units — a work rate and a burst size — that the session count cannot express: a miss costs 18–19× the machine time of a hit, near-identically across the four architectures priced in docs/scenarios.md § 8 (DeepSeek-V4-Flash postdates that table).
A transparent memory model, calibrated to a 1×H200 + 27B FP8 pool of 2.77M KV tokens — a figure projected from the baseline's measured FP16 pool (~1.337M tokens, ×2 for FP8 plus freed activation memory), not itself a direct measurement. The per-GPU activation reserve is solved from that anchor, so every other configuration falls out of the same arithmetic rather than being guessed.
A real 27B + FP16-KV + TP2 bring-up reported 3,233,564 KV tokens at startup (110.59 GiB per worker; 17.54× concurrency at 184,320 tokens) — within 0.3% of this model's 3.24M prediction, with nothing about TP2 or FP16 fitted. First non-circular validation of the reserve + TP arithmetic + FP16 doubling. Run backwards it pins the per-GPU reserve at 18.24 GiB against the anchor-solved 17.98 GiB, i.e. an equivalent 1×H200 FP8 anchor of 2.762M tokens against the projected 2.77M (the ~1.4% residual is left in the constants rather than re-anchored: it would move every published figure by under 0.5%). What it still does not settle: the recurrent-state dtype, the other four models, and — since a globally wrong anchor shifts prediction and measurement together — the FP8 path itself.
Retired 2026-07-29: the study used to carry a low calibration anchor as an adverse case — 2× the measured FP16 lower bound (2.278M tokens ⇒ a 33.0 GiB per-GPU reserve). The TP2 measurement refutes it outright: it predicts a 2.750M-token pool where 3,233,564 were reported (−14.9%), against −0.26% for the anchor in use. An adverse case has to be one the hardware has not already ruled out.
Capacity is estimated by simulation over a user+subagent prompt-length mixture, not solved in closed form; every resident session is also charged its constant recurrent state. Draws are seeded from the configuration (since 2026-08-03), so the same configuration always renders the same numbers, and the load/latency controls — which cannot affect capacity — leave every capacity figure untouched. Two things still move it legitimately: neighbouring slider positions draw a different sample (a few sessions of spread is sampling noise, not a real effect), and while a slider is being dragged the page renders a lower-iteration draft, which can land a session or two from the settled value. The Python sweeps in tables.py use fixed seeds throughout. Real non-linearities (the expert-union kink at n=32, the cap's tail-truncation, the prefix floor) are explained in docs/scenarios.md. Decode speed is per-step bandwidth ÷ (weights + live KV + per-sequence state read/write), sped up by MTP.
TP (tensor-parallel, any N) shares one weight copy across all GPUs, so the pooled KV cache grows faster than N×; effective decode bandwidth scales by N × 0.90^log₂N (the baseline's 0.90 haircut applied per doubling — beyond 2 GPUs this is a projection, not a measurement). Widening TP always raises total capacity, because every DP group re-pays for its own full copy of the weights.
DP (data-parallel, any N) keeps N per-replica caches. Aggregate throughput scales ~N×, but a returning user is only warm on their home replica (sticky routing), and a shared CPU-offload buffer splits N ways.
MoE decode reads only the active experts: weight bytes per step grow with the batch's expert union, from the shared block up to all routed experts. The explorer uses the conservative no-overlap union bound (the docs also show the optimistic expected-union bracket). Past the point where every routed expert is read each step — n = 32 on the 35B-A3B — the fixed weight read stops growing with the batch — the slope break the explorer's chart D shows.
All constants assume the FP8 KV cache (--kv-cache-dtype fp8_e4m3), as tested in the baseline. The FP16 switch doubles KV bytes/token (half the pool, heavier decode reads); weights and the recurrent state are unaffected, and the activation-reserve calibration stays pinned to the FP8 anchor. Sanity: FP16 + 27B + 1×H200 gives a 1.39M-token pool, inside the baseline's measured [1.14M, 1.40M]. GLM-5.2's sparse-MLA path and DeepSeek-V4-Flash's V4 path both require a quantized (FP8) KV cache in vLLM, so the FP16 toggle is disabled there.
A session offloaded to host RAM still counts as warm capacity, but only sessions whose KV is resident in GPU HBM can decode — an offloaded one must be restored over PCIe first (a latency the model does not price). Every decode-side readout (the per-user/aggregate stress tiles, the explorer's chart C capacity zone) is therefore computed over HBM-resident warm sessions only, so moving the offload slider never changes per-user decode speed. For DP the buffer splits across replicas.
Cold / invalidating requests occupy KV but never count as warm — they cap the reusable-hit ceiling at (1−f). Past ~25% invalidation about a third of the pool is paying for sessions that will never be reused.
Other projections carry their own flags where they appear; these two are memory-accounting assumptions with no measurement behind them at all, so they are separate, named controls rather than a bundle.
max_num_seqs rather than per-session. fp32 would double the charge and costs the Qwens ~10% of warm capacity — the largest un-measured structural knob on those models. Mistral-3.5 and GLM-5.2 hold no recurrent state (the control is disabled); DeepSeek-V4-Flash pays a fixed 14.9 MiB/session of attention windows and fp32 compressor buffers whose precision is set by the serving stack, so it has no knob to turn either.w_resident figures are raw or on-disk checkpoint bytes, i.e. under-estimates by an unknown margin; the one calibrated data point is the 27B's as-deployed 28.8 GiB running ~15% above its raw params. Transferring that ratio to a different architecture and quantization recipe is an extrapolation — which is why it is a knob rather than a constant. The knob is disabled on the 27B itself, where the figure is already as-deployed and applying +15% would double-count.The slider sets the speculative-decode speedup (1.0× = off). The base quantity is the per-draft acceptance α: with 2 draft tokens, speedup = 1 + α + α², so 1.7× ⇔ α ≈ 47% — the 27B's measured fit (a measured 87% acceptance would be ≈ 2.6×). The inversion is exact for the Qwens only: GLM-5.2's MTP drafts 5 tokens, DeepSeek-V4-Flash drafts 7 (DSpark), and Mistral-Medium-3.5 ships no MTP module at all (its slider models an external EAGLE-style draft, unmeasured). The slider applies to the selected configuration only — the frontier table prices every row at its own model's capability, so cross-model ranking is never moved by this control. MTP + hybrid-model prefix caching is an immature serving path, so the conservative purchasing view keeps it at 1.0×; the purchasing base likewise excludes CPU offload and N>2 TP. These controls exist so that headroom stays visible, not hidden.
Both request classes draw log-normal lengths. The user class is fitted by maximum likelihood on 1,850 real request lengths (median 31k, σ = 0.81): ~68% of prompts within ×/÷ e^σ ≈ 2.25 of the median, p95 ≈ 3.8× the median, mean ≈ 1.39× the median. The mean — not the median — drives warm capacity, so a fatter tail of huge pool-hogging sessions means fewer warm. The subagent class is assumed (median 8k, σ = 0.90; no subagent trace has been collected yet). Lengths above the max_seq_len cap are truncated to it — a lower cap trims the tail (smaller worst-case cold prefill, less pool hogging), and a median far past the cap degenerates toward all-sessions-at-cap.
The subagent ratio r is subagent requests per user request, so subagents are r/(1+r) of sampled sessions; they draw much shorter prompts behind their own lean 3k prefix (or reuse the user prefix, if that toggle is on — no extra reserved block, but each subagent then carries the bigger base). The system / shared prefix is the stable block every user request shares — system prompt, skills, rules, tool definitions. It is reserved once per cache and deduplicated across every session, so a bigger shared prefix costs the pool once and saves it on every session: raising it raises warm capacity. The reference workload uses 15k; the baseline's real agent already exceeded 30k. This only pays off while the prefix is byte-stable across users and turns — a per-tenant salt or an edited tool list turns it into per-session bytes, which is what the invalidation control prices.
The Concurrent-users control is what converts every ceiling into the same unit: one user is assumed to hold one session (so the count compares against warm capacity) and to send a turn every think-time seconds (so it compares against a work rate). The reference load is 64 users at one turn per 30 s = 2.13 req/s, and at those defaults every ceiling in the explorer reproduces the published tables within Monte-Carlo noise. Think time is the full inter-request interval — it includes the previous response's prefill and decode, not idle time on top: the arrival model is open-loop (rate = users / think), so the system's own latency never feeds back into when the next turn arrives. A closed-loop user would slow down when responses do, which makes the open-loop reading conservative on the load-derived ceilings. Both conversions are assumptions, not facts — a user running several concurrent sessions divides the cache ceiling, and bursty think time invalidates the Poisson arrival model. Two readouts are deliberately priced elsewhere: the ITL spike and the tokens-lost figure use the stress point shown in the per-user tile — every GPU-resident p5-warm session decoding — not the slider's load. A warm hit still prefills its turn size (default 2,000 tokens) — a bigger lever than it looks: at low miss rates the hit leg dominates the duty cycle, so turn size sets the standing load a burst drains into (500 → 8,000 tokens moves B* by ~3.5× on the 27B/TP2).
The one readout priced in FLOPs, not HBM bytes (research/prefill.md). A miss re-prefills its whole context in max_num_batched_tokens chunks (GEMM + the quadratic attention term) against the part's dense FP8 peak; every readout is pinned to vLLM's 32,768 default, where the effective MFU is the calibrated 45% — the soft input: the plausible [30–60%] bracket spans 2× end to end. The headline is 1/(expected miss cost), priced on the context distribution's heavy tail (E[L²] — the quadratic attention term must not be priced at the mean length); the p5–p95 sub-line is the cost spread of a single miss. It is a per-replica-group ceiling that no KV pool, CPU offload or warm headroom can raise. f* is the miss rate that saturates the group at its share of the current load (users / think time, warm turns included; a DP grid splits that load across replicas).
Every chunk is charged its attention over the cache it lands on, so a miss's FLOPs are chunk-size invariant (the pair count telescopes) — but each of its passes also streams the resident weights, an overhead that multiplies by the pass count, so its total machine time is not: that per-pass roofline is what makes the effective MFU fall at small chunk sizes. Small chunks shrink the ITL spike every decoder sees (the spike prices the chunk's marginal FLOPs — its host pass streams the weights anyway) but sink MFU and the cold-request ceiling with it: that is why vLLM's default is 32k, not 2k, and why the explorer sweeps the knob in a chart instead of exposing it as a slider. A warm hit pays its new turn's attention over the whole cached context at the marginal rate (its small pass rides the serving steady state).
Max cold req/s and f* are a duty cycle — a mean rate against a mean service time — and a mean sees neither variance nor correlation. Both bite. A miss's service time runs as L² on a log-normal L, so its squared coefficient of variation lands at 5.5–8.3 where an exponential would sit at 1, and the M/G/1 (Pollaczek–Khinchine) wait diverges well below f*: the latency ceiling f_sla — the miss rate whose mean TTFT reaches the TTFT budget — binds at 0.35–0.93× f* (tightest on the most prefill-fragile config, Mistral-3.5/TP4), with the duty cycle still reading a comfortable 76–93%. And invalidation arrives in clumps, so B* prices the largest simultaneous burst of misses whose last request still gets a first token in budget; it is linear in the budget (a 5 s target halves every number, changing no ranking) and reaches zero exactly at f* — which is what makes f* a limit rather than an operating point. A global flush puts the machine at f = 100% until sessions re-warm; above 100% duty there is no steady state and recovery is set by admission control, unmodelled here. One queue per replica group: a DP burst spreads across replicas only as well as the router balances it, which the sticky routing DP needs for cache reasons works against.
Mean TTFT comes from the same M/G/1 queue, solved against the mean — a p95 budget binds at a lower miss rate than anything shown. The sharpest consequence is on the requests that hit: under first-come-first-served a warm hit waits behind whatever misses are in front of it, so the cache-miss rate is a latency parameter for the hitting users too — at a 20% miss rate on the 27B/TP2 a hit's TTFT reaches 74× its own service time. Processor sharing is the other end of the bracket (vLLM admits in arrival order but runs several admitted prefills concurrently); neither end is uniformly optimistic, and which is which flips by request class.
A backlog drains at (1 − duty) seconds of work per second, because the standing traffic keeps arriving throughout — so the last request's TTFT is the drain time under either scheduling discipline. During the drain the scheduler has a prefill chunk to place in every forward pass, so the ITL spike is the steady state for the whole drain: the warm users who did nothing wrong simply stop receiving tokens at their normal rate, and the tokens-lost figure is that difference integrated over the drain. The "full flush" burst preset sets the burst to the p5 warm population of one replica group, capped at the slider's 512 — the correlated event a prompt-template deploy actually is.
Every one of these makes a real machine worse than the model: kernel-launch/scheduler per-pass costs (the small-chunk end of the explorer's chart E reads better than a real machine), Poisson arrivals (real agentic traffic is burstier), solving against mean rather than percentile TTFT, preemption/recompute under a full KV pool, the decode batch stretching a drain 1–3%, and PCIe restore latency for offloaded sessions. NVFP4 checkpoints are charged the FP8 tensor rate — their mixed W4A4/FP8/BF16 recipes could really run faster or slower, so for NVFP4 configs the prefill figures are a modeling choice, not a bound.
The H200 (141 GB HBM3e, 4.8 TB/s) is the calibrated baseline. The B300 (Blackwell Ultra: 288 GB HBM3e, 8 TB/s, native FP4) reuses the H200-solved ~18 GiB per-GPU reserve, plus a measured +9.75 GB/GPU correction: the H200 actually delivers ~150.75 GB usable against its 141 GB vendor figure (the calibration silently absorbs that margin), while a real B300 nvidia-smi dump shows 275,040 MiB = 288.4 GB — nominal bytes, no Hopper-style over-provision — so the transferred reserve must add the hidden margin back (research/gpu_b300.md; formerly a sensitivity, measured 2026-07-27). Both parts are modelled as 8-GPU NVLink domains, so a deployment is one node at most.
Swaps in real NVFP4-checkpoint byte counts (4.5 bits/param on quantized tensors; what stays high-precision is recipe-specific — see research/nvfp4.md — e.g. RedHatAI's 35B-A3B keeps the DeltaNet blocks BF16 while NVIDIA's GLM recipe quantizes only the routed experts). Gated to native-FP4 GPUs: vLLM's Hopper fallback is weight-only Marlin with a known correctness bug (open as of 2026-07-27), so H-generation NVFP4 is deliberately not modelled. The KV cache is never 4-bit here — an owner policy: vLLM's nvfp4 KV shipped 2026-05 (Blackwell-datacenter-only) but is not modelled. Non-obvious: on the MoE models NVFP4 makes the fixed per-step read heavier (BF16-kept blocks) while expert reads shrink 1.78× — low-concurrency decode slows, high-concurrency speeds up. DeepSeek-V4-Flash has no NVFP4 variant at all — its experts ship natively FP4, and the only conversion is larger than the original.
research/model_35ba3b.md. 262k native, 1M via YaRN.research/model_mistral_medium35.md.research/model_glm52.md.research/model_dsv4flash.md.The B300 / NVFP4 / Mistral-3.5 / GLM-5.2 constants were researched 2026-07-27 with HuggingFace and NVIDIA domains proxy-blocked — chains run through first-party GitHub repos, three-way config mirrors, and cross-checked snippets, with an assumptions / re-verification ledger in each research note and per-claim confidence tiers in nvfp4.md and gpu_b300.md. Strongest cross-check: the derived GLM-5.2 NVFP4 resident matches the vLLM recipe's ~465 GB within 0.05%.
Mirrors scripts/scenario_model.py; constants live in the explorer's CONFIG block. Full write-up: docs/writeup.md and docs/scenarios.md in the repository. Research notes: research/model_35ba3b.md, model_mistral_medium35.md, model_glm52.md, model_dsv4flash.md, gpu_b300.md, nvfp4.md, prefill.md, spike.md.