Model & method

The working hypothesis, and the number the study exists to produce

A request that hits a warm session — one whose KV cache and recurrent state are still resident — prefills only its new turn, and is served well and comfortably. A cold request re-prefills its whole context on the compute roofline, briefly stealing bandwidth from every active user. The pool is therefore measured in warm sessions, planning on p5: 95% of Monte-Carlo draws hold at least that many. That count is where the hit rate starts to fall, not a ceiling: past it the sessions the pool cannot keep warm miss, and the deployment runs out of prefill, not of cache (the hit rate at the load).

That hypothesis is no longer just an assertion. The Max cold req/s and Cold-spike tolerance readouts price it on prefill's own (compute) roofline, in a second pair of units — a work rate and a burst size — that the session count cannot express: a miss costs 18–19× the machine time of a hit, near-identically across the four architectures priced in docs/scenarios.md § 8 (DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next and GLM-5.3-Flash postdate that table).

Calibration & validation

A transparent memory model, calibrated to a 1×H200 + 27B FP8 pool of 2.77M KV tokens — a figure projected from the baseline's measured FP16 pool (~1.337M tokens, ×2 for FP8 plus freed activation memory), not itself a direct measurement. The per-GPU activation reserve is solved from that anchor, so every other configuration falls out of the same arithmetic rather than being guessed.

The measured cross-check (2026-07-22)

A real 27B + FP16-KV + TP2 bring-up reported 3,233,564 KV tokens at startup (110.59 GiB per worker; 17.54× concurrency at 184,320 tokens) — within 0.3% of this model's 3.24M prediction, with nothing about TP2 or FP16 fitted. First non-circular validation of the reserve + TP arithmetic + FP16 doubling. Run backwards it pins the per-GPU reserve at 18.24 GiB against the anchor-solved 17.98 GiB, i.e. an equivalent 1×H200 FP8 anchor of 2.762M tokens against the projected 2.77M (the ~1.4% residual is left in the constants rather than re-anchored: it would move every published figure by under 0.5%). What it still does not settle: the recurrent-state dtype, the other six models, and — since a globally wrong anchor shifts prediction and measurement together — the FP8 path itself.

Retired 2026-07-29: the study used to carry a low calibration anchor as an adverse case — 2× the measured FP16 lower bound (2.278M tokens ⇒ a 33.0 GiB per-GPU reserve). The TP2 measurement refutes it outright: it predicts a 2.750M-token pool where 3,233,564 were reported (−14.9%), against −0.26% for the anchor in use. An adverse case has to be one the hardware has not already ruled out.

What is measured, what is projected, what is analytic

• Measured: the baseline FP16 pool interval, the TP2 bring-up, the 27B's as-deployed 28.8 GiB weights, the B300's usable HBM, the user prompt-length fit.
• Projected from a measurement: the FP8 anchor, TP scaling beyond 2 GPUs, the +15% weight overhead on non-27B models, the draft acceptance carried to models without their own measurement.
• Analytic and unvalidated: the prefill/queueing axis (Max cold req/s, f*, B*, TTFT), the decode roofline, the subagent workload distribution.

The memory model

Monte-Carlo capacity

Capacity is estimated by simulation over a user+subagent prompt-length mixture, not solved in closed form; every resident session is also charged its constant recurrent state. Draws are seeded from the configuration (since 2026-08-03), so the same configuration always renders the same numbers, and the load/latency controls — which cannot affect capacity — leave every capacity figure untouched. Two things still move it legitimately: neighbouring slider positions draw a different sample (a few sessions of spread is sampling noise, not a real effect), and while a slider is being dragged the page renders a lower-iteration draft, which can land a session or two from the settled value. The latency figures share one seeded draw of 20,000 context lengths, where ws predict draws 200,000, so the cold-request wait, the long cold stretch and the latency ceilings can sit 1–2% from the package's figures for the same configuration. The formulas match: on a 1,000,000 draw the page agrees with Python. The Python sweeps in tables.py use fixed seeds throughout. Real non-linearities (the expert-union kink at n=32, the cap's tail-truncation, the prefix floor) are explained in docs/scenarios.md. Decode speed is per-step bandwidth ÷ (weights + live KV + per-sequence state read/write), sped up by MTP.

TP and DP

TP (tensor-parallel, any N) shares one weight copy across all GPUs, so the pooled KV cache grows faster than N×; effective decode bandwidth scales by N × 0.90^log₂N (the baseline's 0.90 haircut applied per doubling — beyond 2 GPUs this is a projection, not a measurement). Widening TP always raises total capacity, because every DP group re-pays for its own full copy of the weights.

DP (data-parallel, any N) keeps N per-replica caches. Aggregate throughput scales ~N×, but a returning user is only warm on their home replica (sticky routing), and a shared CPU-offload buffer splits N ways.

MoE decode

MoE decode reads only the active experts: weight bytes per step grow with the batch's expert union, from the shared block up to all routed experts. The explorer uses the conservative no-overlap union bound (the docs also show the optimistic expected-union bracket). Past the point where every routed expert is read each step — n = 32 on the 35B-A3B — the fixed weight read stops growing with the batch — the slope break the explorer's chart D shows.

KV cache dtype

All constants assume the FP8 KV cache (--kv-cache-dtype fp8_e4m3), as tested in the baseline. The FP16 switch doubles KV bytes/token (half the pool, heavier decode reads); weights and the recurrent state are unaffected, and the activation-reserve calibration stays pinned to the FP8 anchor. Sanity: FP16 + 27B + 1×H200 gives a 1.39M-token pool, inside the baseline's measured [1.14M, 1.40M]. GLM-5.3's sparse-MLA path and DeepSeek-V4-Flash-0731's V4 path both require a quantized (FP8) KV cache in vLLM, and DeepSeek-V4.1-Flash's main KV is FP4 by training (E2M1 with one E4M3 scale per 16 channels; its sliding windows stay FP8) — that native layout is what its FP8 arm prices — so the FP16 toggle is disabled on all three. On Qwen3.8-Flash-Next the switch also doubles the top-2048 sparse-decode read (main-KV bytes); its compressed indexer keys stay fp8. GLM-5.3-Flash inverts the constraint entirely: its fp8 KV cache is Blackwell-only — vLLM's recipe requires BF16 KV on Hopper — so on H200 the KV toggle locks to FP16 and the model itself refuses to price the fp8 arm there; the FP8 numbers are a Blackwell configuration.

CPU offload is storage only

A session offloaded to host RAM still counts as warm capacity, but only sessions whose KV is resident in GPU HBM can decode — an offloaded one must be restored over PCIe first (a latency the model does not price). Every decode-side readout (the per-user/aggregate stress tiles, the explorer's chart C capacity zone) is therefore computed over HBM-resident warm sessions only, so moving the offload slider never changes per-user decode speed. For DP the buffer splits across replicas.

Cold sessions in the pool

Cold / invalidating requests occupy KV but never count as warm — they cap the reusable-hit ceiling at (1−f). Past ~25% invalidation about a third of the pool is paying for sessions that will never be reused.

The hit rate at the load (since 2026-11-02)

With more users than the pool keeps warm, LRU does not lose all of them. It keeps each context until it has been idle for a characteristic time T*, the T* at which the contexts so held fill the pool, so the sessions that come back soonest stay warm and the ones away longest are evicted first. The page knows a think time and nothing else about the gaps, so it takes them as exponential: a session is then resident a share 1 − e^−T*/Z of its cycle and comes back warm with the same probability, and with U users on a pool that keeps W warm that share is W ÷ U. W counts reusable sessions, so the share of requests that miss is f(U) = max(f, 1 − W ÷ U): the configured miss rate f up to W ÷ (1 − f) users, the users the pool cannot hold past it. The Cache hit rate tile reads 1 − f(U) at your load.

Those misses are cold requests like any other. Every figure quoted at your load (prefill duty, the cold wait, the slowed share, TTFT, B*, the bill) is priced at f(U), and each of the cold-wait, slowed-generation and saturation ceilings is the user count U at which its figure, priced at f(U), meets its limit. So there is no cache ceiling: a pool too small for the load shows up as one of those three, usually just past W ÷ (1 − f), since a miss costs about twenty hits of prefill. Past that point every added user is a cold stream; slowed generation, for one, then binds at W + K, where K is the number of all-cold users that fill the slowed-share limit on their own.

The round-robin worst case. A benchmark that sends its prompts in a fixed order is the exception: LRU evicts each prompt just before it comes round again, and every request past W misses (Bélády's cyclic case, measured on one H200 in docs/writeup.md). Chart G draws W as a dotted line for that reason; it is the stated worst case, not a limit.

How far to trust it. On the public traces where a finite cache is the limit and sessions are known (Alibaba Bailian, SemiAnalysis AgentX; research/public_traces.md), the hit rate past W falls gradually, as this rule says, not to zero. The rule fed each trace's own gap distribution lands within 5 points of an exact LRU replay or of the published hit rate on all ten Bailian and AgentX points. With only a mean gap, as here, it is within 5 points on seven of the ten. It reads 9–12 points high on Bailian's to-C trace A at 1M and 3M tokens, where the trace's gaps are far from exponential, and 19 points low on AgentX's B200 point, where most returns come within seconds. The cliff it replaces was off by up to 73 points on the same points. The gap shape is left out on purpose rather than added as another input: for a capacity figure that does not lean on it, replay recorded sessions against the endpoint itself (ws replay), which plays the recorded gaps.

The three un-measured structural knobs

Other projections carry their own flags where they appear; these three are memory-accounting assumptions with no measurement behind them at all, so they are separate, named controls rather than a bundle.

• KV cache across TP ranks (Sharded (DCP) / Replicated): what a group's tensor-parallel ranks store when the model has fewer shardable KV heads than ranks. TP splits attention by head; the rank groups beyond the heads keep a full copy of every token's cache and re-read it each step — and the single-latent caches (GLM-5.3's MLA, GLM-5.3-Flash's NoPE-MLA, the DeepSeek Flash models' MQA, whose windows replicate too) have one head, so plain --tensor-parallel-size 8 stores eight copies. Sharded (the default, and the layout every number here has always assumed) prices vLLM's decode context parallelism, which splits those copies along the sequence; the deploy recipe emits --decode-context-parallel-size tp / kv_heads. Replicated prices plain TP: the pool divides by tp / kv_heads and per-step cache reads multiply by it (GLM-5.3 on 8×B300: 27.3M → 3.4M tokens). Disabled while the group is no wider than the heads (Mistral's 8 heads never replicate; the Qwens shard at TP ≤ 2 or 4). DCP's caveats as of 2026-08: speculative decoding under it is still in development in vLLM, so the MTP slider and this arm may not be available together; attention-DP + expert-EP, the other way to one copy, is not modelled. research/kv_tp_sharding.md.
• Recurrent state dtype (bf16/fp32): the dtype vLLM keeps the Gated-DeltaNet recurrent state in — a flat per-session charge on top of the attention KV. bf16 is inferred, not measured: it is what makes the baseline's observed 75 MiB match the 27B's state arithmetic. The 2×H200 log does not settle it — its 23.8 GiB of allocator state pages would be ~325 sessions at bf16 or ~162 at fp32, and neither matches the 17 concurrent sequences, so that pool looks pre-allocated from max_num_seqs rather than per-session. fp32 would double the charge and costs the Qwens ~10% of warm capacity — the largest un-measured structural knob on those models. Mistral-3.5 and GLM-5.3 hold no recurrent state (the control is disabled); DeepSeek-V4-Flash-0731 (and its Vision-Exp sibling) and V4.1-Flash pay a fixed 14.9 and 2.8 MiB/session of attention windows and fp32 compressor buffers whose precision is set by the serving stack, so they have no knob to turn either.
• Deployed-weight overhead (as-published/+15%): the slack between a checkpoint's stated parameter bytes and what the server actually resides — fused/padded kernel buffers, non-quantized odds and ends, allocator rounding. Eight of the nine models' w_resident figures are raw or on-disk checkpoint bytes, i.e. under-estimates by an unknown margin; the one calibrated data point is the 27B's as-deployed 28.8 GiB running ~15% above its raw params. Transferring that ratio to a different architecture and quantization recipe is an extrapolation — which is why it is a knob rather than a constant. The knob is disabled on the 27B itself, where the figure is already as-deployed and applying +15% would double-count.

MTP / speculative decoding

Two controls: Draft acceptance α and Draft tokens k. A sequence advances 1 + α + … + α^k tokens per step, and each draft token costs a draft pass (a block drafter such as Kimi-K3.1-Pro's DSpark runs one pass for all of them), a verify position per sequence and, on a MoE, more experts read (§ F.2). Both default to the selected model's own: its serving recipe's draft count and its measured or nearest-measured acceptance. The paragraph below is the history of the speedup slider this replaced.

The old slider set the speculative-decode speedup (1.0× = off). The base quantity is the per-draft acceptance α: with 2 draft tokens, speedup = 1 + α + α², so 2.94× ⇔ α ≈ 97% — an accepted length calibrated on Qwen3.6-27B from a measurement not published here, and carried over unmeasured on 3.8; 1.7× ⇔ α ≈ 47% was the earlier fit. The inversion is exact only for the two 2-draft Qwen models (Qwen3.8-27B, Qwen3.6-35B-A3B): GLM-5.3's MTP drafts 5 tokens, DeepSeek-V4-Flash-0731 drafts 7, DeepSeek-V4-Flash-Vision-Exp 3 (its model card's serve command) and DeepSeek-V4.1-Flash 5 (DSpark, the trained block size), Qwen3.8-Flash-Next drafts 3 and GLM-5.3-Flash 5 (their vLLM recipes), and Mistral-Medium-3.5 ships no MTP module at all (its slider models an external EAGLE-style draft, unmeasured). The slider applies to the selected configuration only — the frontier table prices every row at its own model's capability, so cross-model ranking is never moved by this control. MTP + hybrid-model prefix caching is an immature serving path, so the conservative purchasing view keeps it at 1.0×; the purchasing base likewise excludes CPU offload and N>2 TP. These controls exist so that headroom stays visible, not hidden.

The workload model

Prompt lengths

Both request classes draw log-normal lengths. The user class is fitted by maximum likelihood on 1,850 real request lengths (median 31k, σ = 0.81): ~68% of prompts within ×/÷ e^σ ≈ 2.25 of the median, p95 ≈ 3.8× the median, mean ≈ 1.39× the median. The mean — not the median — drives warm capacity, so a fatter tail of huge pool-hogging sessions means fewer warm. The subagent class is assumed (median 8k, σ = 0.90; no subagent trace has been collected yet). Lengths above the max_seq_len cap are truncated to it — a lower cap trims the tail (smaller worst-case cold prefill, less pool hogging), and a median far past the cap degenerates toward all-sessions-at-cap.

Subagents and the shared prefix

The subagent ratio r is subagent requests per user request, so subagents are r/(1+r) of sampled sessions; they draw much shorter prompts behind their own lean 3k prefix (or reuse the user prefix, if that toggle is on — no extra reserved block, but each subagent then carries the bigger base). The system / shared prefix is the stable block every user request shares — system prompt, skills, rules, tool definitions. It is reserved once per cache and deduplicated across every session, so a bigger shared prefix costs the pool once and saves it on every session: raising it raises warm capacity. The reference workload uses 15k; the baseline's real agent already exceeded 30k. This only pays off while the prefix is byte-stable across users and turns — a per-tenant salt or an edited tool list turns it into per-session bytes, which is what the invalidation control prices.

People, sessions, and think time

The model prices concurrent sessions per replica group. The optional population layer derives the system total as headcount × peak_active_share × sessions_per_active_user, then divides it across DP groups. Headcount is the number of people with access. Peak active share is the fraction with a live session in the busiest sizing interval. Sessions per active person counts parallel agents. Leaving headcount empty keeps the direct Concurrent-users control and today's numbers. Both population shares are unmeasured inputs. Estimate them from gateway logs by counting distinct user IDs per 15-minute window and simultaneous active sessions per user ID. The tool takes the conservative reading that every parallel session runs a full request stream with its own think-time loop. A person holding several sessions but attending one at a time is the lower-load bound: cache use multiplies, but prefill load does not. Think time is the full inter-request interval. It includes the previous response's prefill and decode, not idle time on top. The open-loop arrival rate is sessions divided by think time, so response latency never changes the next arrival. A closed loop slows when responses do. The reference load remains 64 sessions at one turn per 30 s, or 2.13 main requests/s. Two readouts use a separate stress point: the ITL spike and tokens-lost figure price every GPU-resident p5-warm session decoding. Act 2's steady-state tiles read the selected load. A warm hit still prefills its turn size, 2,000 tokens by default. At low miss rates the hit leg dominates duty, so 500 to 8,000 tokens moves B* by about 3.5× on the 27B/TP2.

The steady-state decode point — what the load actually produces

Act 1's per-user decode figure is a stress test: it prices the curve with every GPU-resident warm session decoding at once. That is the right worst case — a cache flush or a correlated burst really does put the whole population in the batch — and the wrong expectation. Because arrivals are open-loop, a user spends most of the think-time interval waiting on a tool or a human, so the number of sequences in the decode batch at any instant follows from Little's law on the decode phase alone (a request queueing for prefill, or being prefilled, is not yet decoding): E[n] = λ × out / v(n). Multiplied through, that is a flow balance with no inversion in it — n × v(n) = λ × out, delivered output tok/s against demanded. The left side is the aggregate decode curve (chart D), strictly increasing in n, so the crossing is unique and both charts C and D mark it.

The gap is large enough to change a purchase. On the 27B / 2×H200 at the reference load the batch holds ~7 sequences at ~137 tok/s, against ~195 warm sessions at ~18 tok/s if they all decoded at once — the same curve, read at two batch sizes, so the ~8× is the size of the reporting error, not a hardware result. The batch cannot outgrow the pool: a batch of n needs n contexts resident in HBM, so the search stops at the GPU-resident warm population, and a load that population cannot retire decoding at once is reported as saturated rather than priced at a batch the machine cannot hold (past it the engine is evicting, and every return is a cold prefill that the miss-rate control prices). Three caveats travel with the number, and the tiles state all three. Mean field: v is evaluated at the mean batch rather than averaged over the batch-size distribution; v is convex in n, so by Jensen E[v(N)] ≥ v(E[N]) and this is the conservative side. Decode only: prefill chunks sharing a forward pass are the ITL spike, priced separately — this is the clean-decode speed between spikes. It assumes the requests are served at all: if prefill duty has reached 100% the queue is unbounded and nothing reaches a steady state, which is why the tile refuses to quote a speed there.

The point depends on the request rate and the output length only through their product, and on nothing else — so those two inputs are the entire error budget. n is not linear in that product: per-user speed falls as the batch grows, so successive output-length multiples move n super-linearly until the curve saturates outright. Output length was long the one assumed input on this page — the workload model is fitted on 1,850 real prompt lengths and had never fitted output lengths. The 400-token default comes from an agentic trace not published here (the earlier 1,000-token assumption was a consistency guess against the traced 10.8 s served per request), and it remains an exposed slider rather than a buried constant. Mirrored as steady_decode_point() in src/workingset/model.py; the tables regenerate from tables.py.

The prefill & queueing model (analytic and unvalidated)

Max cold req/s — the prefill ceiling

The one readout priced in FLOPs, not HBM bytes (research/prefill.md). A miss re-prefills its whole context in max_num_batched_tokens chunks (GEMM + the quadratic attention term) against the part's dense FP8 peak; every readout is pinned to vLLM's 32,768 default, where the effective MFU is the central 45% — the soft input: the plausible [30–55%] bracket spans ~1.8× end to end. The headline is 1/(expected miss cost), priced on the context distribution's heavy tail (E[L²] — the quadratic attention term must not be priced at the mean length); the p5–p95 sub-line is the cost spread of a single miss. It is a per-replica-group ceiling that no KV pool, CPU offload or warm headroom can raise. f* is the miss rate that saturates the group at its share of the current load (users / think time, warm turns included; a DP grid splits that load across replicas).

The chunk-size trade (the explorer's chart E)

Every chunk is charged its attention over the cache it lands on, so a miss's FLOPs are chunk-size invariant (the pair count telescopes) — but each of its passes also streams the resident weights, an overhead that multiplies by the pass count, so its total machine time is not: that per-pass roofline is what makes the effective MFU fall at small chunk sizes. Small chunks shrink the ITL spike every decoder sees (the spike prices the chunk's marginal FLOPs — its host pass streams the weights anyway) but sink MFU and the cold-request ceiling with it: that is why vLLM's default is 32k, not 2k, and why the explorer sweeps the knob in a chart instead of exposing it as a slider. A warm hit pays its new turn's attention over the whole cached context at the marginal rate (its small pass rides the serving steady state).

Cold-spike tolerance — queueing and bursts

Max cold req/s and f* are a duty cycle — a mean rate against a mean service time — and a mean sees neither variance nor correlation. Both bite. A miss's service time runs as L² on a log-normal L, so its squared coefficient of variation lands at 4.7–8.3 across the nine models where an exponential would sit at 1, and the M/G/1 (Pollaczek–Khinchine) wait diverges well below f*: the first-token point f_sla — the miss rate whose mean TTFT reaches 10 s, a reference figure that no longer decides the verdict — binds at 0.35–0.93× f* (tightest on the most prefill-fragile config, Mistral-3.5/TP4), with the duty cycle still reading a comfortable 76–93%. And invalidation arrives in clumps, so B* prices the largest simultaneous burst of misses whose last request still gets a first token in budget; it is linear in the budget (a 5 s target halves every number, changing no ranking) and reaches zero exactly at f* — which is what makes f* a limit rather than an operating point. A global flush puts the machine at f = 100% until sessions re-warm; above 100% duty there is no steady state and recovery is set by admission control, unmodelled here. One queue per replica group: a DP burst spreads across replicas only as well as the router balances it, which the sticky routing DP needs for cache reasons works against.

Slowed generation — what a decoder feels

While a cold prompt is prefilled, every GPU step carries one chunk of it (32,768 tokens by default) on top of the running decodes. The step takes mixed = decode + chunk seconds instead of decode, so every stream drops from its normal speed to mtp / mixed tok/s. The share of generation time spent in such steps is min(1, λ · f · E[S | miss] · mixed / chunk): the cold duty (machine seconds of cold prefill per second), stretched by the ratio of a mixed step to its chunk. The limit Generation slowed at most caps that share, and the slowed generation ceiling is the load where it is reached; the share is linear in the rate, so the ceiling is closed form. Approximation: warm-turn prefills are left out of the share, since a few thousand tokens slow a step far less than a full chunk. One long cold request is how long one cold prompt of the p-th percentile length keeps every stream slowed: ⌈L_p / chunk⌉ · decode + prefill(L_p), each chunk priced with the cache it attends over.

Cold request wait — the queue in front of a cold prefill

A cold request (a cache miss, a compaction) queues behind the prefills already waiting before its own starts. The mean wait comes from the M/G/1 queue above. The p-th percentile reads an exponential tail off that mean: P(W > t) ≈ ρ · exp(−t ρ / E[W]), so W_p = (E[W] / ρ) · ln(ρ / (1 − p)) when p > 1 − ρ and zero otherwise. Approximation: the tail is exact for exponential service; this workload's heavy-tailed miss service makes the true tail heavier, and vLLM is neither first-come-first-served nor processor sharing, so read the figure as the first-come-first-served end of the bracket. The cold wait ceiling is the load where W_p reaches Cold request waits at most.

The Cold request, idle server table is the prefill time of one cold prompt with nothing else running. It is a property of the deployment, not a limit, and no limit uses it. The p95 TTFT over all requests, once the verdict's latency limit, is no longer in it: with about 9% of requests cold it mostly reports prompt length, which a 200k-token cold request misses on an idle server.

TTFT, and what a hit waits for

Mean TTFT comes from the same M/G/1 queue, solved against the mean — a p95 budget binds at a lower miss rate than anything shown. The sharpest consequence is on the requests that hit: under first-come-first-served a warm hit waits behind whatever misses are in front of it, so the cache-miss rate is a latency parameter for the hitting users too — at a 20% miss rate on the 27B/TP2 a hit's TTFT reaches 74× its own service time. Processor sharing is the other end of the bracket (vLLM admits in arrival order but runs several admitted prefills concurrently); neither end is uniformly optimistic, and which is which flips by request class.

What a burst costs while it clears

A backlog drains at (1 − duty) seconds of work per second, because the standing traffic keeps arriving throughout — so the last request's TTFT is the drain time under either scheduling discipline. During the drain the scheduler has a prefill chunk to place in every forward pass, so the ITL spike is the steady state for the whole drain: the warm users who did nothing wrong simply stop receiving tokens at their normal rate, and the tokens-lost figure is that difference integrated over the drain. The "full flush" burst preset sets the burst to the p5 warm population of one replica group, capped at the slider's 512 — the correlated event a prompt-template deploy actually is.

The sensitivity panel — "what would flip this decision"

Chart G sweeps one assumption (the miss rate); the sensitivity panel sweeps every soft input — miss rate, think time, the two latency limits, warm turn, prefill MFU, draft acceptance, decode η, and the prompt-length median and shape — one at a time with the rest held fixed, and reports where the binding constraint changes hands or the current load stops fitting. The closed-form ceilings (cold wait, slowed generation, saturation) are exact along every sweep. The two sampled inputs, the warm pool (which prices the capacity misses inside the other ceilings) and the decode ceiling, cannot be Monte-Carlo re-filled at every point, so the workload rows (marked ≈) use mean-field stand-ins — expected pool cost per session for the pool, mean-context decode speed for decode — each calibrated to pass exactly through the Monte-Carlo value at the current settings. Flip locations on those rows are approximate to the stand-ins' curvature error; the ranking of which assumption sits closest to a flip is robust.

What is deliberately not priced

Every one of these makes a real machine worse than the model: kernel-launch/scheduler per-pass costs (the small-chunk end of the explorer's chart E reads better than a real machine), Poisson arrivals (real agentic traffic is burstier), solving against mean rather than percentile TTFT, preemption/recompute under a full KV pool, the decode batch stretching a drain 1–3%, and PCIe restore latency for offloaded sessions. NVFP4 checkpoints are charged the FP8 tensor rate — their mixed W4A4/FP8/BF16 recipes could really run faster or slower, so for NVFP4 configs the prefill figures are a modeling choice, not a bound.

Hardware & model constants

H200 and B300

The H200 (141 GB HBM3e, 4.8 TB/s) is the calibrated baseline. The B300 (Blackwell Ultra: 288 GB HBM3e, 8 TB/s, native FP4) reuses the H200-solved ~18 GiB per-GPU reserve, plus a measured +9.75 GB/GPU correction: the H200 actually delivers ~150.75 GB usable against its 141 GB vendor figure (the calibration silently absorbs that margin), while a real B300 nvidia-smi dump shows 275,040 MiB = 288.4 GB — nominal bytes, no Hopper-style over-provision — so the transferred reserve must add the hidden margin back (research/gpu_b300.md; formerly a sensitivity, measured 2026-07-27). Both parts are modelled as 8-GPU NVLink domains, so a deployment is one node at most.

NVFP4 weights (B300-only)

Swaps in real NVFP4-checkpoint byte counts (4.5 bits/param on quantized tensors; what stays high-precision is recipe-specific — see research/nvfp4.md — e.g. RedHatAI's 35B-A3B keeps the DeltaNet blocks BF16 while NVIDIA's GLM recipe quantizes only the routed experts). Gated to native-FP4 GPUs: vLLM's Hopper fallback is weight-only Marlin with a known correctness bug (open as of 2026-07-27), so H-generation NVFP4 is deliberately not modelled. The KV cache is never 4-bit here — an owner policy: vLLM's nvfp4 KV shipped 2026-05 (Blackwell-datacenter-only) but is not modelled. Non-obvious: on the MoE models NVFP4 makes the fixed per-step read heavier (BF16-kept blocks) while expert reads shrink 1.78× — low-concurrency decode slows, high-concurrency speeds up. Every model but DeepSeek-V4-Flash-Vision-Exp has a measured NVFP4 checkpoint as of 2026-10-02 (research/nvfp4_2026-09.md and research/nvfp4_2026-10.md, every figure read from every shard header): NVIDIA's for Qwen3.8-Flash-Next (routed experts only, 132.6 vs 185.5 GB) and DeepSeek-V4-Flash-0731 (its experts already ship 4-bit with E8M0 block-32 scales; NVIDIA's repack to E4M3 block-16 scales is 5% heavier — the arm exists for the NVFP4 kernel path, and the explorer prices what it weighs), RedHatAI's for Qwen3.8-27B (dense MLPs only, 23.4 GB) and GLM-5.3-Flash (the vLLM recipe's named checkpoint: 42 decode layers' experts NVFP4, the rest upcast to BF16 — 197.8 vs 328.3 GB resident, but the fixed per-step read grows 14.0 → 16.7 GB). NVIDIA's GLM-5.3 (routed experts NVFP4, attention and shared experts upcast to BF16: 464.1 vs 755.6 GB resident, fixed per-step read 18.9 → 34.8 GB) and DeepSeek-V4.1-Flash (the same repack as 0731, 3% heavier than native: 527.3 vs 510.3 GB). DeepSeek-V4-Flash-Vision-Exp has community repacks only (2026-10-02), so its option is greyed out rather than projected.

The nine models

• Qwen3.8-27B (2026-08-14) — dense hybrid, the calibrated baseline: 64 layers (48 DeltaNet + 16 full-attention) → 32 KiB/token FP8 KV plus a 75 MiB recurrent state per session. 262k native context, 1M via YaRN. Its decode measurements (the held-batch sweep, MTP acceptance) are a Qwen3.6-27B one, the model it replaced on 2026-09-05: the two share every config field and every tensor shape (1,606 tensors, identical FP8 checkpoint size), so the memory, prefill and decode constants carry over exactly; the MTP acceptance (2.94) is a property of the draft head's weights and is carried over unmeasured. Terminal-Bench rises from 60.7% to 79.8% on 2.1 (AA, xhigh effort); on 4.0 it scores 5.6% and Qwen3.6-27B has no run. research/model_qwen38_27b.md.
• Qwen3.6-35B-A3B — MoE, ~3B active: 40 layers (30 DeltaNet + 10 full-attention) → 10 KiB/token KV; 256 experts / 8 routed (expert-union kink at n = 32). Constants from the published config — derivations in research/model_35ba3b.md. 262k native, 1M via YaRN.
• Mistral-Medium-3.5-128B (open weights, 2026-04) — dense, 88 uniform GQA layers → 176 KiB/token FP8 KV, the study's KV-hungriest model. No recurrent state, no MTP module. Hard 262k context. research/model_mistral_medium35.md.
• GLM-5.3 (open weights, 2026-08-25, GLM-5.3 License — MIT plus a security-review clause for Model-as-a-Service operators above $10B revenue) — 744B-A40B, MLA + DeepSeek Sparse Attention: stores 47.3 KiB/token on B300 and 53.4 KiB on H200 (MLA latent + indexer keys; vLLM's FP8 latent row is 576 B on Blackwell and 656 B on Hopper) but decode reads only the top-2048 tokens per layer plus an indexer scan (2,772 B per context token). FP16 KV (BF16) is servable on both parts. FP8 weights (~756 GB) fit from 7×H200 / 4×B300 per the vLLM recipe's 893 GB floor (3×B300 = 864 GB passes the pool arithmetic only); NVFP4 (~465 GB) from 2×B300. 1M native context. Every constant was derived for GLM-5.2, which it replaced on 2026-09-06: the two agree on 55 of 56 config keys (only transformers_version differs) and on every one of 118,629 tensor shapes (identical FP8 checkpoint size), so the memory, prefill and decode constants carry over exactly; the MTP fit (1.7, itself transplanted) stays unmeasured. The NVFP4 bytes are measured on NVIDIA's GLM-5.3 checkpoint. Terminal-Bench rises from 77.9% to 83.9% on 2.1 (AA, max effort); on 4.0 it scores 41.9%, the best AA-measured figure on this frontier, and GLM-5.2 has no run. research/model_glm52.md, research/model_glm53.md.
• DeepSeek-V4-Flash-0731 — 284B-A13B, compressed sparse attention (21 CSA ratio-4 + 20 HCA ratio-128 + 2 SWA layers): only 3.4 KiB/token stored — ~10× below V3-class MLA — plus a fixed 14.9 MiB/session. Fits from 2×H200 / 1×B300. 1M native context. A different architecture from V4.1-Flash below, not an older revision of it: out of the study from 2026-09-10 and back alongside V4.1 since 2026-09-16, with one correction on return — decode charges its per-session state at what a step moves (1.05 MB/sequence: ring-slot writes and the compressors' slot writes, pooling reads and overlap rolls) rather than twice the whole 14.9 MiB, whose window reads are already priced with the cache. Terminal-Bench 78.7% on 2.1 and 12.1% on 4.0 (AA, max effort). research/model_dsv4flash.md.
• DeepSeek-V4-Flash-Vision-Exp (open weights, 2026-08-31, MIT) — DeepSeek's first V4 multimodal model: V4-Flash-0731's text backbone plus a 0.47B vision tower (32-layer ViT + aligner), continued-trained. Its text config and reference cache code are 0731's unchanged, so every cache, decode and prefill constant is too; only the resident weights grow, by the tower's 0.93 GB (167.8 GB, every shard header read). Fits from 2×H200 / 1×B300. No official NVFP4 checkpoint, so the option is greyed out. Priced on the text workload only — image encoding and image tokens are not modelled. Terminal-Bench 74.2% on 2.1 and 12.1% on 4.0 (AA, max effort). research/model_dsv4flash_vision.md.
• DeepSeek-V4.1-Flash — 552B backbone + 196B Engram tables, 8B active in prefill / 16B in decode: a Causal Encoder-Decoder whose decoder global KV is projected from the 20-layer encoder, so prefill runs half the network. CSA2 shares four FP4 compressed caches (three ratio-2, one ratio-1) across the 40 layers: 890 B/token stored — 4× below V4-Flash-0731 — plus a fixed 2.8 MiB/session. Decode reads four full-axis indexer scans, top-512 latents per layer, four candidate-pool reads and the windows. vLLM keeps the 203 GB Engram tables in HBM, so 510 GB resident: fits from 8×H200 (5 by arithmetic) / 2×B300. 1M native context. Terminal-Bench 26.8% on 4.0 is Artificial Analysis's run (max effort, read 2026-11-04), which replaced the card's 31.2. 90.6% on 2.1 is the vendor card's figure (DeepSeek Harness Minimal mode, max reasoning effort, 1M context), admitted by owner decision on release day (2026-09-10) and kept because AA has no 2.1 run: the one score on the frontier not measured under AA's protocol. The same card scores the same model 84.1–90.6 across eight scaffolds on 2.1, which is the size of the caveat. research/model_dsv41flash.md.
• Qwen3.8-Flash-Next (open weights, 2026-08) — MoE 125B-A6B plus a 51B FP8 n-gram embedding table: 48 layers (36 DeltaNet + 12 QSA sparse attention) → 13.4 KiB/token stored plus a 56.8 MiB bf16 recurrent state; decode reads the top-2048 selected tokens and a ratio-4 compressed indexer scan. 512 experts / 10 routed — the study's deepest expert-union kink (n = 51.2). Fits from 2×H200 / 1×B300; NVFP4 from NVIDIA (routed experts only, 132.6 GB). 262k native, 1M via YaRN. research/model_qwen38flashnext.md.
• GLM-5.3-Flash (open weights, 2026-08-25, MIT) — MoE 320B-A18B hybrid: 34 KDA linear-attention + 11 NoPE sparse-MLA layers (+1 MTP DSA layer whose cache is charged in storage, the GLM-5.2 convention) → 6.4 KiB/token stored (nope-only 512-B latents + a kpool-4 compressed indexer, 7.4× below GLM-5.2) plus a 74.4 MiB bf16 KDA state per session, the study's second-heaviest. 288 experts / 8 routed (kink n = 36). Fits from 3×H200 / 2×B300 by pool arithmetic; fp8 KV is Blackwell-only — Hopper must serve BF16 KV, the study's first GPU-coupled KV dtype, enforced: the model refuses to price the fp8 arm on H200 and every published H200 figure is the BF16-KV arm. 1M native context. research/model_glm53flash.md.
• Mistral-Large-4 (open weights, 2026-10-29, Mistral Ecosystem Open-Weight License: MIT plus a separate commercial licence for Model-as-a-Service operators) — MoE 1.05T-A49B (52B with embeddings and lm_head), the study's largest: 72 layers of MLA + DSA as GLM-5.3, with cross-layer KV sharing (9 late NoPE layers attend to layer 35's latents and store none) and index sharing (28 layers reuse the previous indexer's top-k) → 46.0 KiB/token stored in vLLM's 656-B fp8_ds_mla rows, 3% below GLM-5.3 on B300, plus 3.8 MiB per session for the DSpark drafter's three 2,048-token windows. 128 experts / 4 routed (kink n = 32), 32.2 GB of routed experts read per token, the heaviest here. 1,060.5 GB in FP8, so it fits no 8×H200 node (9 by arithmetic; the vLLM recipe runs 2×8 H200, which the page does not price); 8×B300 in FP8, from 4×B300 with the official NVFP4 checkpoint (610.0 GB, routed experts only). FP8 KV only (fp8_ds_mla). 1M native, 262k validated by the recipe. Terminal-Bench 24.7% on 4.0 (AA, high effort); AA has no 2.1 run. research/model_mistral_large4.md.
• Kimi-K3.1-Pro (open weights, 2026-11-04, Kimi K3.1 Pro License: MIT terms plus a separate agreement for Model-as-a-Service operators above $20M a year, internal use exempt) — MoE 2.8T-A104B: 69 KDA linear-attention layers and 24 gated DSA layers (MLA latents plus a top-2048 indexer on each), the routed experts in a 3,584-wide latent space, 896 experts / 16 routed (kink n = 56). 19.4 KiB/token stored on B300 (24 latents in plain fp8_e4m3 rows of 576 B, 24 indexer keys, and the DSpark drafter's 5 dense latents, which vLLM keeps in the same row) plus a 226 MiB bf16 KDA state per session, the study's heaviest. The checkpoint ships MXFP4 experts (quantization-aware trained) and keeps everything else BF16, so every step reads 111.5 GB of always-active weights, 61 GB of them KDA projections, before 25.8 GB of experts per token. 1,566.6 GB with the drafter: no 8×H200 node holds it (13 by arithmetic; vLLM's recipe supports no Hopper part), 8×B300 at TP8. No NVFP4 checkpoint. Its DSpark drafter writes all five drafts in one forward pass, priced as one draft pass per step (p = 1, § F.2). 1M native context. Terminal-Bench 67.2% on 4.0 (AA); AA has no 2.1 run. research/model_kimi_k31pro.md.

Power & the bill

Wall power is priced from the duty cycle the model already computes, in three GPU states (research/power.md): prefill at ~0.90× TDP — compute-bound prefill is power-cap-limited, so the figure is flat across the MFU 30–55% band (the cap binds before the FLOP peak); decode at ~0.55× TDP (bandwidth-bound; the softest constant, measured band 0.45–0.75 on Hopper); idle at a measured warm-idle floor. The prefill fraction is min(1, ρ) — the duty cycle, clamped at saturation, past which the GPU simply cannot prefill more than 100% of the time; the decode fraction is the output-token demand (~400 tokens per request, from a trace not published in this repo; previously assumed 1,000) against the decode capacity at the per-user decode floor (40 tok/s by default; the decode-floor slider moves it), capped at what prefill leaves. On top: a per-GPU host/chassis adder (DGX spec ceiling — conservative) and the facility PUE. The GPU term carries ±20–25%; every B300 figure is extrapolated — no published Blackwell Ultra power-state measurements exist. The GPUs are paid for one of two ways, and power is counted once. Rent: GPU-hours at an on-demand rental price (cross-provider medians, getdeploying.com, September 2026: H200 $4.40, B300 $7.89 per GPU-hour at EUR/USD 1.16; the GPU-rent slider overrides it) for the hours the team works (210 h a month by default: 10 h a day, 21 days), since a rented GPU is up and billed only then; the rent includes power, cooling and the host, so there is no electricity line. A monthly reservation is rent at 720 h. Own: the purchase price of an 8-GPU server over 8 (Supermicro store, November 2026: H200 $328k, B300 about $663k, at EUR/USD 1.1267) written off over 36 months, plus the electricity above at the meter: the load’s draw for the working hours and the idle draw for the rest of the 720-hour month. Neither counts staff, networking or storage. The bill, the frontier’s cost column and the €/user and €/Mtok figures are that one total. Mirrored as power_draw() / energy_cost() in scenario_model.py.

The frontier as a picture (chart H)

The frontier table ranks every model × split on this GPU by max users and prints the bill. The hardware line of the bill is a function of the GPU count alone (only the electricity term follows the load), so a users-versus-cost view collapses every row of one topology onto a band and says "run fewer GPUs". Chart H asks the purchasing question instead: what does one seat cost against what the model can do? y = the bill with the configuration full — at its max users, electricity included at that load — divided by those users, its €/seat at capacity (log). Not the bill at your load over your users: that is the GPU count again, and every row on a topology would price the same; x = the model's Terminal-Bench pass@1 as measured by Artificial Analysis (one lab, one harness per version; vendor cards quote other versions and harnesses and are deliberately not used, with one exception: DeepSeek-V4.1-Flash carries its card's 90.6 on 2.1 by owner decision, AA having no 2.1 run of it; research/terminal_bench.md). Which version is a toggle on the chart, because the two do not rank this field the same way: 2.1 (89 tasks × 3 repeats, Terminus 2 — AA's legacy eval) puts the top four models inside 7 points, about what one scaffold change moves a single model, while 4.0 (66 tasks × 3 repeats, mini-SWE-agent v2.4.6 — the version in AA's Intelligence Index since v4.3, v5.0.1 today) puts them inside 17 and reverses the order at the top: GLM-5.3 leads Qwen3.8-Flash-Next by 17 points there and trails it by 2 on 2.1. The staircase is redrawn per version, and the frontier table prints both columns whichever is on the axis. 4.0 being the harder benchmark does not make it the better axis here — it is also a different harness and a smaller task set, and two models score a measured 0.0% (0 of 198 runs), which is a score and is plotted, not a missing run. The default axis is 4.0, the version every model has an AA run on; AA no longer runs its legacy 2.1 eval on new models and has none of Mistral-Large-4, which therefore appears on the 4.0 axis only. The score is a model property: every split of a model shares its x, so a model's rows stack in a column and the column is the price of the topology choice; the FP16-KV and NVFP4 arms inherit their base model's score because the quantisation loss is unmeasured. Only rows that carry your load are drawn — a configuration that cannot has no seat price — and the rest are counted in-chart. The Pareto-efficient set (no other row scores at least as high for no more money) is joined as a staircase: for any capability floor, the cheapest seat within the tolerance. Two bills within 1% of each other are the same money — the power model carries ±20–25%, and without the tolerance three 1-GPU rows differing by a few euros of electricity would all count as efficient — so a row can be dimmed by one that scores higher and bills up to 1% more; rows at the same score are dimmed by any strictly cheaper one. Nothing here measures the score against the study's workload: AA's runs are at the vendor's maximum effort, and a cheaper effort setting scores lower and generates fewer tokens — neither side of that trade is modelled.

Testing the hypotheses — the generated harness

The planning stance defines "comfortable capacity" as measured SLO capacity — the largest replayed population meeting explicit p95 TTFT and per-user decode targets — and is explicit that this model cannot produce it. The explorer's Test step hands out the on-screen configuration as a workingset.toml, and ws test measures it: closed-loop simulated users sharing one byte-identical prefix (so vLLM prefix caching engages), log-normal contexts, forced-miss injection, a load ladder bracketing the predicted limit, and a predicted-vs-measured verdict table. The predictions travel in no file — ws predict recomputes them from the config, so a stale number cannot be smuggled into a run. One run bounds warm capacity from below unless it reaches eviction, and B* needs --burst — the report prints its own "what this run does not establish" trailer.

Provenance

The B300 / NVFP4 / Mistral-3.5 / GLM-5.2 constants were researched 2026-07-27 with HuggingFace and NVIDIA domains proxy-blocked — chains run through first-party GitHub repos, three-way config mirrors, and cross-checked snippets, with an assumptions / re-verification ledger in each research note and per-claim confidence tiers in nvfp4.md and gpu_b300.md. Strongest cross-check: the derived GLM-5.2 NVFP4 resident matches the vLLM recipe's ~465 GB within 0.05%. The 2026-08 notes (Qwen3.8-Flash-Next, GLM-5.3-Flash) sit on a stronger tier: HuggingFace was read directly, every FP8 shard header was fetched via HTTP range requests (131 and 62 shards respectively), and the per-module byte sums reproduce each checkpoint's total_size exactly (the vLLM recipe's 172.78 GiB floor is the on-disk file total; net of the 21.08 MB of safetensors headers across the 131 shards it reproduces the same figure to the byte).

The formulas

Every equation the model computes, transcribed from src/workingset/model.py — the source of truth the explorer mirrors in JS (scripts/scenario_model.py is a compatibility alias of it). Each block names the file and function that compute it (model.py is src/workingset/model.py, predict.py and config.py sit beside it), and below each equation a table defines every symbol that appears in it: definition and code name, its value in the running example, and its unit. Conventions: B = bytes, tok = tokens, s = seconds; GB = 10⁹ B, GiB = 2³⁰ B, KiB = 1,024 B; "—" marks a dimensionless quantity.

The running example, used for every Example column: Qwen3.8-27B, FP8 weights and KV cache, on 2×H200 tensor-parallel — one replica group (dp = 1, t = 2), KV sharded (DCP) — under the reference workload and load: 64 users, 30 s think time, subagent ratio 0.10, miss rate f = 0.01, 2,000-token warm turn, 400 output tokens per request, 32,768-token chunks, MFU 0.45, MBU 0.22, 10 s TTFT budget, 40 tok/s decode floor, generation slowed at most 15% of the time, cold requests waiting at most 5 s, PUE 1.5, €0.19/kWh, €3.80 per GPU-hour. Where an equation needs a decode batch, n = 64. MoE-specific quantities, undefined on the dense 27B, get their example from Qwen3.6-35B-A3B and say so. Every example is what the function returns at its defaults: misses at flat MFU (per_pass_overhead=False, the docs/ tables' convention) and mean statistics unless a percentile is named. ws predict and the explorer price misses with the per-pass overhead and check the p95, so their figures differ in the last digits.

F.1 — Memory & warm capacity

The solved activation reservemodel.py · _act_reserve() → ACT_RESERVE

The reserve is solved from the anchor, not guessed: it is whatever is left of the anchor GPU's HBM once the 27B's weights and the anchor pool are subtracted. If the FP8 pool is ever measured directly, updating P₀ re-derives everything.

KV pool of one replica groupmodel.py · kv_pool_tokens()

Weights are sharded across the group and counted once; the reserve is scratch space and charged per GPU. R_extra exists because the H200 actually delivers ~150.75 GB against the 141 GB vendor figure the calibration uses — the anchor silently absorbs that surplus into R_act, so a part that delivers nominal bytes with no over-provision (the B300, per a measured nvidia-smi dump) must add the hidden margin back or inherit ~9.75 GB/GPU of phantom HBM. A DP deployment has dp independent pools of this size — the system total multiplies by dp only under session-sticky routing. FP16 KV doubles b_kv and the sparse models' top-k read c_top (with_kv_dtype), and is refused where the sparse path needs a quantized cache (kv_fp16_ok=False: Mistral-Large-4 and the three DeepSeek Flash models). GLM-5.3-Flash is the reverse: its FP8 KV is Blackwell-only, so check_dtype_supported() refuses that arm on the H200, which prices the FP16 one. NVFP4 swaps the weight constants (with_weight_dtype, B300 only), never b_kv.

KV layout across the TP ranksmodel.py · kv_replication() / dcp_size() / _replicated()

_replicated() multiplies b_kv, b_dec, c_top by r_kv and S, s_step by r_st before kv_pool_tokens(), warm_capacity() and decode_curves() read them, so a copied cache divides the pool and multiplies every per-step cache read by the same factor. DCP is the layout every published number assumes. A single-latent cache (GLM-5.3, GLM-5.3-Flash, the DeepSeek Flash models; the DeepSeek windows too, h_st = 1) has one head and is copied on every rank under plain TP: GLM-5.3 on 8×B300 holds 27.3×10⁶ tokens with DCP 8 and 3.41×10⁶ replicated. The 27B on 8×H200 runs DCP 2 (28.8×10⁶ tokens) or holds 14.4×10⁶ replicated.

Session cost and the Monte-Carlo fillmodel.py · _warm_once() / warm_capacity()

Cold draws (probability f, § F.5) occupy their full length and never count as warm. Percentiles are taken over 4,000 draws of 2,000 sessions (n_iter, draw); the explorer seeds the RNG from the configuration. Token equivalents of S on the other models: 3,264 (35B-A3B), 4,521 (DeepSeek-V4-Flash-0731 and its Vision-Exp sibling), 3,293 (DeepSeek-V4.1-Flash), 4,701 (Qwen3.8-Flash-Next), 11,925 (GLM-5.3-Flash at FP8 KV, its Blackwell-only arm; 5,962 at the FP16 KV H200 must run), 0 (Mistral-3.5, GLM-5.3). warm_capacity() fills with the replicated model (block above), so the pool, the state charge and the offload buffer all count per stored copy. CPU offload: a session past the pool costs u_i·b_kv + S bytes of host RAM — counted as capacity (which="all") but not as decodable (which="gpu"); with no offload buffer the two coincide, as in the example.

F.2 — Decode (HBM roofline)

TP efficiency and group bandwidthmodel.py · tp_efficiency() / effective_bw()

η(2) = 0.90 is the baseline's assumption, consistent with the TP2 measurement; η(4) = 0.81 and η(8) = 0.73 are projections. η_x is an unmeasured pessimistic guard no configuration in the study reaches (both parts are 8-GPU nodes).

Weight bytes per decode stepmodel.py · Model.w_decode(n)

The 27B is dense, so the MoE rows take their example from the 35B-A3B, where W_sh = 1.94×10⁹ B and W_dec(64) = 34.15×10⁹ B linear / 29.93×10⁹ B coverage. The two unions bracket reality: linear is the no-overlap upper bound on bytes, coverage the expectation under independent uniform routing (real routing is correlated, so it grows slower). The linear union's slope break sits at n = w_E/w_e: 32 on the 35B-A3B and GLM-5.3, 36 on GLM-5.3-Flash, ≈ 42.7 on DeepSeek-V4-Flash-0731 and Vision-Exp, 51.2 on Qwen3.8-Flash-Next, 64 on DeepSeek-V4.1-Flash.

Bytes and speed of one decode stepmodel.py · decode_curves()

j indexes the n active sequences; percentiles are over 3,000 sampled batches. In the example the step splits into 30.9 GB weights + 83.8 GB KV + 10.1 GB state. p50(·) is the median over draws. The draft passes add their own bytes in the step time below; DecodeConstants.roofline() (bytes at full bandwidth, no compute or fixed terms) gives 282.5 tok/s here. The DeepSeek Flash models set state_step_bytes to what a step actually writes (1.05×10⁶ B on V4-Flash-0731 and Vision-Exp, 45.7×10³ B on V4.1-Flash): their windows are fixed storage whose reads are already in c_top.

Sparse-attention decode read (sparse-attention models only)model.py · kv_decode_* in decode_curves()

Applies to the six models that set kv_decode_bpt: GLM-5.3 (DSA), GLM-5.3-Flash (DSA over NoPE-MLA), Qwen3.8-Flash-Next (QSA), DeepSeek-V4-Flash-0731 and Vision-Exp (CSA) and DeepSeek-V4.1-Flash (CSA2); their constants are in § F.7. The min(·) term keeps a sequence shorter than the window from being charged tokens it does not have. The dense 27B of the running example never enters this branch.

Speculative decoding: tokens per stepmodel.py · tokens_per_step()

Acceptance is measured on the 27B and, outside this repo, on GLM-5.3 (InferenceX: 2.94 tokens per step at 3 drafts, α = 0.797), Mistral-Large-4 (its vLLM recipe: 2.79–2.98 at 3 DSpark drafts, α = 0.777) and Kimi-K3.1-Pro (InferenceX on B300, SPEED-Bench coding: 3.08 at 5 DSpark drafts, α = 0.721). Every other model carries its nearest measured sibling's figure, flagged unmeasured: the Qwen MTP heads the 27B's, GLM-5.3-Flash GLM-5.3's, the DeepSeek Flash DSpark drafters Mistral-Large-4's. Mistral-Medium-3.5 ships no draft model (k = 0, m = 1). research/decode-model.md.

The steady decode point — the batch a load actually producesmodel.py · steady_decode_point()

Little's law on the decode phase alone, n = λ_g·N_out / v_user(n), multiplied through: delivered output tok/s against demanded. The left side is the aggregate curve, strictly increasing in n, so the crossing is unique; it is bisected on the integers and interpolated across the last one. Past N_res the engine is evicting, so the point is reported saturated rather than priced at a batch HBM cannot hold. Mean field: v is read at the mean batch, and v is convex in n, so this is the conservative side (Jensen). Decode only: the prefill chunks sharing a pass are the ITL spike of § F.4. It assumes ρ < 1 upstream. The stress readouts read the same curve at the warm population (§ F.4: n_w = 194, 27.6 tok/s).

Step time: bytes, verify compute and fixed costsmodel.py · decode_step_seconds()

One equation prices every decode figure. η is the 0.36 a held-batch sweep to n = 64 at two contexts read (Qwen3.6-27B, 4×H200 TP4, 3 drafts) times 0.81, the TP4 haircut it had been divided by; τ_rec is what that sweep's 2.7 ms intercept leaves after the draft passes and the two priors. It reproduces the sweep's seven batched rungs within 4%. Above ~64 sequences the measured step jumps for an unknown reason, which this form does not price. ws calibrate fits the four constants from a decode sweep. research/decode-model.md.

F.3 — Prefill (compute roofline)

FLOPs of one forward passmodel.py · prefill_flops()

In the example the GEMM term is 1.61×10¹⁵ and the quadratic attention term 0.21×10¹⁵ FLOP. The cache spares recomputing the H tokens' keys and values, not attending over them — hence the cross term. The pair count telescopes: however a context is chunked, its total FLOPs are identical. Linear-attention (DeltaNet) layers have no quadratic term and live inside N_p.

Group compute and pass timemodel.py · peak_flops() / prefill_seconds()

MFU is stated in this model's convention (it divides C_g, which already carries η(t)); its bracket rests on measurements not published here (research/prefill.md §1). It is the softest prefill input: it scales every prefill figure almost proportionally, and the [0.30, 0.55] bracket spans ~1.8×. Reusing η(t) for compute is an assumption flagged in the code — the real prefill TP haircut is probably gentler. Roofline ridge C_fp8/BW_HBM: 412 FLOP/B (H200), 563 (B300); prefill sits orders of magnitude above it, decode far below — which is why the two interfere when batched.

Per-pass overhead and the solved MFU ceilingmodel.py · prefill_overhead_seconds() / mfu_ceiling() / prefill_pass_seconds()

μ_c is solved so that a first cache-empty chunk of chunk₀ tokens nets exactly the anchor MFU — by construction t_pass = t_pf there, as the example shows. At small chunks the overhead no longer amortises and the effective MFU falls (mfu_effective() — the explorer's chart E); warm hits and the ITL-spike chunk stay on the marginal t_pf pricing because their pass's weight stream is already paid by the decode batch they join. miss_context_seconds() sums t_pass over a whole chunked context, each chunk at the cache it lands on; idle_cold_ttft() is that sum for one cold prompt on an idle group — 0.32 s at 10,000 tokens, 1.84 s at 50,000, 4.28 s at 100,000, 9.47 s at the 180,000 cap (idle_cold_ttft_table()) — a property of the deployment that no limit uses.

Expected miss and hit costmodel.py · cold_request_seconds() / warm_request_seconds()

E[L²] is 1.74× E[L]²: the quadratic term must be priced on the heavy tail, never on the mean draw. Both example moments are the reference workload's (user + subagent mixture, § F.5). The published tables use the flat-MFU form; the explorer and ws predict price the parenthesised per-pass-overhead variant (1.571 s here) — at chunk₀ the two agree within 1%.

The hypothesis numbersmodel.py · thrash_ratio() / max_cold_rate() / prefill_duty() / breakeven_miss_rate()

The thrash ratio is the study's founding number — ≈ 18–19× near-identically across the architectures priced in docs/scenarios.md § 8, and 17.8–20.0× across all nine models at the reference workload. In the flat-MFU form both MFU and the GPU part cancel out of it. f* solves ρ(f) = 1, which is linear in f; it is a limit, not an operating point — burst tolerance (§ F.4) reaches zero exactly there.

F.4 — Queueing & bursts

Service moments and the M/G/1 queuemodel.py · prefill_service_moments() / queue_wait_seconds() / prefill_ttft_seconds()

E[W] diverges at ρ ≥ 1. A miss's service time runs as L² on a lognormal L, so the stream's squared coefficient of variation E[S²]/E[S]² − 1 lands at 5.5 in the example (4.7–8.3 across the nine models at the reference workload: DeepSeek-V4.1-Flash lowest, GLM-5.3 highest) where an exponential server would sit at 1 — E[S²] is exactly what the duty cycle cannot see. FCFS and PS bracket vLLM's chunked-prefill scheduling; neither end is uniformly optimistic, and which is which flips by request class.

The TTFT ceiling, closed formmodel.py · max_users_latency() / ttft_service_quantile() / sla_miss_rate()

Solving TTFT_FCFS = SLA for λ, possible in closed form because the moments do not depend on the arrival rate. Always λ_sla < 1/b for any a > 0 — the algebraic statement of "the queue diverges before the server saturates". sla_miss_rate() inverts the same curve for f instead of λ, by bisection (TTFT is monotone in f). The proxy adds a mean wait to a percentile service time, because the model has no distribution for the wait; how far it sits from the true percentile is unmeasured. Since issue #94 this is operating_point()'s ttft_ceiling, reported beside the ceilings that decide and no longer one of them: at a realistic miss share the TTFT percentile sits in the miss tail and reports prompt length more than load. Its two replacements follow the burst block.

Bursts, drains, and what the warm users losemodel.py · burst_drain_seconds() / spike_tolerance() / itl_spike() / spike_token_debt()

A backlog drains at (1 − ρ) seconds of work per second because standing traffic keeps arriving; the burst's last request gets its first token at T_drain under either discipline. B* inverts the drain equation, is linear in the SLA, and reaches zero exactly at f* — with B = B* the drain equals the budget by construction, as the example shows. The spike chunk is priced mid-re-prefill (H = E[L]/2); during a drain the scheduler has a chunk to place in every pass, so t_mix is the steady state for the whole drain, and Δ_tok integrates the difference. t_dec is read at the stress batch n_w; slowed generation below prices the decode step at the floor instead.

Slowed generation — what a decoder feels while a cold prompt prefillsmodel.py · slowed_steps() / slowed_share() / max_users_slowed() / slowed_stretch()

The decode step is priced at the floor rather than at the load's own batch: the floor is the slowest step the decode SLO admits, and a fixed step keeps the share linear in λ, so the ceiling is closed form; the chunk pass dominates the mixed step, so the choice moves the share little. Approximation: warm-turn prefills are left out of the share, since a few thousand tokens slow a step far less than a full chunk. The stretch prices each chunk at the cache it actually attends over (prefill_context_seconds()), so a long prompt's later chunks cost more than the E[L]/2 chunk the slowed speed quotes. U_slow is inf when nothing misses or when λ_slow lies past the prefill wall (saturation binds first).

Cold-request wait — the queue in front of a cold prefillmodel.py · cold_wait_seconds() / wait_quantile() / max_users_cold_wait()

FCFS makes every request wait the same, so a cold request's wait is the queue's. The tail keeps the exact atom P(W = 0) = 1 − ρ and the exact mean and is exact for exponential service; this workload's heavy-tailed miss service makes the true tail heavier, and vLLM is neither FCFS nor processor sharing, so read W_p as the FCFS end of the bracket. ws predict and the explorer check the p95; operating_point() reads the mean unless given ttft_percentile.

F.5 — Workload & the user conversion

Prompt lengthsmodel.py · Workload.sample()

The user-class (M, σ) is a maximum-likelihood fit on 1,850 real request lengths; the subagent class is assumed. Lognormal mean = M·e^σ²/2 = 1.39·M at σ = 0.81 — the mean, not the median, drives warm capacity and prefill cost. A cold draw's prefix is set to 0 (it matches nothing).

Users ↔ request rates, and the ceilings' common unitmodel.py · sessions_from_headcount() / request_rate() / max_users_saturation() / max_users_latency() / closed_request_rate(); config.py · RunConfig.users_per_group()

These conversions are the planner's two stated assumptions, and both are limitations (docs/scenarios.md § 9). T_think is the full interval, response time included (measured cycle 43.3 s; the 30 s reference is the conservative side); the closed variants take waiting-only Z_think and let the model supply its own response time, so a slower deployment lightens its own arrival rate. The population layer is optional: sessions_from_headcount() gives the system total and users_per_group() divides it by dp before any ceiling sees it; a person holding κ sessions but attending one at a time is the lower-load end, multiplying cache use but not request rate. The other ceilings come from other sections: max_users_decode() = the n where p50 v_user hits the floor (example: 74), and the two latency limits of § F.4 (slowed generation 247; cold-request wait 267 at the mean, 234 at p95), as published, at the configured miss rate. max_users_cache(), the warm p5 user-class count of § F.1 (example: 177 at its 400-draw default), is no longer a ceiling: operating_point() passes it as warm_users, so past the knee W/(1 − f) = 179 users each latency ceiling is priced at the miss share there (the hit rate at the load). It takes the minimum of four — decode, cold wait, slowed, saturation, here 74, 183, 179 and 185 — and decode (74) binds; the TTFT ceiling (184) is reported beside them and does not decide. At 64 users 99.0% of requests hit. ws predict also caps the decode batch at max_num_seqs (next block), which lifts decode to 186 and leaves slowed generation binding at 179, just past the knee and two users past W (it prices misses with the per-pass overhead and the p95).

The max_num_seqs cap on the decode ceilingmodel.py · recommended_max_num_seqs() / decode_slot_mean() / max_users_decode_slots() / cap_decode_ceiling(); applied in predict.py · predict()

The engine never decodes more than c_seq sequences at once. At a given load the batch that decodes is the steady point's (§ F.2), not the warm population, and it fluctuates around that mean as an infinite-server queue's occupancy — Poisson for Poisson arrivals — so the cap binds where the batch's p99 reaches it, and past that requests queue for a slot. A cap at or below the bandwidth crossing keeps every batch at or above the floor, so that ceiling drops out, as in the example. Burstier arrivals than Poisson widen the batch: the percentile is the optimistic side. Under a closed loop T_think is the waiting time alone, so the open conversion undercounts. operating_point() alone does not apply the cap; predict() and the explorer do.

F.6 — Power & cost

Duty split and drawmodel.py · power_draw()

The three states partition time — no double counting. The phase watts are measured Hopper fractions of TDP (0.90 / 0.55 / warm idle); every B300 figure is extrapolated. The GPU term carries ±20–25%.

The billmodel.py · energy_cost()

The literals are unit conversions: /1000 W → kW; /h_mo then /3600 turn the monthly bill into a cost rate in €/s; ×10⁶ tokens → millions; the denominator λ_g·R·N_out is the system's output-token rate in tok/s, R the replica groups (Topology.replicas, written dp elsewhere on this page; renamed here so it does not read as the duty d_p). The electricity line is linear in both PUE and c_kWh (asserted in the self-checks), so tariff and facility scenarios are one multiply. €_user and €_Mtok divide the whole bill: a €/Mtok that priced the watts but not the silicon read an order of magnitude too cheap. The hardware line is GPU-hours at an on-demand rental list price (cross-provider medians, September 2026, at EUR/USD 1.16), a market snapshot and not a cost of ownership. A rental rate includes power, so for a rented GPU the electricity line double-counts (6% of the bill here); for owned hardware, enter an amortised €/GPU-hour and both lines apply.

F.7 — Constants

Global constants

GPUsmodel.py · GPUS = {…}

Watts per GPU; decode/prefill = 0.55/0.90 × TDP (measured Hopper fractions). Every B300 power row is extrapolated — no published Blackwell Ultra power-state measurement exists. C_fp8 is the dense rate (vendor sheets lead with 2:4 sparsity, which no LLM GEMM reaches). c_gpu is the on-demand rental list price (getdeploying.com cross-provider medians, September 2026: $4.40 and $7.89 at EUR/USD 1.16), a market snapshot.

Models (FP8 serving checkpoints)model.py · MODELS = {…}

GB = 10⁹ B; the key is the MODELS entry (and [deployment] model); GLM-5.3 kept GLM-5.2's key, its config and checkpoint being identical. *The two DeepSeek-V4-Flash rows' d is 26,624 summed across their 41 mixed-class layers, stored as attn_d = 26,624/41; DeepSeek-V4.1-Flash prices only its three encoder indexers (QK only, ℓ = 3). The DSA and QSA rows (GLM-5.3, GLM-5.3-Flash, Qwen3.8-Flash-Next, Mistral-Large-4, Kimi-K3.1-Pro) price the quadratic term as dense attention over their sparse layers, an upper bound. m is measured on the 27B only; 1.70 elsewhere is the transplanted fit, 1.00 means no MTP module. The DeepSeek rows' S is fixed window and compressor storage; a decode step moves s_step = 1,048,768 B of it on V4-Flash-0731 and Vision-Exp and 45,696 B on V4.1-Flash, not 2S. Mistral-Large-4's S is its DSpark drafter's latent windows, which the speedup m already prices, so s_step = 0. NVFP4 checkpoints swap (W_res, W_sh, w_e, w_E) per nvfp4_w on every row but DeepSeek-V4-Flash-Vision-Exp, which has none (research/nvfp4.md, research/nvfp4_2026-09.md). FP16 KV doubles b_kv; it is refused on Mistral-Large-4, DeepSeek-V4-Flash-0731, DeepSeek-V4-Flash-Vision-Exp and DeepSeek-V4.1-Flash, and it is the only arm GLM-5.3-Flash prices on the H200. †GLM-5.3's FP8 row depends on the GPU: the constants are B300's (FlashInfer, plain fp8_e4m3, 576 B per latent); on the H200 vLLM's FlashMLA stores fp8_ds_mla rows of 656 B, so b_kv = 54,728 B and c_top = 104.8 MB there (with_gpu, kv_hopper_fp8). Kimi-K3.1-Pro's row follows the same rule: 22,192 B and 32.2 MB off Blackwell, where no 8-GPU node holds its weights.

Sparse-attention decode constantsmodel.py · kv_decode_*

Storage stays b_kv; these price the decode read only (§ F.2). FP16 KV, where allowed (Qwen3.8-Flash-Next, GLM-5.3-Flash), doubles c_top — the top-k gathers read main-KV bytes — and leaves the quantized indexer scan b_dec as it is.

Glossary

MFU Model FLOP Utilisation

What fraction of a GPU's advertised arithmetic throughput a real kernel achieves. An H200 is rated 1,979 TFLOP/s of dense FP8; no real prefill reaches that, because time also goes on memory stalls, attention kernels, collectives and launch overhead. MFU = achieved ÷ advertised. This study uses 45%, with a plausible 30–55% bracket resting on measurements not published here, stated against the model's own TP-efficiency-adjusted denominator (research/prefill.md §1). Still the softest input on the page: it scales every prefill time and every derived ceiling almost proportionally. That is why B* carries a band instead of a single number. It does not affect warm capacity, which is an arithmetic-free byte count.

TTFT time to first token

How long a request waits before its first output token — queueing plus its own prefill. A cache miss re-prefills its whole context; a hit prefills only the new turn, but still queues behind whatever misses are in front of it.

Prefill duty the fraction of the machine spent prefilling

Arrival rate × mean prefill time. At 100% the queue grows without bound. It is an average, which is exactly why it cannot see the two things below.

f* the saturation miss rate

The cache-miss rate at which prefill duty reaches 100%. Not a target: it is the point at which burst tolerance has already fallen to zero.

B* cold-spike tolerance

How many cache misses can arrive at the same instant and still have the last of them answered inside a fixed 10 s first token. The miss-rate slider models misses arriving independently; B* models them arriving together, which is how a template deploy or a cache wipe actually behaves.

n@load the steady-state decode batch

The mean number of sequences decoding at any instant at the load you set — Little's law, arrival rate × seconds spent decoding, solved where the aggregate decode curve delivers exactly the output tokens the load demands (method). It is an expected occupancy, not a setting: max_num_seqs stays at the ceiling so the batch can absorb bursts. Far below the warm population, because arrivals are open-loop — which is why the per-user speed beside it is far above the all-warm stress figure.