A.I.
A.I.
A next-token engine with an post-train and a tool loop, sitting on a liquid-cooled rack.
Modern language models are next-token predictors shaped by staged training (especially reinforcement learning from verifiable rewards) and tool use. This page explains what each stage actually buys, what still discriminates capability, and the hardware you actually purchase.
Don’t get this wrong
- 1 Chain-of-thought is not the model’s thoughts.
- 2 Bigger context ≠ it read everything.
- 3 RLHF made it preferred, not truthful.
- 4 Reasoning models are spiky on verifiable domains, not generally smarter.
- 5 A SWE-bench pass is not merge-ready in your SDLC.
- 6 Production agents loop. Budget human checkpoints.
- 7 Native multimodal samples tokens/frames. “1M context” is not “watched the hour.”
- 8 You will not reliably “see it in the logs.” Hidden or unfaithful CoT is the default.
How it works
Training is not one process. Each stage buys something; none of them buy the rest.
| Stage | What it buys | What it does not |
|---|---|---|
| Pretraining | Fluency, frequent facts, code patterns, a prior RL can search | Honesty, refusal, tool use, calibration on rare facts |
| Mid-training | Specialist density (code/math), long-context stretch, a better SFT/RL init | A product |
| Supervised fine-tuning () | Assistant format, style, behaviors you can write down | Discovery of better-than-human strategies |
| Reinforcement learning from human feedback () / RLAIF / direct preference optimization () | Helpfulness, harmlessness, chat preference | Truth over agreeableness |
| Reinforcement learning from verifiable rewards () | Long search on verifiable domains; test-time compute as a knob | Writing quality, SWE-scale engineering by itself, judgment on unverifiable work |
| Distillation | The teacher’s visible procedure at lower inference cost | The teacher’s internal search, automatically |
Distill view of training. Accent on RLVR.
Pretraining
Buys. Fluency, frequent facts, code patterns, a prior RL can search. Does not. Honesty, refusal, tool use, calibration on rare facts.
Mid-training
Buys. Specialist density (code/math), long-context stretch, a better SFT/RL init. Does not. A product.
Supervised fine-tuning (SFT)
Buys. Assistant format, style, behaviors you can write down. Does not. Discovery of better-than-human strategies.
RLHF / RLAIF / DPO
Buys. Helpfulness, harmlessness, chat preference. Does not. Truth over agreeableness.
Reinforcement learning from verifiable rewards (RLVR)
Buys. Long search on verifiable domains; test-time compute as a knob. Does not. Writing quality, SWE-scale engineering by itself, judgment on unverifiable work.
Distillation
Buys. The teacher’s visible procedure at lower inference cost. Does not. The teacher’s internal search, automatically.
Treat the model like a pianist who has only ever been asked to play the next note.
There is no score in the head, and no inner room where a world is being simulated. Text is chopped into tokens. Each token becomes a vector. A stack of transformer blocks updates those vectors in a shared residual stream. Attention moves information between positions. The last state is projected into a vocabulary distribution, a token is sampled, and the loop repeats. What looks like reasoning is that same loop, sometimes stretched into a long chain of thought by post-training. The chain is generated text, not a window into thoughts.
Chat vision-language models sit on that loop. They are not world models. World models (Genie-class, Cosmos-class) predict future states. They last minutes, not hours, and they are not a digital twin of anything you run.
The 2025–2026 jump is plus test-time compute: drill on domains a checker can grade, then spend extra tokens at inference. Math and code move. Writing, long-horizon engineering, and calibrated judgment on unverifiable work do not come along for the ride.
An agent is the model calling a tool, reading the observation, and calling again, with an autonomy slider in the app (Claude Code, Cursor, Codex-class). Recursive self-improvement, in the architecture-rewrites-itself sense, is not demonstrated in production.
Failure modes are incentives. Hallucination is, in part, a test-taking problem: OpenAI argued in September 2025 that evals scoring zero for “I don’t know” teach bluffing. Sycophancy is what preference data buys. Scaffold and contamination move leaderboards. Dual-use classifiers are a routing layer in the API, not a footnote.
Serving mechanics Prefill vs decode · KV cache · MoE routing · FlashAttention
Mixture-of-experts () is a roster with a dispatcher; only a subset fires. DeepSeek-V3 is 671B total / about 37B active (lab-published). Qwen3.8-Max launched as an Alibaba API SKU on 3 August 2026 (2.4T / 95B active); open weights followed around 12 August. Total is not compute per token. Routing is a new outage class.
reads the prompt and fills the key-value cache. Decode emits one token at a time and is memory-bandwidth bound: every new token reloads the of everything before it. Advertised context is an architectural number. Usable recall is empirical. A million-token window is a whiteboard size, not a guarantee the model read the board.
FlashAttention is an IO-aware attention kernel (same math, less traffic). PagedAttention is vLLM paging the KV cache in blocks so many sequences can share GPU RAM. Different jobs. Speculative decoding is a fast draft plus a target check; NVIDIA and vLLM do not treat it as lossless in all settings.
What still counts as capability
Still discriminating
- Terminal-Bench 3.0
- OSWorld 2.0
- ARC-AGI-3
- METR
Saturated
- SWE-Verified
- GPQA Diamond
Claude Opus 5 is $5 / $25 per million tokens, 1M context. Sonnet 5 is $2 / $10. Fable 5 is $10 / $50. Dual-use classifiers and 30-day retention are Mythos-class terms, including Mythos 5; Fable is the generally available SKU in that class, and Mythos 5 is trusted access with the classifiers lifted. Grok 4.6 (xAI, 12 August 2026) is 500k context; $2 / $6 per million tokens when the prompt is below 200k, and $4 / $12 for the whole request at or above 200k prompt tokens (xAI pricing). GPT-5.6 Sol, Terra, and Luna, and Gemini 3.x, are closed SKUs; prices and parameter counts for those lines are not published here.
SWE-Verified and GPQA Diamond are saturated. OpenAI stopped reporting SWE-Verified for frontier launches. Epoch found scaffold choice alone moved Verified by about 11–15 points. What still discriminates: Terminal-Bench 3.0, OSWorld 2.0 binary success, ARC-AGI-3, and METR’s time horizon.
The failure mode is state, not mouse-clicking. The working layer is a coding agent with a human on the irreversible step. Unattended multi-hour computer use, and “the model that thinks for days,” are still demo.
The rack, not the GPU
You buy a liquid-cooled 72-GPU rack, not a GPU.
Bind order on the drawing: grid → → → rack.
Google’s line is TPU, Ironwood / TPU7x-class, Google-specific. For open-weight serving on NVIDIA, vLLM is the default.
The unit you buy is a liquid-cooled 72-GPU rack (NVIDIA GB200 or GB300 NVL72), not a GPU. Blackwell is the 2026 fleet. Rubin is announced as next; treat a GA date as unconfirmed until it is fleet.
This essay’s rank of the bind (the glossary lists the same four unranked): grid interconnect, then Chip-on-Wafer-on-Substrate () and other advanced packaging, then high-bandwidth memory (), then liquid-cooled rack integration.
Glossary on this page Key terms, brief definitions.
- Drill on domains a checker can grade, then spend extra tokens at inference. Does not buy writing quality, SWE-scale engineering by itself, or judgment on unverifiable work.
- Assistant format, style, behaviors you can write down. Does not buy discovery of better-than-human strategies.
- Helpfulness, harmlessness, chat preference. RLHF made it preferred, not truthful.
- A roster with a dispatcher; only a subset fires. DeepSeek-V3 is 671B total / about 37B active (lab-published). Total is not compute per token. Routing is a new outage class.
- Prefill reads the prompt and fills the key-value cache. Advertised context is an architectural number. Usable recall is empirical.
- Prefill fills the cache. Decode emits one token at a time. Decode is memory-bandwidth bound: every new token reloads the KV cache of everything before it.
- CoWoS and other advanced packaging sit in the bind after grid interconnect. Then HBM, then liquid-cooled rack integration.
- The third constraint in this essay’s rank of the bind, after CoWoS. FlashAttention is the same attention math with less HBM traffic.
- EAR and deemed export are a GRC pointer, not legal advice here. Export control is live.
- GPAI duties have applied in the EU since 2 August 2025. Commission and AI Office enforcement and fines since 2 August 2026, with a systemic-risk presumption at 10²⁵ FLOP.