The AI hardware race pivots from training FLOPs to inference dollars
Summary
Across summer 2026 the competitive front in AI infrastructure moved to the cost of serving a token. AMD and Cerebras (July 23) shipped a disaggregated inference system -- Helios racks for prompts and context, a Cerebras wafer-scale engine for token generation -- claiming up to 5x tokens per second per watt over a WSE-only setup. Google split its TPU line into a training chip (8t) and an inference chip (8i) that claims up to 80% better performance-per-dollar than Ironwood. Groq raised $650M (June 22) to build inference capacity. And AdaMX (Aug 4), a 22nm accelerator prototype, buys back 82-83% of the accuracy 4-bit quantization loses for about 1% more energy.
The scarce resource is no longer peak training throughput but dollars, watts, and memory bandwidth per token generated. AMD and Cerebras split those jobs across two chips; Google split them across two TPUs; Groq is buying megawatts; AdaMX squeezes them at the number format. Every headline figure here is inference economics, not model quality.
View data as table
| vs MXFP4, commonsense | 83% | at ~1% extra system energy |
|---|---|---|
| vs MXFP4, MMLU | 82% | across 3B-70B models |
| vs NVFP4, commonsense | 43% | NVFP4 is the stronger baseline |
| vs NVFP4, MMLU | 27% | smaller headroom over NVFP4 |
Four moves, one front
- AMD + Cerebras, disaggregated inference (July 23): AMD Helios racks handle prompts and long context while the Cerebras Wafer-Scale Engine generates tokens -- AMD and Cerebras model the pair at up to 5x tokens per second per watt versus a WSE-only setup (Kimi 2.6 1T), on Cerebras Cloud in H2 2026.
- Google TPU 8t / 8i, a training/inference split (Cloud Next, deep dive Apr 22): 8t trains at 9,600 chips per superpod; 8i serves, claiming up to 80% better performance-per-dollar than Ironwood at low-latency targets for large mixture-of-experts models.
- Groq's $650M inference raise (June 22): led by Disruptive and Infinitum, earmarked to expand Groq's inference cloud toward 200 MW by the end of 2027 -- capital aimed at serving capacity, not training.
- AdaMX, efficiency at the bit level (Aug 4): a 22nm accelerator prototype that picks the precision-recovery scheme per block, removing 82-83% of MXFP4's 4-bit accuracy loss for about 1% extra energy across models from 3B to 70B (and up to 96% of FP16 accuracy on Gemma-4 12B vision-language).
- The through-line: the money and the engineering have moved downstream -- the figure that now decides the winner is dollars, and watts, per token served, not FLOPs per training run.
Two corrections to widely circulated framings. AMD's page makes no tokens-per-dollar comparison to Nvidia's Vera Rubin NVL72 -- its only benchmark is up to 5x tokens/sec/watt against a Cerebras-WSE-only baseline, from AMD/Cerebras modelling. And Google's deep dive states no '121 FP4 exaflops per superpod' figure; its stated per-chip peak is 10.1 FP4 PFLOPs for 8i (12.6 for 8t). The TPU deep dive dates to Google Cloud Next (Apr 22, 2026), earlier than the other three items. All performance figures are vendor- or author-reported and, for these releases, not independently replicated; AdaMX's results come from a research prototype.
Sources(4) ▾
- AMD (newsroom), AMD and Cerebras deliver a disaggregated inference solution (2026-07-23) — AMD's own announcement (Advancing AI 2026) of a disaggregated inference system pairing AMD Helios racks with the Cerebras Wafer-Scale Engine. Source for the July 23, 2026 date, the disaggregated split (Helios processes prompts and large context windows; Cerebras WSE accelerates memory-bandwidth-intensive token generation at ultra-low latency), the 'up to 5x higher tokens per second per watt' figure versus a Cerebras-WSE-only configuration (AMD Performance Labs + Cerebras modelling, July 2026, Kimi 2.6 1T model), and Cerebras Cloud availability in H2 2026. Fetched and read in full. NOTE: the page contains NO comparison to Nvidia's Vera Rubin NVL72 and no tokens-per-dollar claim; its only benchmark is the 5x tokens/sec/watt figure against a WSE-only baseline. newsroom.amd.com
- Google Cloud (blog), TPU 8t and TPU 8i: a technical deep dive (2026-04-22) — Google Cloud's technical deep dive on its TPU 8t/8i generation, announced at Cloud Next. Source for the training/inference split (8t: pre-training and embedding, 9,600-chip superpod, 3D torus topology; 8i: post-training and high-concurrency reasoning, highest on-chip SRAM, Collectives Acceleration Engine, Boardfly serving topology), the 'up to 80% performance-per-dollar improvement over Ironwood TPU' for 8i at low-latency targets on large MoE models, and the peak FP4 figures (8i 10.1 PFLOPs, 8t 12.6 PFLOPs per chip). Fetched and read in full. NOTE: the page states NO '121 FP4 exaflops per superpod' figure; the only fabric-level figure is 'over 1.7K ExaFlops' across 134,000+ TPU 8t chips. cloud.google.com
- Groq (newsroom), Groq raises $650M to scale its AI inference cloud business (2026-06-22) — Groq's own funding announcement. Source for the $650M raise dated June 22, 2026, the lead investors (Disruptive and Infinitum), and the stated purpose: accelerate the expansion of Groq's AI inference cloud and scale toward 200 MW by the end of 2027. Fetched and read in full. groq.com
- Luo et al. (arXiv:2608.03867), Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference (2026-08-04) — The AdaMX (Adaptive Microscaling) technical paper. The load-bearing primary source for: a 22nm FD-SOI AI accelerator prototype that selects the precision-recovery scheme per block and representation per operand; adds about 1% system energy versus an otherwise identical MXFP4 accelerator; removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU across LLMs from 3B to 70B; removes 43% (commonsense) and 27% (MMLU) of the NVFP4 loss; and on Gemma-4 12B keeps up to 96% of FP16 accuracy on vision-language benchmarks. Abstract fetched and read. arxiv.org
Comments
Always open. Logged-in readers can annotate paragraphs in place.
The competitive front in AI infrastructure has moved from training FLOPs to the cost of serving a token. Four moves in one summer say so: AMD and Cerebras shipped a disaggregated inference system (July 23), Google split its TPUs into a training chip and an inference chip that claims up to 80% better performance-per-dollar, Groq raised $650M to build inference capacity (June 22), and a low-bit accelerator, AdaMX, buys back 82-83% of the accuracy 4-bit quantization loses for about 1% more energy (Aug 4).