- Published on
Microsoft Commits Maia to Multi-Gigawatt Inference, and its Efficiency Metric Draws Questions
- Authors
- Name
- DeepHorus
Microsoft will scale its Azure Maia family to multi-gigawatt deployments starting in 2027, making Maia its primary inference system for Microsoft products and services. Maia 200 is already in production for Copilot and the MAI team, with GitHub Copilot and Microsoft Foundry to follow by year-end. It is deployed across the United States, Europe, and Australia, and Microsoft claims up to 1.7 times more tokens per watt for built-in AI in Teams and Copilot than leading GPU infrastructure.
Maia 300, entering production deployments in 2027, is inference-first, targeting long-context, memory-intensive, and decode-heavy work rather than training. Each accelerator carries eight HBM4 stacks, delivering 288 GB and 18 TB/s within a 950-watt TDP. A rack holds 96 accelerators in 167 kW across 24 four-accelerator trays, reaching 27.6 TB of HBM and 1.7 PB/s aggregate.
Compute chiplets use TSMC N2P, with memory-controller and I/O functions on separate dies. Microsoft says this raises compute FLOPs by 36% and contributes to 2.5 times the memory bandwidth and roughly twice the HBM bandwidth per watt of Maia 200. Memory organization moves to a NUMA layout across chiplets so software can place data near the compute using it.
The networking is notably opinionated. Maia’s AI network controller implements a custom AI transport on Ethernet, which Microsoft describes as “one fabric rather than separate scale-up and scale-out domains.” Accelerators initiate communication to any peer in a superpod without involving the host or a discrete scale-out NIC. Two accelerators are directly linked within a tray, a switched fabric joins all 96 in a rack, and optical second-tier switches connect up to 128 racks into a 12,288-accelerator superpod.
That fabric carries a specific inference pattern. Prefill and decode workers are sized and placed independently, and after prefill completes, KV state moves directly to selected decode workers over the same scale-up fabric. A spare tray in one rack can offset a failed tray in another transparently to software kernels.
Microsoft expects Maia 300 to deliver 40% more tokens per megawatt than leading GPU-based systems, up to seven times rack-scale token throughput, and more than 2.5 times the performance per watt of Maia 200. The efficiency claim drew criticism. Rick Bullotta called “tokens per watt” an absurd metric that lacks a uniform token definition and rewards verbosity. Michel Floyd asked whether it should be tokens per second per watt or tokens per KWh. Vinay Bansal, building a competing inference accelerator, questioned the context length, noting long-context agents are memory-bound, not compute-bound.
