- Published on
GLM5.2 on AMD MI355X at 2626 tok/s/node at over 2x lower cost than Blackwell
- Authors
- Name
- DeepHorus
Wafer, an AI infrastructure company, has achieved state-of-the-art inference performance for the GLM5.2 frontier model on AMD’s MI355X GPUs, delivering 2626 tokens per second per node and 213 tokens per second single-stream throughput at over twice the cost efficiency of NVIDIA’s Blackwell architecture. The breakthrough demonstrates that AMD hardware, when combined with advanced optimization, can compete with NVIDIA in high-demand inference workloads while significantly reducing operational costs. This milestone comes as the AI industry faces soaring demand for tokens amid rapid model releases, making cost-effective inference a critical priority.
The demand for large language model inference is skyrocketing, with frontier models such as Claude Fable, GLM5.2, and Minimax M3 being released nearly every other week. This surge has outpaced the supply of NVIDIA’s Blackwell GPUs, driving up prices and making tokens increasingly expensive. Enterprises are seeking alternatives that maintain performance without the premium cost, and AMD’s Instinct MI355X has emerged as a promising contender.
AMD’s MI355X GPUs are approximately 2.75 times cheaper per unit than NVIDIA’s B300 Blackwell equivalents, offering comparable hardware specifications. However, AMD has traditionally faced a software disadvantage, lacking the day-zero support and optimized libraries that NVIDIA provides. Without significant engineering effort, achieving peak performance on AMD hardware has been challenging, often requiring weeks of custom kernel and model optimization.
Wafer’s recent optimization of GLM5.2 on AMD MI355X yielded an aggregate throughput of 2626 tokens per second per node at 2.4 requests per second with a ≤5-second time-to-first-token, achieving about 80% of the performance measured on a B200 GPU. Crucially, this performance came at over twice the cost savings, making it a compelling option for budget-conscious deployments. In single-stream workloads, Wafer recorded 213 tokens per second on 10k input tokens and 1.5k output tokens, following Artificial Analysis standards.
The team achieved these results by quantizing the base bf16 GLM5.2 model to MXFP4 using AMD’s Quark tool, maintaining lossless accuracy on benchmarks such as GPQA-Diamond and GSM8K compared to the official FP8 quantization. They chose the sglang inference engine for its native support of MXFP4 and lower friction with AMD’s ROCm stack. To further boost throughput, Wafer enabled speculative decode, resolving two critical bugs—including a module naming mismatch in the multi-token prediction head—that yielded nearly a 3x improvement in single-stream performance.
These results underscore the growing viability of AMD GPUs for high-throughput AI inference, particularly as agent-driven models automate kernel optimization and reduce the engineering overhead once required. For enterprises grappling with rising NVIDIA costs, AMD offers a path to maintain performance while halving expenditures. Wafer’s success also highlights that software support gaps are narrowing, with frameworks like sglang and tooling from AMD increasingly closing the gap.
Looking ahead, as AI models become more complex and token demand continues to surge, the price-performance advantage of AMD’s Instinct line will likely attract more providers and enterprises. With ongoing improvements in ROCm ecosystem and community optimizations, AMD could shift from an underdog to a mainstream choice for inference. Wafer’s demonstration serves as a proof point that the era of cheap, high-performance inference without NVIDIA is already underway.
