- Published on
A New RDMA Transport Built for AI-Scale Ethernet
- Authors
- Name
- DeepHorus
Meta has unveiled MetaRoCE, a new Remote Direct Memory Access transport protocol designed specifically for AI workloads running on standard Ethernet infrastructure. The company is releasing the specification, reference software, and a compliance test suite through the Open Compute Project to foster industry-wide adoption. This announcement underscores Meta's commitment to making Ethernet the foundational network for large-scale AI systems.
The development comes as training and serving frontier AI models demands incredibly fast, reliable networks to move data between GPUs without wasting compute cycles. Meta has already demonstrated that RoCE can power distributed AI training at scale, and MetaRoCE represents a ground-up redesign for Ethernet at the million-GPU scale. The company has scaled clusters to hundreds of thousands of GPUs across multiple data centers and regions.
In such massive environments, network performance is critical. Collective operations like all-reduce synchronize thousands of accelerators, where the slowest transfer sets the pace for the entire job. During inference, low-latency communication between distributed model shards directly impacts response times for hundreds of millions of users. Even minor network friction can strand significant compute capacity.
Unlike standard RoCE, which relies on network switches to enforce losslessness and packet order, MetaRoCE moves intelligence to the network interface card (NIC). The core insight is that the fabric sees packets, but the NIC sees intent. The protocol decomposes the network into many fine-grained logical paths, each monitored with real-time telemetry like per-path round-trip time and utilization.
This design enables several key innovations. MetaRoCE supports native out-of-order packet delivery by design, spraying packets across many paths. Every packet carries its destination, allowing data to be written directly to its final memory location with no reordering buffer or head-of-line blocking. The protocol also features native multipathing, giving each connection first-class paths and allowing the NIC to manage traffic distribution.
A fundamental principle is loss tolerance by design. MetaRoCE treats the Ethernet fabric as inherently lossy, eliminating the need for Priority Flow Control (PFC) and pause frames. Each path carries its own sequence numbering, and gaps in acknowledgments clearly signal loss rather than reordering, enabling efficient recovery without network-enforced guarantees.
This approach promises high throughput, low tail latency, and operational simplicity as AI clusters grow in scale and geographic spread. By open-sourcing MetaRoCE through OCP, Meta aims to enable the broader industry to adopt, implement, and build upon the technology. The move could accelerate the shift toward Ethernet as the dominant fabric for next-generation AI infrastructure.
