- Published on
Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed
- Authors
- Name
- DeepHorus
Perplexity, the AI search startup, has revealed the detailed technical architecture behind its GPU embedding stack, a critical component powering its search, computer, and API platforms. The company published a technical blog post titled "Fast Embeddings on GPUs," providing an under-the-hood look at its custom serving infrastructure. This move comes as the company seeks to explain how it achieves both high-quality retrieval and cost-effective scaling for its core AI-powered search product.
The engineering team explains that embedding inference on mature GPU hardware like NVIDIA's Hopper and Blackwell has largely converged across the industry. The key differentiators for Perplexity are not the hardware itself, but the software stack built around it. Their innovation focuses on a specialized runtime and harness designed to maximize efficiency, leveraging CUDA graph management, asynchronous result tracking, and a Rust-based request pipeline.
Perplexity categorizes embedding workloads into two main types: batch embedding for building the vector database, where throughput is king, and online embedding for real-time query processing, where low latency is essential. A unique aspect of their approach is the decision not to build a separate engine for these tasks. Recognizing that embedding models are small Transformers, the team cleverly reuses the prefill and decode kernels from its main large language model (LLM) stack, as batch and online workloads map to compute-bound and memory-bound patterns respectively.
The request flow is handled by a trio of services. First, Ivy, a Rust HTTP gateway, manages CPU-side tasks like tokenization and splits large batches for balanced loading. Second, Tulip, a gRPC server built with Rust and Tokio, acts as the inference scheduler. Finally, ROSE (Runtime-Optimized Serving Engine) executes the model inference. This multi-service design allows for clear separation of concerns and optimization at each stage.
A particularly insightful detail is the deliberately simple first-come, first-served scheduler in Tulip. Perplexity justifies this with measurements showing that for small embedding models, latency is dominated by the linear cost of dense layers, not the quadratic cost of attention. This means adding more sequences to a batch beyond a saturation point of around 512 tokens does not improve GPU efficiency, making complex scheduling unnecessary.
To combat the overhead of launching many small GPU kernels, Perplexity builds entire CUDA graphs for its embedding models. This captures every kernel launch into a single driver call, dramatically reducing CPU-side latency. To manage the resulting explosion of possible graph configurations, they implement "lazy capture," where graphs are built on-demand after a warmup run. They also use a custom "LazyTensor" abstraction to allow asynchronous data transfer and computation, further overlapping CPU and GPU work.
The technical deep-dive underscores Perplexity's focus on engineering optimization to create a competitive advantage. By fine-tuning every layer of the serving stack from the Rust gateway to the CUDA kernels, the company aims to provide faster and more cost-effective embeddings. This infrastructure is foundational to improving the retrieval quality that powers its AI answers, giving it a potential edge in the crowded AI search market.
