- Published on
A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon
- Authors
- Name
- DeepHorus
Perplexity has open sourced Lily, the high-performance local inference engine designed specifically for Apple Silicon. This engine powers the "Hybrid Compute" feature within Perplexity Computer. It is a specialized, single-process runtime built with Rust and hand-written Metal kernels, deliberately optimized for one model: the Qwen3.6-35B-A3B mixture-of-experts model.
The project's public release includes a standalone demo in the pplx-garden repository, providing a Rust and Metal server with an OpenAI-compatible API for text generation. The engine requires a Mac with Apple Silicon, a minimum of 32 GB of unified memory to handle the 19.4 GB quantized model checkpoint. This hardware requirement sets a realistic floor for deployment.
Lily's design philosophy prioritizes extreme specialization over general compatibility. Unlike the default Mac machine learning stack using MLX, which must support various architectures, Lily integrates the model structure, execution plans, and kernel selection into one cohesive runtime. This narrow focus is central to its performance argument.
The engine employs sophisticated optimizations for both the prefill and decode phases. During prefill, it performs on-the-fly dequantization of weights within GPU threadgroup memory, avoiding costly transfers to unified memory. This fusion reportedly increased end-to-end prefill speed by 77.4% for a 512-token prompt in internal tests.
For decode, which is often memory-bandwidth bound, Lily minimizes data movement per token. It fuses kernel chains and uses advanced techniques like grouped-query awareness to improve cache efficiency. These optimizations resulted in significant throughput gains, particularly for longer contexts, with decode speed improvements of up to 40.2% at 128K tokens.
Benchmark results on a high-end M5 Max chip show Lily outperforming the MLX-LM framework. Across a range of prompt lengths, Lily achieved 1.23x faster prefill and 1.35x faster decode speeds on average. At specific test points, such as a 4K prompt with 4K context, it reached nearly 5,750 tokens per second for prefill.
Perplexity's decision to open source Lily provides the community with a reference implementation for building high-performance, model-specific runtimes. It demonstrates the potential performance ceiling achievable through deep hardware and model co-design, offering a blueprint for future local AI inference on Apple Silicon platforms.
