1-bit and Ternary Builds of Qwen3.6-27B That Run on Laptops and Phones

PrismML has released Bonsai 27B, a radically compressed version of Alibaba's Qwen3.6-27B large language model, designed to run efficiently on laptops and even smartphones. The model achieves this through extreme quantization—using ternary or binary weight representations that slash memory requirements by up to 14x compared to standard FP16 formats. Available under the Apache 2.0 license, the release includes two variants: a ternary version using weights in {−1, 0, +1} at just 1.71 bits per weight (ideal size 5.9GB), and a 1-bit binary version using {−1, +1} at 1.125 bits per weight (3.9GB). Both are multimodal, retaining the original model's vision tower, and support a context window of 262,000 tokens—possible because approximately 75% of Qwen3.6-27B's attention uses linear mechanisms.

Compression is achieved through a codebook approach where each weight is a code, with one shared FP16 scale per group of 128 weights. For the ternary variant, each weight carries log2(3) ≈ 1.585 bits, plus 16/128 bits from the shared scale, yielding an average of 1.71 bits per weight—a 9.4x reduction against FP16. The binary variant achieves 1.125 bits per weight (14.2x reduction). Importantly, the compression is applied end-to-end across all matrix-heavy components: embeddings, attention projections, MLP projections, and the LM head. Only normalization and scale parameters remain in higher precision. This approach departs from methods like BitNet, which requires pretraining from scratch to avoid performance collapse.

On 15 benchmarks evaluated in thinking mode using EvalScope on H100 GPUs, the ternary Bonsai 27B retains 94.6% of the FP16 baseline's accuracy, while the 1-bit version retains 89.5%. These results stand in stark contrast to conventional sub-4-bit quantizations like IQ2_XXS, which collapses on challenging benchmarks such as AIME26 (57.5%) and LiveCodeBench (56.4%) despite scoring well on MMLU-Redux. PrismML argues that short-form benchmarks mask such degradation, whereas Bonsai maintains robust performance.

Memory constraints are the primary motivation. For mobile deployment, iOS limits a single app to roughly half of physical RAM: a 12GB iPhone provides only about 6GB. Additionally, the KV cache—which grows with context length—must be budgeted. With 16 out of 64 layers carrying full-attention caches, a 262K-token window costs about 17.2GB in FP16, or 4.3GB at 4-bit quantization. Peak memory at 100K tokens is 11.6GB for the 1-bit variant and 14.7GB for ternary, versus ~25.6GB for a typical 4-bit build. The gains enable deployment on devices that were previously impossible for models of this scale.

PrismML also includes a DSpark speculative decoding drafter trained against the Bonsai 27B target. On an H100 at draft depth k=4, the binary target achieves an accepted length τ=3.6, yielding 143.8 tokens per second—a 1.37x speedup. Verification is lossless, preserving the target distribution. On Apple Silicon, the drafter is disabled by default at batch size 1. The inference server can be launched via a shell script, and the model supports generation, multimodal input, and OpenAI-compatible tool calling.

Industry reactions have been cautiously optimistic, with developers noting that extreme quantization often sacrifices too much accuracy. However, the retention of 94.6% accuracy on ternary and 89.5% on 1-bit, combined with the massive memory reductions, could unlock new on-device AI applications. Analysts point out that the ability to run a 27B-parameter model on a phone—albeit with quantized weights—represents a significant milestone for edge AI, especially given the maintained long context window.

Looking ahead, PrismML plans to refine the compression method and expand support for more hardware backends. The company may also explore training from scratch using these low-bit formats, as done in BitNet, to potentially close the remaining accuracy gap. The code and models are available on GitHub and Hugging Face, inviting community experimentation and optimization.

This release fits into broader trends toward extreme quantization and on-device large language models. As model sizes balloon, the need for efficient deployment grows. Bonsai 27B demonstrates that with careful compression and architectural adaptation, even models previously confined to data centers can run on consumer hardware. The success of this approach may accelerate the shift toward local, private AI inference, reducing reliance on cloud APIs and lowering latency for real-time applications.

Turn any website into clean JSON.

Ready-made scrapers & AI agents

Try Apify