- Published on
An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM
- Authors
- Name
- DeepHorus
Cactus Compute today announced Needle 2, a compact open-source model designed exclusively for tool calling, device control, and structured data extraction. The 45-million-parameter system is packaged as a single 14MB binary and runs an entire interactive session in roughly 28MB of RAM, making it one of the smallest practical AI models for edge deployment. Weights are trained and shipped at CQ2-bit precision using the company's Cactus Quants format and are sealed inside a custom C++ engine, eliminating any runtime dependencies or runtime downloads.
The release comes as developers struggle to fit AI assistants into low-power devices without cloud connectivity. Most small language models still require hundreds of megabytes and significant compute, but Cactus argues that tool calling is a narrow task: mapping a messy sentence to a typed function signature needs neither world knowledge nor open-ended prose. That design philosophy lets the company cut parameters to 45M while targeting hardware with no GPU and no NPU, such as Raspberry Pi, smart glasses, and budget phones.
Needle 2 reports decode throughput of 500 tokens per second on a Raspberry Pi 5, 400–1,500 tokens per second on Meta Quest 3S and Apple Vision Pro, and 300–700 tokens per second on sub-$200 phones. It ships as prebuilt binaries and a static library across macOS, Linux, Windows, Android, iOS, embedded Apple platforms, and WebAssembly. Cactus says Pebble already runs Needle locally in the Index 01 app for offline voice actions.
Architecturally, Needle 2 uses a Simple Attention Network, replacing the traditional FFN with a Hadamard MLP while keeping grouped-query attention and adding engram key-value memory from hashed n-gram tables. The network is 27 layers wide at 512 channels, spending only 70 MFLOPs per token, compared with 460 MFLOPs for LFM2.5-230M or roughly 6,000 for Apple's FM models. Pretraining leveraged a proprietary 115B-token corpus with 38B additional post-training tokens.
A byte-level grammar compiled from user JSON schemas constrains every emitted token, allowing the engine to skip up to 98% of vocabulary projection on structural tokens. Attention uses a 256-token sliding window with system prompts pinned as KV sinks, keeping memory near 28MB regardless of conversation length. For more than five tools, a contrastive retrieval head selects only the top five per turn, making unselected tools unreachable rather than merely unlikely. Every response includes a calibrated confidence value, and off-topic requests return an empty call.
Companies building smart home devices, wearables, low-end mobile, automotive cabin controls, service robotics, retail kiosks, routers, and IP cameras could benefit from on-device routing that escalates to the cloud only on low confidence. Regulated sectors where audio cannot leave the device also stand to gain, while cloud-first SaaS teams may see little value. Analysts note that the 28MB footprint and binary-only distribution substantially lower the barrier to embedding AI in firmware, though the 45M-parameter ceiling naturally limits general reasoning.
Cactus has published its underlying research on arXiv and plans to continue refining the retrieval and confidence mechanisms. The release signals a broader industry shift toward narrowly specialized, ultra-efficient models that run entirely offline. As vendors pack more sensors into everyday objects, Needle 2 offers a concrete path for providing responsive, private voice-to-action without waiting for the next cloud round-trip.
