- Published on
Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model
- Authors
- Name
- DeepHorus
Linkup Research has released SPARSEUP, an open-source sparse embedding model built on a 149M-parameter ModernBERT backbone. Available under the Apache 2.0 license on Hugging Face, it scores 56.4 average nDCG@10 on the BEIR-13 benchmark. The team says it is the strongest public vocabulary-based sparse encoder under 150 million parameters.
Sparse models differ from the dense models that dominate open retrieval. Dense models produce one vector per text, while sparse models output weights over a vocabulary, with each dimension mapping to a real token. That design makes vectors compatible with inverted indexes and human-readable, and sparse models tend to match rare words well.
The release was triggered by LightOn's DenseOn and LateOn models. LightOn published open data, a training recipe, a dense model, and a late-interaction model. SPARSEUP fills the missing sparse slot using the same backbone family and fine-tuning data, allowing all three retrieval styles to be compared side by side.
Training started from the LateOn-unsupervised checkpoint, with ModernBERT's original MLM head grafted back on. Fine-tuning used LightOn's mixture with contrastive learning only, sampling 7 hard negatives per query from a pool of 50, plus in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100 GPU.
Linkup applied three key fixes to avoid the stopword-heavy bags produced by a vanilla SPLADE. A logit shift computes log(1 + ReLU(x - 15)), per-position top-k keeps only the 12 strongest vocabulary dimensions per token, and case folding reduces output dimensions from about 50,000 to about 34,000. Queries and documents use [Q] and [D] prefixes, with dot-product scoring.
On BEIR-13, LateOn scores 58.9, DenseOn 57.9, and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search while LightOn reports exact search. It wins on ArguAna and Touché and beats DenseOn on HotpotQA, but lags on semantic sets such as FiQA and DBPedia. On decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points.
On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document. With the Seismic inverted index, it reaches over 97% recall against exact search in about 380 microseconds per query, single-threaded. Linkup says inflating vector size could add 1 to 2 BEIR points, but it chose to stay sparse.
SPARSEUP is Linkup Research's first open model. With identical data, it trails DenseOn by 1.52 points and LateOn by 2.5 points on BEIR-13. The weights and technical details are available on Hugging Face for developers and researchers to deploy immediately.
