- Published on
Tokensift, an open-sourced token-efficiency linter for LLM prompts
- Authors
- Name
- DeepHorus
A new open-source tool called Tokensift has been released to help developers reduce the cost and improve the efficiency of their large language model (LLM) prompts. The tool acts as a "token-efficiency linter," performing deterministic, local static analysis on prompt strings, message arrays, and tool schemas. It identifies where token counts are inflated by inefficient text patterns, providing specific suggestions for optimization.
LLM APIs typically charge users based on token usage, and token counts often do not align intuitively with character or word counts. Complex strings like UUIDs, base64-encoded data, or indented JSON can consume a disproportionate number of tokens due to the limitations of the tokenizer. Tokensift addresses this by using the actual tokenizer from the model provider to perform its analysis, not just estimates.
For OpenAI models, Tokensift delivers exact token counts by using the real BPE vocabulary. For Anthropic's Claude, where no public tokenizer is available, it uses a calibrated estimate and clearly labels its findings with a confidence level of "estimate" versus "exact." This provides developers with transparent data about their prompt costs.
The tool is currently in an early stage of active development but is fully functional. It features a core engine with 20 built-in rules, calculates the real dollar cost for each finding, and offers both a command-line interface (CLI) and testing libraries for vitest and jest. Its initial rule set catches common inefficiencies, such as "uuid-bloat" and "pretty-json" formatting.
In a demonstration, Tokensift analyzed a sample support ticket classifier prompt and identified three issues. It flagged a UUID that cost 18 tokens, suggesting a shorter ID would only cost 3. It also found that a block of instructions was duplicated unnecessarily and that a pretty-printed JSON schema could be minified to save tokens from newlines and indentation.
The release has been met with interest from the developer community as a practical solution for managing LLM expenses. By framing token cost as a form of code waste, similar to how traditional linters flag unused variables, Tokensift provides a tangible way to optimize both API costs and valuable context-window space.
Looking ahead, the project aims to expand its rule set and support for more LLM providers. As token-based pricing remains a standard model, tools that offer precise, actionable insights into prompt efficiency are expected to become a key part of the AI development workflow.
