- Published on
Ttok 1.0
- Authors
- Name
- DeepHorus
The ttok command-line tokenizer tool reached version 1.0 on 9th October 2026, with its headline change being a new default tokenizer that targets the GPT-5 and GPT-6 model families rather than the older GPT-4 standard. The release marks the project's first major version number, shipped by its maintainer shortly after version 0.4 went out.
The change came about through an ordinary debugging moment. After releasing ttok 0.4, the maintainer ran uv tool upgrade ttok, piped a file into the freshly updated version, and noticed that the tool was still defaulting to the GPT-4 tokenizer. Given the model landscape, he reasoned that switching the default was a reasonable excuse to finally ship a 1.0 rather than leaving a 0.x release in place.
The version bump is backed by a question that OpenAI has not yet formally answered: whether GPT-6 actually uses the same tokenizer as the GPT-5 family. The uncertainty has generated an angry issue in the project's tracker, reflecting developer frustration at the lack of an official confirmation from OpenAI.
Evidence for the switch, however, comes from a commit by William Liu summarizing an experiment he ran on the question. His results report that all seven GPT models tested — GPT-5.5, GPT-5.6 Sol/Terra/Luna, and GPT-6 Astra/Sol/Luna — each report 44,794 tokens and match one another on every one of the 31 fixtures used. In short, GPT-6 introduces no input-count change on that corpus, suggesting the tokenizers are likely identical.
For developers and prompt engineers, the practical implication is direct: token counts obtained from ttok will now align with the newest model generation rather than undercounting against GPT-5 and GPT-6 endpoints, which matters for cost estimation, context-window budgeting, and truncation logic.
The timing places the release in a busy stretch of the AI tooling calendar, following OpenAI DevDay 2026 on 29th September and alongside fresh model news such as Claude Haiku 5.5 on 7th October.
Looking ahead, the maintainer's default now rests on community-reported evidence rather than an OpenAI statement. Until the company confirms tokenizer parity for GPT-6, the angry issue is likely to stay open, and ttok users should treat the new default as well-supported but unofficial.
