ttok 1.0 Switches Default Tokenizer to GPT‑5/6, Shakes Up LLM Tooling
Developer Simon Willison releases ttok 1.0, changing its default tokenizer from GPT‑4 to GPT‑5/6, a move that could streamline token counting for businesses using OpenAI models.
BuildPreview AI Desk
2 min read
Simon Willison, maintainer of the Python tokenizer library ttok, announced a 1.0 release after noticing the tool defaulted to GPT‑4 when it should have used GPT‑5/6. The change follows a recent upgrade with uv and a test pipeline that revealed the discrepancy.
Key points
- ttok 1.0 shifts default tokenizer from GPT‑4 to GPT‑5/6.
- OpenAI has not yet confirmed GPT‑6 shares GPT‑5 tokenizer.
- William Liu’s experiment shows identical token counts across seven GPT models.
- All models report 44,794 tokens on 31 test fixtures.
- GPT‑6 introduces no input‑count change on this corpus.
What happened
On October 9, 2026, Willison released ttok 0.4 and then upgraded the package with uv. While piping a file through the updated tool, he discovered it was still using the GPT‑4 tokenizer. He noted that the default should have been GPT‑5 or GPT‑6, prompting him to ship a major version bump. The decision was justified by external evidence: William Liu’s commit documents that GPT‑5.5, 5.6 Sol/Terra/Luna, and 6 Astra/Sol/Luna all return 44,794 tokens on the same 31 fixtures, indicating identical tokenization.
The backstory
Tokenizer consistency has long been a pain point for developers integrating multiple OpenAI models. Prior releases of ttok defaulted to GPT‑4, which mismatched newer models and caused unexpected token counts. The OpenAI documentation has yet to confirm whether GPT‑6 inherits the GPT‑5 tokenizer, leaving a gap that third‑party libraries must bridge.
Why the switch matters
By aligning the default tokenizer with GPT‑5/6, ttok reduces friction for teams migrating to newer OpenAI releases. The identical token counts across models mean developers can rely on a single library for all GPT‑5 and GPT‑6 variants, simplifying cost calculations and compliance checks. The trade‑off is minimal: users must update their environments to 1.0 to benefit.
Potential ripple effects
Businesses that batch large documents for GPT‑5/6 will see more accurate token tallies, affecting pricing and quota usage. Tooling ecosystems that depend on ttok may need to adjust compatibility layers. Those who continue using older versions risk inconsistent tokenization across model calls.
What builders and businesses should do
- Upgrade to ttok 1.0 as soon as possible to match GPT‑5/6 defaults.
- Re‑run token count scripts to confirm new defaults.
- Update CI pipelines to reference the latest ttok version.
- Monitor OpenAI announcements for tokenizer changes in future releases.
- Consider contributing to the ttok repository if you encounter edge cases.
The bottom line
Watch for OpenAI’s official confirmation of GPT‑6’s tokenizer in the next product brief. For now, ttok 1.0 offers a reliable, unified tokenization strategy for GPT‑5 and GPT‑6 users.
