tiktoken repositories
Trending repositories tagged tiktoken, ranked by trending score. Updated daily.
-
1
tryAGI/Tiktoken
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
GitHub · C# · 93 stars · 7 forks · 0 stars in 24 hours · Trending score 0.07
-
2
CNSeniorious000/free-chat
An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia
GitHub · TypeScript · 192 stars · 54 forks · 0 stars in 24 hours · Trending score 0.07
-
3
georgemandis/tokenizer-mono
A monospace color font that highlights LLM tokens as you type. 28 faces, one per tokenizer (cl100k, o200k, Llama, Gemma, Qwen, DeepSeek, Kimi K2 and more), built from OpenType substitution rules and a color table. No JavaScript.
GitHub · Python · 10 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
4
behrouzbk/tokensaver
Count tokens across AI models and get concrete, priced suggestions to cut your prompt costs. Runs offline.
GitHub · Python · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.01
-
5
melonmath/microkimi
The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against the official code.
GitHub · Rust · 23 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00
-
6
mandeal303/tiktokenizer
An interactive playground for comparing how different language models tokenize the same text.
GitHub · TypeScript · 50 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00