bpe repositories
Trending repositories tagged bpe, ranked by trending score. Updated daily.
-
1
myylogic/cevahir-ai
Cevahir Zihin: a mind that learns by living, not a Transformer. It keeps knowledge as evidence on ties, attends with a Seven-Place Focus and learns from every conversation. Also: a Turkish BPE tokenizer, a Transformer language-model core and cognitive systems.
GitHub · Python · 312 stars · 12 forks · +17 stars in 24 hours · Trending score 2.44
-
2
tuetoken-org/tuetoken
The fastest tokenizer for LLMs on Earth
GitHub · Rust · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.08
-
3
tryAGI/Tiktoken
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
GitHub · C# · 93 stars · 7 forks · 0 stars in 24 hours · Trending score 0.08
-
4
georgemandis/tokenizer-mono
A monospace color font that highlights LLM tokens as you type. 28 faces, one per tokenizer (cl100k, o200k, Llama, Gemma, Qwen, DeepSeek, Kimi K2 and more), built from OpenType substitution rules and a color table. No JavaScript.
GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
5
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
GitHub · Rust · 411 stars · 76 forks · 0 stars in 24 hours · Trending score 0.00
-
6
akretion/nfelib
nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e
GitHub · Python · 219 stars · 73 forks · 0 stars in 24 hours · Trending score 0.00
-
7
niieani/gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
GitHub · TypeScript · 862 stars · 65 forks · 0 stars in 24 hours · Trending score 0.00
-
8
alphaparkinc/genpark-byte-pair-encoding-bpe-tokenizer-skill
Autonomous Agent Byte-Pair Encoding (BPE) Subword Tokenizer Training & Compression Engine
GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
9
actual-computer/toks
exact tokenizers ids, as fast as the hardware allows
GitHub · C · 115 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00
-
10
gillverd/accelerando-8192
Accelerando-8192 — an 8,192-token alien glyph alphabet, interactive encoder, and complete visual atlas.
GitHub · JavaScript · 57 stars · 11 forks · 0 stars in 24 hours · Trending score 0.00