GitHub · Python
umran666/UniqToken
High-precision, Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.
- Stars
- 18
- Forks
- 8
- Stars gained in 24 hours
- 0
- Forks gained in 24 hours
- 0
Topics: byte-fallback, deep-learning, good-first-issue, hacktoberfest, help-wanted, nlp, pyo3, rust, subwords, tokenizer
Latest data point: Nov 21, 2026 · 18 stars · 8 forks