GitHub · C
actual-computer/toks
exact tokenizers ids, as fast as the hardware allows
- Stars
- 115
- Forks
- 9
- Stars gained in 24 hours
- 0
- Forks gained in 24 hours
- 0
Topics: assembly, avx2, bpe, c, huggingface, neon, sentencepiece, simd, tokenizer, wordpiece
Latest data point: Nov 21, 2026 · 115 stars · 9 forks