tokenizer repositories

Trending repositories tagged tokenizer, ranked by trending score. Updated daily.

  1. 1
    rasbt/LLMs-from-scratch

    Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

    GitHub · Jupyter Notebook · 108,737 stars · 16,814 forks · +95 stars in 24 hours · Trending score 3.95

  2. 2
    myylogic/cevahir-ai

    Cevahir Zihin: a mind that learns by living, not a Transformer. It keeps knowledge as evidence on ties, attends with a Seven-Place Focus and learns from every conversation. Also: a Turkish BPE tokenizer, a Transformer language-model core and cognitive systems.

    GitHub · Python · 312 stars · 12 forks · +17 stars in 24 hours · Trending score 2.44

  3. 3
    libraz/suzume

    A lightweight Japanese tokenizer for browsers and native apps — WebAssembly, Node.js, Python, Go, and C++, returning POS tags and lemmas with the dictionaries built in

    GitHub · C++ · 23 stars · 4 forks · +16 stars in 24 hours · Trending score 2.16

  4. 4
    GLOBUS-studio/PhpObfuscator

    Token-based PHP source-code obfuscator: renames identifiers, Base64-encodes string literals, strips comments and minifies whitespace - built purely on PhpToken, no regex. Optional eval-wrapping and deterministic seeding. PHP 8.3+.

    GitHub · PHP · 35 stars · 5 forks · +10 stars in 24 hours · Trending score 1.69

  5. 5
    chizkidd/Karpathy-Neural-Networks-Zero-to-Hero

    Detailed python notes & code for lectures and exercises of Andrej Karpathy's course "Neural Networks: Zero to Hero." The course is focused on building neural networks from scratch.

    GitHub · Jupyter Notebook · 292 stars · 60 forks · 0 stars in 24 hours · Trending score 1.39

  6. 6
    onurtirpan/agentic-llm-token-and-performance-tests

    The same HTTP API in 10 languages at 3 sizes, 30 implementations. Token cost for LLM agents, plus cold start, latency, throughput, CPU and memory. All measured, all verified over live HTTP.

    GitHub · Python · 28 stars · 1 forks · +6 stars in 24 hours · Trending score 1.38

  7. 7
    approximatelabs/affordance

    A browser for models: apps in TSX, rendered to the exact token ids a model reads

    GitHub · Rust · 22 stars · 3 forks · +6 stars in 24 hours · Trending score 0.99

  8. 8
    ml-rust/splintr

    A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

    GitHub · Rust · 63 stars · 6 forks · +2 stars in 24 hours · Trending score 0.78

  9. 9
    phpsyntax/phpsyntax

    🌳 PhpSyntax: a lossless concrete syntax tree (CST) for PHP. Every token, space and comment is in the tree and printing it gives the file back byte for byte, so a tool changes only what it touches. Comes with a mutation API for refactoring, formatters and code generation.

    GitHub · PHP · 13 stars · 0 forks · 0 stars in 24 hours · Trending score 0.50

  10. 10
    taishi-i/nagisa

    A Japanese tokenizer based on recurrent neural networks

    GitHub · Python · 424 stars · 24 forks · 0 stars in 24 hours · Trending score 0.37

  11. 11
    lindera/lindera

    A multilingual morphological analysis library.

    GitHub · Rust · 708 stars · 70 forks · 0 stars in 24 hours · Trending score 0.25

  12. 12
    MahdiAlikhah/rishe

    Persian NLP in Rust: fast tokenization, lemmatization, morphology and embeddings.

    GitHub · Rust · 10 stars · 1 forks · 0 stars in 24 hours · Trending score 0.24

  13. 13
    cognisoc/zigllm

    Learn how LLMs work by building one in Zig -- from tensors to text generation.

    GitHub · Zig · 12 stars · 1 forks · 0 stars in 24 hours · Trending score 0.24

  14. 14
    jflex-de/jflex

    The fast scanner generator for Java™ with full Unicode support

    GitHub · Java · 637 stars · 120 forks · −1 stars in 24 hours · Trending score 0.19

  15. 15
    uhop/stream-json

    A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint — parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.

    GitHub · JavaScript · 1,207 stars · 60 forks · 0 stars in 24 hours · Trending score 0.17

  16. 16
    behrouzbk/tokensaver

    Count tokens across AI models and get concrete, priced suggestions to cut your prompt costs. Runs offline.

    GitHub · Python · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.16

  17. 17
    navthings/airgpt

    train your own chatbot from scratch on a mac

    GitHub · Python · 26 stars · 1 forks · 0 stars in 24 hours · Trending score 0.14

  18. 18
    adbar/simplemma

    Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

    GitHub · Python · 219 stars · 18 forks · 0 stars in 24 hours · Trending score 0.09

  19. 19
    tuetoken-org/tuetoken

    The fastest tokenizer for LLMs on Earth

    GitHub · Rust · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.08

  20. 20
    eugenioenko/textmate-go

    Pure-Go TextMate grammar tokenization engine

    GitHub · Go · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08

  21. 21
    tryAGI/Tiktoken

    High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

    GitHub · C# · 93 stars · 7 forks · 0 stars in 24 hours · Trending score 0.08

  22. 22
    lirantal/tokenu

    a unix-like du command line tool to count token usage per files and directories

    GitHub · TypeScript · 67 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    hephaex/mecab-ko

    MeCab-Ko: Rust로 구현된 한국어 형태소 분석기. 세종 코퍼스 호환.

    GitHub · C++ · 29 stars · 4 forks · 0 stars in 24 hours · Trending score 0.06

  24. 24
    georgemandis/tokenizer-mono

    A monospace color font that highlights LLM tokens as you type. 28 faces, one per tokenizer (cl100k, o200k, Llama, Gemma, Qwen, DeepSeek, Kimi K2 and more), built from OpenType substitution rules and a color table. No JavaScript.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05