bpe repositories

Trending repositories tagged bpe, ranked by trending score. Updated daily.

  1. 1
    myylogic/cevahir-ai

    Cevahir Zihin: a mind that learns by living, not a Transformer. It keeps knowledge as evidence on ties, attends with a Seven-Place Focus and learns from every conversation. Also: a Turkish BPE tokenizer, a Transformer language-model core and cognitive systems.

    GitHub · Python · 312 stars · 12 forks · +17 stars in 24 hours · Trending score 2.44

  2. 2
    tuetoken-org/tuetoken

    The fastest tokenizer for LLMs on Earth

    GitHub · Rust · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.08

  3. 3
    tryAGI/Tiktoken

    High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

    GitHub · C# · 93 stars · 7 forks · 0 stars in 24 hours · Trending score 0.08

  4. 4
    georgemandis/tokenizer-mono

    A monospace color font that highlights LLM tokens as you type. 28 faces, one per tokenizer (cl100k, o200k, Llama, Gemma, Qwen, DeepSeek, Kimi K2 and more), built from OpenType substitution rules and a color table. No JavaScript.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  5. 5
    zurawiki/tiktoken-rs

    Ready-made tokenizer library for working with GPT and tiktoken

    GitHub · Rust · 411 stars · 76 forks · 0 stars in 24 hours · Trending score 0.00

  6. 6
    akretion/nfelib

    nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

    GitHub · Python · 219 stars · 73 forks · 0 stars in 24 hours · Trending score 0.00

  7. 7
    niieani/gpt-tokenizer

    The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

    GitHub · TypeScript · 862 stars · 65 forks · 0 stars in 24 hours · Trending score 0.00

  8. 8
    alphaparkinc/genpark-byte-pair-encoding-bpe-tokenizer-skill

    Autonomous Agent Byte-Pair Encoding (BPE) Subword Tokenizer Training & Compression Engine

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    actual-computer/toks

    exact tokenizers ids, as fast as the hardware allows

    GitHub · C · 115 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    gillverd/accelerando-8192

    Accelerando-8192 — an 8,192-token alien glyph alphabet, interactive encoder, and complete visual atlas.

    GitHub · JavaScript · 57 stars · 11 forks · 0 stars in 24 hours · Trending score 0.00