tokenization repositories

Trending repositories tagged tokenization, ranked by trending score. Updated daily.

  1. 1
    WorksApplications/sudachi.rs

    Sudachi in Rust 🦀 and new generation of SudachiPy

    GitHub · Rust · 493 stars · 64 forks · 0 stars in 24 hours · Trending score 0.63

  2. 2
    AntonSeveri803525/decoupling-ephemeral-cloud-gpu-clusters-from-retained-credentials

    Hardening ephemeral machine-learning infrastructure: Mitigating zero-day card skimming and runaway hourly compute billing across decentralized cloud GPU providers.

    GitHub · 15 stars · 0 forks · 0 stars in 24 hours · Trending score 0.42

  3. 3
    YD811/potbot-v2

    PotBot: buy a basket of tokenized assets on Solana in one go. BAG (Bitcoin, Anthropic, gold) is live in beta. POTFolio turns a basket into one token (devnet, mainnet after audit).

    GitHub · TypeScript · 22 stars · 3 forks · 0 stars in 24 hours · Trending score 0.33

  4. 4
    bimal1023/AI-native-engineer

    Tracking my journey into AI engineering, one module at a time. Fundamentals to agents, with hands-on projects. Open to all.

    GitHub · 9 stars · 2 forks · 0 stars in 24 hours · Trending score 0.19

  5. 5
    securitybunker/databunker

    Secure Vault for Customer PII/PHI/PCI/KYC Records

    GitHub · Go · 1,498 stars · 102 forks · −1 stars in 24 hours · Trending score 0.10

  6. 6
    fastbyte19/Ondo-Flux-Finance

    An open-source DeFi protocol with smart contracts and a web dashboard for tokenized funds, automated liquidity provisioning, and on-chain yield optimization.

    GitHub · Solidity · 33 stars · 3 forks · −3 stars in 24 hours · Trending score 0.09

  7. 7
    adbar/simplemma

    Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

    GitHub · Python · 219 stars · 18 forks · 0 stars in 24 hours · Trending score 0.09

  8. 8
    mensfeld/llm-docs-builder

    Transform and optimize your markdown documentation for Large Language Models (LLMs) and RAG systems. Generate llms.txt automatically.

    GitHub · Ruby · 95 stars · 5 forks · 0 stars in 24 hours · Trending score 0.09

  9. 9
    aymara/lima

    The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.

    GitHub · C++ · 119 stars · 20 forks · 0 stars in 24 hours · Trending score 0.08

  10. 10
    tuetoken-org/tuetoken

    The fastest tokenizer for LLMs on Earth

    GitHub · Rust · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.08

  11. 11
    min-1225/veil-skill

    Veil: mask sensitive data before the AI sees it, unmask it after. A Claude Code skill (KO/EN/JA/ZH + local NER).

    GitHub · JavaScript · 9 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08

  12. 12
    lirantal/tokenu

    a unix-like du command line tool to count token usage per files and directories

    GitHub · TypeScript · 67 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07

  13. 13
    Apil-Shrestha/token-efficiency-lex

    Token Cost Parity: Multilingual LLM Efficiency Analysis 2026

    GitHub · HTML · 115 stars · 0 forks · 0 stars in 24 hours · Trending score 0.06

  14. 14
    Edgefireex/rpocket-MoneroOceanMinerSetup

    Fastest And Most Efficient XMR Mining Program (A MUST WATCH!!!)

    GitHub · 21 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  15. 15
    toon-format/toon

    🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

    GitHub · TypeScript · 25,628 stars · 1,124 forks · 0 stars in 24 hours · Trending score 0.03

  16. 16
    devlab-group/rwa-platform-evm

    A self-hosted, single-tenant platform for tokenizing a real-world asset class as a permissioned ERC-20, with an auditor-attested supply lifecycle and on-chain cash redemption.

    GitHub · Go · 46 stars · 0 forks · 0 stars in 24 hours · Trending score 0.01

  17. 17
    Aajil-Labs/arabic-pii-py

    Keep Arabic & Gulf personal data off the LLM — local-first, reversible PII tokenization (CLI, proxy, and a Claude Code hook).

    GitHub · Python · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    gil-son/language-ai-engineering-lab

    Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.

    GitHub · Jupyter Notebook · 130 stars · 27 forks · 0 stars in 24 hours · Trending score 0.00

  19. 19
    andreihar/taibun

    Taiwanese Hokkien Transliterator and Tokeniser

    GitHub · Python · 52 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    tinyhumansai/tinyjuice

    A juicy token compression algorithm. Compress upto 95% of tokens without any loss. Free & Opensource

    GitHub · Rust · 22 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    daac-tools/vibrato

    🎤 vibrato: Viterbi-based accelerated tokenizer

    GitHub · Rust · 428 stars · 26 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    daac-tools/vaporetto

    🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

    GitHub · Rust · 301 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    georg-jung/FastBertTokenizer

    Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.

    GitHub · C# · 55 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    CMTA/CMTAT

    Reference Solidity implementation of the CMTAT security token framework developed by CMTA to tokenize financial instruments.

    GitHub · JavaScript · 84 stars · 40 forks · 0 stars in 24 hours · Trending score 0.00