tokenization repositories
Trending repositories tagged tokenization, ranked by trending score. Updated daily.
-
1
WorksApplications/sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
GitHub · Rust · 493 stars · 64 forks · 0 stars in 24 hours · Trending score 0.63
-
2
AntonSeveri803525/decoupling-ephemeral-cloud-gpu-clusters-from-retained-credentials
Hardening ephemeral machine-learning infrastructure: Mitigating zero-day card skimming and runaway hourly compute billing across decentralized cloud GPU providers.
GitHub · 15 stars · 0 forks · 0 stars in 24 hours · Trending score 0.42
-
3
YD811/potbot-v2
PotBot: buy a basket of tokenized assets on Solana in one go. BAG (Bitcoin, Anthropic, gold) is live in beta. POTFolio turns a basket into one token (devnet, mainnet after audit).
GitHub · TypeScript · 22 stars · 3 forks · 0 stars in 24 hours · Trending score 0.33
-
4
bimal1023/AI-native-engineer
Tracking my journey into AI engineering, one module at a time. Fundamentals to agents, with hands-on projects. Open to all.
GitHub · 9 stars · 2 forks · 0 stars in 24 hours · Trending score 0.19
-
5
securitybunker/databunker
Secure Vault for Customer PII/PHI/PCI/KYC Records
GitHub · Go · 1,498 stars · 102 forks · −1 stars in 24 hours · Trending score 0.10
-
6
fastbyte19/Ondo-Flux-Finance
An open-source DeFi protocol with smart contracts and a web dashboard for tokenized funds, automated liquidity provisioning, and on-chain yield optimization.
GitHub · Solidity · 33 stars · 3 forks · −3 stars in 24 hours · Trending score 0.09
-
7
adbar/simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
GitHub · Python · 219 stars · 18 forks · 0 stars in 24 hours · Trending score 0.09
-
8
mensfeld/llm-docs-builder
Transform and optimize your markdown documentation for Large Language Models (LLMs) and RAG systems. Generate llms.txt automatically.
GitHub · Ruby · 95 stars · 5 forks · 0 stars in 24 hours · Trending score 0.09
-
9
aymara/lima
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
GitHub · C++ · 119 stars · 20 forks · 0 stars in 24 hours · Trending score 0.08
-
10
tuetoken-org/tuetoken
The fastest tokenizer for LLMs on Earth
GitHub · Rust · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.08
-
11
min-1225/veil-skill
Veil: mask sensitive data before the AI sees it, unmask it after. A Claude Code skill (KO/EN/JA/ZH + local NER).
GitHub · JavaScript · 9 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08
-
12
lirantal/tokenu
a unix-like du command line tool to count token usage per files and directories
GitHub · TypeScript · 67 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07
-
13
Apil-Shrestha/token-efficiency-lex
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
GitHub · HTML · 115 stars · 0 forks · 0 stars in 24 hours · Trending score 0.06
-
14
Edgefireex/rpocket-MoneroOceanMinerSetup
Fastest And Most Efficient XMR Mining Program (A MUST WATCH!!!)
GitHub · 21 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
15
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
GitHub · TypeScript · 25,628 stars · 1,124 forks · 0 stars in 24 hours · Trending score 0.03
-
16
devlab-group/rwa-platform-evm
A self-hosted, single-tenant platform for tokenizing a real-world asset class as a permissioned ERC-20, with an auditor-attested supply lifecycle and on-chain cash redemption.
GitHub · Go · 46 stars · 0 forks · 0 stars in 24 hours · Trending score 0.01
-
17
Aajil-Labs/arabic-pii-py
Keep Arabic & Gulf personal data off the LLM — local-first, reversible PII tokenization (CLI, proxy, and a Claude Code hook).
GitHub · Python · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
18
gil-son/language-ai-engineering-lab
Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.
GitHub · Jupyter Notebook · 130 stars · 27 forks · 0 stars in 24 hours · Trending score 0.00
-
19
andreihar/taibun
Taiwanese Hokkien Transliterator and Tokeniser
GitHub · Python · 52 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00
-
20
tinyhumansai/tinyjuice
A juicy token compression algorithm. Compress upto 95% of tokens without any loss. Free & Opensource
GitHub · Rust · 22 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00
-
21
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
GitHub · Rust · 428 stars · 26 forks · 0 stars in 24 hours · Trending score 0.00
-
22
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
GitHub · Rust · 301 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00
-
23
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
GitHub · C# · 55 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00
-
24
CMTA/CMTAT
Reference Solidity implementation of the CMTAT security token framework developed by CMTA to tokenize financial instruments.
GitHub · JavaScript · 84 stars · 40 forks · 0 stars in 24 hours · Trending score 0.00