Mbeeee111/tokenizer.cpp
📦 Optimize tokenization in C++ for HuggingFace models with a fast, production-ready library supporting BPE, WordPiece, and Unigram methods.
GitHub repository with 5 stars and 0 forks.
Language: C++
Topics: awesome, bert, bpe, c, dac, encodec, ggml, icu, machine-translation, nlp