tryAGI/Tiktoken
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
GitHub repository with 87 stars and 7 forks.
Language: C#
Topics: csharp, openai, tiktoken, ai, bpe, dotnet, gpt4o, high-performance, huggingface, sdk