synchopate/nemotron-30b-cpu-256k
Running a 30B MoE LLM on CPU at 30-50+ tok/s with a 256K Context Window via Hybrid Mamba-2 SSM, L3 NTA Slicing, AVX-512 VNNI Tile Repacking & 2MB HugePages
GitHub repository with 69 stars and 11 forks.
Language: Python
Topics: avx512, cpu-inference, hpc, llama-cpp, llm, mamba2, mixture-of-experts, nemotron, speculative-decoding, zen5