moe repositories
Trending repositories tagged moe, ranked by trending score. Updated daily.
-
1
FareedKhan-dev/kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37
-
2
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26
-
3
mw00/project-maya
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.
GitHub · C++ · 3,094 stars · 430 forks · +69 stars in 24 hours · Trending score 3.63
-
4
wici-ai/inferssd
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
GitHub · C++ · 1,262 stars · 113 forks · +32 stars in 24 hours · Trending score 3.48
-
5
FlashML-org/FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
GitHub · Python · 17,331 stars · 1,721 forks · +44 stars in 24 hours · Trending score 3.43
-
6
hiyouga/LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
GitHub · Python · 76,148 stars · 9,394 forks · +48 stars in 24 hours · Trending score 3.32
-
7
Helldez/BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
GitHub · C++ · 1,033 stars · 107 forks · +23 stars in 24 hours · Trending score 3.07
-
8
RustingStudio/RustingStrata
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90
-
9
flashinfer-ai/flashinfer
FlashInfer: Kernel Library for LLM Serving
GitHub · Cuda · 6,961 stars · 1,773 forks · +21 stars in 24 hours · Trending score 2.81
-
10
NVIDIA/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
GitHub · Python · 15,105 stars · 2,944 forks · +21 stars in 24 hours · Trending score 2.65
-
11
sangharshadhyeta/Strata-HotRod
Run Qwen3.8-Flash-Next (125B) on an RTX 3060 12 GB with 30 GB of RAM, streaming experts from the SSD. 50-70 tok/s, 262K context. A fork of Strata.
GitHub · C++ · 37 stars · 3 forks · +24 stars in 24 hours · Trending score 2.53
-
12
modelscope/ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
GitHub · Python · 16,135 stars · 1,817 forks · +12 stars in 24 hours · Trending score 2.42
-
13
sergqwer/strata-nvfp4
Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.
GitHub · C++ · 251 stars · 31 forks · +23 stars in 24 hours · Trending score 2.42
-
14
prabanta-dev/janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41
-
15
Indras-Mirror/Xolotl
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
GitHub · C++ · 545 stars · 67 forks · +12 stars in 24 hours · Trending score 2.18
-
16
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
GitHub · Python · 38,872 stars · 10,647 forks · −123 stars in 24 hours · Trending score 1.80
-
17
davide-troiani/moeshift
Memory efficient and high performance Mixture-of-Experts inference for Apple Silicon
GitHub · C++ · 127 stars · 7 forks · +4 stars in 24 hours · Trending score 1.49
-
18
NVIDIA/cudnn-frontend
cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.
GitHub · Cuda · 1,007 stars · 340 forks · +3 stars in 24 hours · Trending score 1.47
-
19
Bruce-Lee-LY/gc-moe
GC MoE is a CuTe DSL implementation of FP8/FP4 mixture-of-experts inference with CUDA Green Contexts for NVIDIA Hopper and Blackwell GPUs.
GitHub · Python · 36 stars · 5 forks · +4 stars in 24 hours · Trending score 1.45
-
20
kacper-daftcode/vLLM-Moet
Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and 2-bit experts on vLLM v0.24.0
GitHub · Sass · 579 stars · 49 forks · +7 stars in 24 hours · Trending score 1.44
-
21
Morrowmake/glm53-flash-cmp170hx-recipe
GLM-5.3-Flash on 4× NVIDIA CMP 170HX with vLLM: up to 482.6 tok/s single-user, up to 943.7 tok/s at 8 users, 262K context, TP4 or PP4, one-command container install
GitHub · Python · 93 stars · 14 forks · +1 stars in 24 hours · Trending score 1.42
-
22
T-Crypt/halocline
Sparse + Dense. Qwen4 family on one RTX 4090 (sm_89 only)
GitHub · C++ · 18 stars · 1 forks · +12 stars in 24 hours · Trending score 1.35
-
23
whitecircle/halo
Halo is an open-source framework built by White Circle for training large language and multimodal models
GitHub · Python · 868 stars · 50 forks · +2 stars in 24 hours · Trending score 1.32
-
24
uccl-project/uccl
UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)
GitHub · C++ · 1,588 stars · 199 forks · +2 stars in 24 hours · Trending score 1.12