inference-engine repositories
Trending repositories tagged inference-engine, ranked by trending score. Updated daily.
-
1
FareedKhan-dev/kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37
-
2
datawhalechina/zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
GitHub · Python · 2,295 stars · 188 forks · +103 stars in 24 hours · Trending score 3.69
-
3
simplexai-labs/PolyStrata
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
GitHub · C++ · 1,280 stars · 166 forks · +60 stars in 24 hours · Trending score 3.63
-
4
peonist-ai/halogen-flash-server
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66
-
5
sangharshadhyeta/Strata-HotRod
Run Qwen3.8-Flash-Next (125B) on an RTX 3060 12 GB with 30 GB of RAM, streaming experts from the SSD. 50-70 tok/s, 262K context. A fork of Strata.
GitHub · C++ · 37 stars · 3 forks · +24 stars in 24 hours · Trending score 2.53
-
6
RainBowLuoCS/Gamer
Official Implementation of Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agents
GitHub · 1,025 stars · 22 forks · +207 stars in 24 hours · Trending score 2.50
-
7
prabanta-dev/janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41
-
8
rizagp/inferzero
arcinferzero
GitHub · C++ · 62 stars · 11 forks · +12 stars in 24 hours · Trending score 2.40
-
9
OmniJev/Flide
🔥🔥 The extremely fast inference engine for decision model
GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36
-
10
nobodywho-ooo/nobodywho
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
GitHub · Rust · 1,691 stars · 102 forks · +16 stars in 24 hours · Trending score 2.03
-
11
TandemLLM/TandemEngine
TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.
GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73
-
12
youssofal/MTPLX
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
GitHub · Python · 2,727 stars · 214 forks · +6 stars in 24 hours · Trending score 1.72
-
13
Luce-Org/lucebox
Lucebox inference engine for heterogeneous hardware & consumer GPUs
GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63
-
14
NeuroBrix/neurobrix
Run real AI models on the GPU you already own: language, speech, image and video models from one engine and one container format, with or without PyTorch. NVIDIA GPUs, Mac GPUs through Metal, CPU; other GPUs experimental. Apache 2.0.
GitHub · Python · 67 stars · 0 forks · +8 stars in 24 hours · Trending score 1.62
-
15
jaedync/crucible-strix-halo
Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.
GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45
-
16
xLLM-AI/xllm
A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
GitHub · C++ · 1,625 stars · 347 forks · +2 stars in 24 hours · Trending score 1.34
-
17
qualcomm/ai-hub-models
Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
GitHub · Python · 1,265 stars · 221 forks · +4 stars in 24 hours · Trending score 1.27
-
18
jaswanthsanjay88/Bit
An Android Agentic harness supports all local and remote models.
GitHub · C++ · 24 stars · 3 forks · +4 stars in 24 hours · Trending score 1.25
-
19
shyringo/qwen3.8-27b-in-c
Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools; no GPU or Python. | 单颗笔记本 CPU 运行 Qwen3.8-27B:最快 2.52 token/s,最低 8 GB 内存可运行,推理加速不牺牲准确性。原生 OpenAI 兼容接口支持函数工具;无需 GPU 或 Python。
GitHub · C · 24 stars · 6 forks · +4 stars in 24 hours · Trending score 1.16
-
20
pie-project/pie
A programmable system for high-performance model serving
GitHub · Rust · 244 stars · 40 forks · +3 stars in 24 hours · Trending score 1.09
-
21
modelplaneai/modelplane
The open source control plane for AI inference
GitHub · Python · 242 stars · 38 forks · +1 stars in 24 hours · Trending score 1.08
-
22
Picom-code/pico-dgx-ds4
DeepSeek V4.1 Flash on 4x DGX Spark: 90/144 tok/s chat/code single-user, 376 tok/s at 16 distinct users, 1M context, images. Fork of blaesus/dsgx10.
GitHub · Cuda · 121 stars · 23 forks · 0 stars in 24 hours · Trending score 1.07
-
23
ewhacc/tandem.cpp
llama.cpp patch: faster MoE offload for models that don't fit in VRAM
GitHub · Python · 104 stars · 18 forks · 0 stars in 24 hours · Trending score 1.00
-
24
pylint-dev/astroid
A common base representation of python source code for pylint and other projects
GitHub · Python · 590 stars · 398 forks · 0 stars in 24 hours · Trending score 0.94