inference repositories
Trending repositories tagged inference, ranked by trending score. Updated daily.
-
1
openvinotoolkit/openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
GitHub · C++ · 11,649 stars · 3,629 forks · +348 stars in 24 hours · Trending score 4.53
-
2
FlashML-org/FreeVideo
Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
GitHub · Python · 3,578 stars · 487 forks · +194 stars in 24 hours · Trending score 4.29
-
3
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
GitHub · Python · 97,255 stars · 25,073 forks · +109 stars in 24 hours · Trending score 4.26
-
4
ggml-org/whisper.cpp
Port of OpenAI's Whisper model in C/C++
GitHub · C++ · 55,359 stars · 6,336 forks · +70 stars in 24 hours · Trending score 3.62
-
5
FlashML-org/FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
GitHub · Python · 17,331 stars · 1,721 forks · +44 stars in 24 hours · Trending score 3.43
-
6
vllm-project/vllm-omni
A framework for efficient model inference with omni-modality models
GitHub · Python · 7,856 stars · 2,162 forks · +36 stars in 24 hours · Trending score 3.29
-
7
Veloresearch/kinetic
The open engine for running AI on your own computer. Fast, exact, open source.
GitHub · Rust · 202 stars · 20 forks · +62 stars in 24 hours · Trending score 3.17
-
8
RunanywhereAI/trnrt
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
GitHub · Python · 1,639 stars · 24 forks · +31 stars in 24 hours · Trending score 3.16
-
9
Helldez/BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
GitHub · C++ · 1,033 stars · 107 forks · +23 stars in 24 hours · Trending score 3.07
-
10
vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
GitHub · Python · 3,124 stars · 2,788 forks · +24 stars in 24 hours · Trending score 2.96
-
11
RustingStudio/RustingStrata
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
GitHub · Rust · 291 stars · 36 forks · +31 stars in 24 hours · Trending score 2.90
-
12
sgl-project/sglang-omni
High Performance Serving Runtime For Interactive Super Intelligence System
GitHub · Python · 1,827 stars · 762 forks · +15 stars in 24 hours · Trending score 2.83
-
13
llm-d/llm-d
State of the art inference performance with modern accelerators on Kubernetes
GitHub · Python · 5,319 stars · 944 forks · +25 stars in 24 hours · Trending score 2.83
-
14
kvcache-ai/Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
GitHub · C++ · 7,088 stars · 1,507 forks · +22 stars in 24 hours · Trending score 2.80
-
15
sybil-solutions/inference-optimisation-skill
Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
GitHub · Python · 1,284 stars · 114 forks · +372 stars in 24 hours · Trending score 2.77
-
16
stas00/ml-engineering
Machine Learning Engineering Open Book
GitHub · Python · 19,732 stars · 1,304 forks · +26 stars in 24 hours · Trending score 2.67
-
17
vllm-project/semantic-router
An open, programmable decision layer for models and compute.
GitHub · Go · 6,512 stars · 1,140 forks · +17 stars in 24 hours · Trending score 2.66
-
18
LMCache/LMCache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
GitHub · Python · 12,398 stars · 2,174 forks · +18 stars in 24 hours · Trending score 2.63
-
19
DevDock-AI/Speedy
Qwen3.8-Flash-Next (125B MoE) up to 1M context on one or two RTX cards (tuned on 3080 10gb). A measured, tuned and benchmarked fork of Strata.
GitHub · C++ · 125 stars · 8 forks · +26 stars in 24 hours · Trending score 2.59
-
20
aryamandarda/tamserve
Live, multi-camera object tracking with SAM3, SAM 2 and EdgeTAM, on one GPU or many. One model per GPU serves every camera: faster than the original code, about 4x less GPU memory, the same accuracy.
GitHub · Python · 255 stars · 14 forks · +163 stars in 24 hours · Trending score 2.55
-
21
google-ai-edge/mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
GitHub · C++ · 37,753 stars · 6,236 forks · +13 stars in 24 hours · Trending score 2.54
-
22
chigwell/llm7.io
LLM7.io offers a single API gateway that connects you to a wide array of leading AI models from various providers.
GitHub · TypeScript · 265 stars · 14 forks · +25 stars in 24 hours · Trending score 2.47
-
23
SemiAnalysisAI/InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
GitHub · Python · 2,069 stars · 382 forks · +15 stars in 24 hours · Trending score 2.47
-
24
gpustack/gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42