inference-engine repositories

Trending repositories tagged inference-engine, ranked by trending score. Updated daily.

  1. 1
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37

  2. 2
    datawhalechina/zero-to-sglang

    Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

    GitHub · Python · 2,295 stars · 188 forks · +103 stars in 24 hours · Trending score 3.69

  3. 3
    simplexai-labs/PolyStrata

    Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.

    GitHub · C++ · 1,280 stars · 166 forks · +60 stars in 24 hours · Trending score 3.63

  4. 4
    peonist-ai/halogen-flash-server

    Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

    GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66

  5. 5
    sangharshadhyeta/Strata-HotRod

    Run Qwen3.8-Flash-Next (125B) on an RTX 3060 12 GB with 30 GB of RAM, streaming experts from the SSD. 50-70 tok/s, 262K context. A fork of Strata.

    GitHub · C++ · 37 stars · 3 forks · +24 stars in 24 hours · Trending score 2.53

  6. 6
    RainBowLuoCS/Gamer

    Official Implementation of Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agents

    GitHub · 1,025 stars · 22 forks · +207 stars in 24 hours · Trending score 2.50

  7. 7
    prabanta-dev/janas

    37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

    GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41

  8. 8
    rizagp/inferzero

    arcinferzero

    GitHub · C++ · 62 stars · 11 forks · +12 stars in 24 hours · Trending score 2.40

  9. 9
    OmniJev/Flide

    🔥🔥 The extremely fast inference engine for decision model

    GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36

  10. 10
    nobodywho-ooo/nobodywho

    NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

    GitHub · Rust · 1,691 stars · 102 forks · +16 stars in 24 hours · Trending score 2.03

  11. 11
    TandemLLM/TandemEngine

    TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.

    GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73

  12. 12
    youssofal/MTPLX

    The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

    GitHub · Python · 2,727 stars · 214 forks · +6 stars in 24 hours · Trending score 1.72

  13. 13
    Luce-Org/lucebox

    Lucebox inference engine for heterogeneous hardware & consumer GPUs

    GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63

  14. 14
    NeuroBrix/neurobrix

    Run real AI models on the GPU you already own: language, speech, image and video models from one engine and one container format, with or without PyTorch. NVIDIA GPUs, Mac GPUs through Metal, CPU; other GPUs experimental. Apache 2.0.

    GitHub · Python · 67 stars · 0 forks · +8 stars in 24 hours · Trending score 1.62

  15. 15
    jaedync/crucible-strix-halo

    Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.

    GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45

  16. 16
    xLLM-AI/xllm

    A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

    GitHub · C++ · 1,625 stars · 347 forks · +2 stars in 24 hours · Trending score 1.34

  17. 17
    qualcomm/ai-hub-models

    Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

    GitHub · Python · 1,265 stars · 221 forks · +4 stars in 24 hours · Trending score 1.27

  18. 18
    jaswanthsanjay88/Bit

    An Android Agentic harness supports all local and remote models.

    GitHub · C++ · 24 stars · 3 forks · +4 stars in 24 hours · Trending score 1.25

  19. 19
    shyringo/qwen3.8-27b-in-c

    Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools; no GPU or Python. | 单颗笔记本 CPU 运行 Qwen3.8-27B:最快 2.52 token/s,最低 8 GB 内存可运行,推理加速不牺牲准确性。原生 OpenAI 兼容接口支持函数工具;无需 GPU 或 Python。

    GitHub · C · 24 stars · 6 forks · +4 stars in 24 hours · Trending score 1.16

  20. 20
    pie-project/pie

    A programmable system for high-performance model serving

    GitHub · Rust · 244 stars · 40 forks · +3 stars in 24 hours · Trending score 1.09

  21. 21
    modelplaneai/modelplane

    The open source control plane for AI inference

    GitHub · Python · 242 stars · 38 forks · +1 stars in 24 hours · Trending score 1.08

  22. 22
    Picom-code/pico-dgx-ds4

    DeepSeek V4.1 Flash on 4x DGX Spark: 90/144 tok/s chat/code single-user, 376 tok/s at 16 distinct users, 1M context, images. Fork of blaesus/dsgx10.

    GitHub · Cuda · 121 stars · 23 forks · 0 stars in 24 hours · Trending score 1.07

  23. 23
    ewhacc/tandem.cpp

    llama.cpp patch: faster MoE offload for models that don't fit in VRAM

    GitHub · Python · 104 stars · 18 forks · 0 stars in 24 hours · Trending score 1.00

  24. 24
    pylint-dev/astroid

    A common base representation of python source code for pylint and other projects

    GitHub · Python · 590 stars · 398 forks · 0 stars in 24 hours · Trending score 0.94