llm-inference repositories

Trending repositories tagged llm-inference, ranked by trending score. Updated daily.

  1. 1
    openvinotoolkit/openvino

    OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

    GitHub · C++ · 11,649 stars · 3,629 forks · +348 stars in 24 hours · Trending score 4.53

  2. 2
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37

  3. 3
    bojieli/ai-infra-book

    《深入理解 AI Infra:量化分析与系统设计》(李博杰 著)开源书稿:从硬件约束和模型架构出发,量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

    GitHub · Python · 8,154 stars · 666 forks · +93 stars in 24 hours · Trending score 3.93

  4. 4
    incoai/splash

    A local inference engine for Apple silicon, built around the model.

    GitHub · C++ · 2,543 stars · 245 forks · +112 stars in 24 hours · Trending score 3.77

  5. 5
    datawhalechina/zero-to-sglang

    Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

    GitHub · Python · 2,295 stars · 188 forks · +103 stars in 24 hours · Trending score 3.69

  6. 6
    mw00/project-maya

    Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.

    GitHub · C++ · 3,094 stars · 430 forks · +69 stars in 24 hours · Trending score 3.63

  7. 7
    ashhart/TensorFold

    LLM Inference Engine for Metal, CUDA and Vulkan.

    GitHub · Zig · 2,352 stars · 549 forks · +30 stars in 24 hours · Trending score 3.62

  8. 8
    wici-ai/inferssd

    Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    GitHub · C++ · 1,262 stars · 113 forks · +32 stars in 24 hours · Trending score 3.48

  9. 9
    JakeATX/llamAmpere

    llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

    GitHub · C++ · 403 stars · 50 forks · +87 stars in 24 hours · Trending score 3.35

  10. 10
    felladrin/awesome-ai-web-search

    List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search

    GitHub · HTML · 1,520 stars · 154 forks · +55 stars in 24 hours · Trending score 3.14

  11. 11
    FredYakumo/zihuan-next

    Zihuan is a high-performance, highly customizable Rust-based AI Agent platform. Zihuan Impls a interesting Workspace Agent (zihuan code for coding) and an IM Agent (QQ bots, etc.)

    GitHub · Rust · 101 stars · 4 forks · +60 stars in 24 hours · Trending score 2.98

  12. 12
    ray-project/ray

    Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

    GitHub · Python · 44,347 stars · 8,298 forks · +28 stars in 24 hours · Trending score 2.93

  13. 13
    flashinfer-ai/flashinfer

    FlashInfer: Kernel Library for LLM Serving

    GitHub · Cuda · 6,961 stars · 1,773 forks · +21 stars in 24 hours · Trending score 2.81

  14. 14
    ai-dynamo/dynamo

    A Datacenter Scale Distributed Inference Serving Framework

    GitHub · Rust · 8,537 stars · 1,882 forks · +13 stars in 24 hours · Trending score 2.63

  15. 15
    loxilb-io/loxilb-inference-gateway

    Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb

    GitHub · Go · 71 stars · 14 forks · +18 stars in 24 hours · Trending score 2.62

  16. 16
    nace-ai/kapitsa

    Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai

    GitHub · C++ · 833 stars · 115 forks · +14 stars in 24 hours · Trending score 2.56

  17. 17
    AtomicBot-ai/Atomic-Chat

    Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

    GitHub · TypeScript · 2,034 stars · 234 forks · +21 stars in 24 hours · Trending score 2.50

  18. 18
    SemiAnalysisAI/InferenceX

    Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

    GitHub · Python · 2,069 stars · 382 forks · +15 stars in 24 hours · Trending score 2.47

  19. 19
    gpustack/gpustack

    A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

    GitHub · Python · 6,086 stars · 746 forks · +11 stars in 24 hours · Trending score 2.42

  20. 20
    sergqwer/strata-nvfp4

    Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.

    GitHub · C++ · 251 stars · 31 forks · +23 stars in 24 hours · Trending score 2.42

  21. 21
    OmniJev/Flide

    🔥🔥 The extremely fast inference engine for decision model

    GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36

  22. 22
    lemonade-sdk/lemonade

    Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

    GitHub · C++ · 6,253 stars · 571 forks · +8 stars in 24 hours · Trending score 2.23

  23. 23
    pallavi-shekhar/ai-engineering-interview-questions-company-wise

    Your Cheat Sheet For AI Engineering Interviews at Top AI Companies - Questions and Answers.

    GitHub · Markdown · 2,093 stars · 214 forks · +11 stars in 24 hours · Trending score 2.21

  24. 24
    tetherto/qvac

    Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

    GitHub · TypeScript · 791 stars · 165 forks · +12 stars in 24 hours · Trending score 2.19