inference-optimization repositories

Trending repositories tagged inference-optimization, ranked by trending score. Updated daily.

  1. 1
    google/XNNPACK

    High-efficiency floating-point neural network inference operators for mobile, server, and Web

    GitHub · C · 2,505 stars · 595 forks · +5 stars in 24 hours · Trending score 1.70

  2. 2
    AI-Hypercomputer/tpu-codesign

    Hardware-grounded model architecture design principles, MoE co-design patterns, low-precision numerics, and parallelism guides for training and serving on Google Cloud TPUs.

    GitHub · 90 stars · 7 forks · +25 stars in 24 hours · Trending score 1.58

  3. 3
    datawhalechina/llm-algo-leetcode

    Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。

    GitHub · Jupyter Notebook · 797 stars · 162 forks · 0 stars in 24 hours · Trending score 1.38

  4. 4
    Hiruynk/AnifLive-TTS

    A low-latency, high-quality multilingual voice-cloning TTS runtime

    GitHub · Python · 56 stars · 4 forks · 0 stars in 24 hours · Trending score 1.08

  5. 5
    Superone77/FlashLoop

    FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    GitHub · Python · 30 stars · 4 forks · 0 stars in 24 hours · Trending score 0.78

  6. 6
    Lowaltitudeflight/local-llm-tuning-guide

    A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.

    GitHub · Python · 105 stars · 0 forks · 0 stars in 24 hours · Trending score 0.72

  7. 7
    syv-ai/HyperQwen

    Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

    GitHub · Python · 2,696 stars · 340 forks · 0 stars in 24 hours · Trending score 0.61

  8. 8
    zhangshuibai/molten

    Run a coding model at 300+ tok/s on your Mac. Native Rust + Metal inference engine for LiquidAI LFM2.5-24B-A2B (MLX 4-bit) on Apple Silicon.

    GitHub · Metal · 29 stars · 2 forks · +1 stars in 24 hours · Trending score 0.59

  9. 9
    modular/llm-inference-handbook

    Everything you need to know about LLM inference

    GitHub · TypeScript · 489 stars · 61 forks · 0 stars in 24 hours · Trending score 0.36

  10. 10
    Brontir/NInferno

    High-performance single-GPU inference for (Qwen, Gemma, Nemotron & other) checkpoints and GPUs.

    GitHub · C++ · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.31

  11. 11
    david-xinyuwei/david-share

    AI Agents & Deep Learning: code, benchmarks and explainers. Building and operating agents on Microsoft Foundry and Azure; fine-tuning, inference optimization, precision and GPU benchmarks on NVIDIA and AMD; multimodal models.

    GitHub · Jupyter Notebook · 451 stars · 92 forks · 0 stars in 24 hours · Trending score 0.20

  12. 12
    redhat-performance/configiq

    Distributed inference configuration tools

    GitHub · TypeScript · 23 stars · 13 forks · 0 stars in 24 hours · Trending score 0.10

  13. 13
    truespar/paddock

    Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

    GitHub · Rust · 164 stars · 25 forks · 0 stars in 24 hours · Trending score 0.08

  14. 14
    YuhuaBillChen/justfit-llm

    Run Qwen3.8-27B with its full 256K context on a 24 GB Mac. Long-context local LLM serving for Apple silicon, built on Inco AI's Splash. OpenAI-compatible API.

    GitHub · C++ · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  15. 15
    benmaster82/picchio

    A streaming Mixture-of-Experts (MoE) inference engine written in pure C, that runs models larger than your RAM on ordinary consumer hardware - MiniMax-M2 (230B) - GPT-OSS (20B and 120B) and Qwen3-MoE

    GitHub · C · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.05

  16. 16
    cl-vv-h/SGLang_Tutor

    Bilingual SGLang tutorials and source-code walkthroughs for LLM inference, AI infrastructure, scheduling, KV cache, attention, Ascend NPU, and kernel optimization. | SGLang 中英双语教程与源码解析

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.03

  17. 17
    amitshekhariitbhu/llm-inference-engineering

    Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

    GitHub · Markdown · 405 stars · 50 forks · 0 stars in 24 hours · Trending score 0.02

  18. 18
    pleasedodisturb/awesome-llm-token-optimization

    A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

    GitHub · Shell · 119 stars · 39 forks · 0 stars in 24 hours · Trending score 0.02

  19. 19
    Scottcjn/ram-coffers

    LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.

    GitHub · Python · 173 stars · 59 forks · 0 stars in 24 hours · Trending score 0.01

  20. 20
    htobty/ReadyLLM

    Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

    GitHub · Python · 100 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    brontoguana/krasis

    Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

    GitHub · C++ · 522 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    tanavc1/local-llm-autotune

    Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

    GitHub · Python · 37 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    qr0n/p100-llama-cpp

    Patches for running llama.cpp fast on Tesla P100 (GP100/sm_60)

    GitHub · C++ · 15 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    andrei-borcea/liyab

    🔥 Run large open language models on your phone. A C++20 GGUF inference engine that streams models larger than RAM from flash, paces itself by the device's thermal forecast, and powers a private on-device assistant. Android · iOS · macOS · Linux.

    GitHub · C++ · 15 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00