inference-optimization repositories
Trending repositories tagged inference-optimization, ranked by trending score. Updated daily.
-
1
google/XNNPACK
High-efficiency floating-point neural network inference operators for mobile, server, and Web
GitHub · C · 2,505 stars · 595 forks · +5 stars in 24 hours · Trending score 1.70
-
2
AI-Hypercomputer/tpu-codesign
Hardware-grounded model architecture design principles, MoE co-design patterns, low-precision numerics, and parallelism guides for training and serving on Google Cloud TPUs.
GitHub · 90 stars · 7 forks · +25 stars in 24 hours · Trending score 1.58
-
3
datawhalechina/llm-algo-leetcode
Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。
GitHub · Jupyter Notebook · 797 stars · 162 forks · 0 stars in 24 hours · Trending score 1.38
-
4
Hiruynk/AnifLive-TTS
A low-latency, high-quality multilingual voice-cloning TTS runtime
GitHub · Python · 56 stars · 4 forks · 0 stars in 24 hours · Trending score 1.08
-
5
Superone77/FlashLoop
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
GitHub · Python · 30 stars · 4 forks · 0 stars in 24 hours · Trending score 0.78
-
6
Lowaltitudeflight/local-llm-tuning-guide
A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.
GitHub · Python · 105 stars · 0 forks · 0 stars in 24 hours · Trending score 0.72
-
7
syv-ai/HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
GitHub · Python · 2,696 stars · 340 forks · 0 stars in 24 hours · Trending score 0.61
-
8
zhangshuibai/molten
Run a coding model at 300+ tok/s on your Mac. Native Rust + Metal inference engine for LiquidAI LFM2.5-24B-A2B (MLX 4-bit) on Apple Silicon.
GitHub · Metal · 29 stars · 2 forks · +1 stars in 24 hours · Trending score 0.59
-
9
modular/llm-inference-handbook
Everything you need to know about LLM inference
GitHub · TypeScript · 489 stars · 61 forks · 0 stars in 24 hours · Trending score 0.36
-
10
Brontir/NInferno
High-performance single-GPU inference for (Qwen, Gemma, Nemotron & other) checkpoints and GPUs.
GitHub · C++ · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.31
-
11
david-xinyuwei/david-share
AI Agents & Deep Learning: code, benchmarks and explainers. Building and operating agents on Microsoft Foundry and Azure; fine-tuning, inference optimization, precision and GPU benchmarks on NVIDIA and AMD; multimodal models.
GitHub · Jupyter Notebook · 451 stars · 92 forks · 0 stars in 24 hours · Trending score 0.20
-
12
redhat-performance/configiq
Distributed inference configuration tools
GitHub · TypeScript · 23 stars · 13 forks · 0 stars in 24 hours · Trending score 0.10
-
13
truespar/paddock
Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
GitHub · Rust · 164 stars · 25 forks · 0 stars in 24 hours · Trending score 0.08
-
14
YuhuaBillChen/justfit-llm
Run Qwen3.8-27B with its full 256K context on a 24 GB Mac. Long-context local LLM serving for Apple silicon, built on Inco AI's Splash. OpenAI-compatible API.
GitHub · C++ · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
15
benmaster82/picchio
A streaming Mixture-of-Experts (MoE) inference engine written in pure C, that runs models larger than your RAM on ordinary consumer hardware - MiniMax-M2 (230B) - GPT-OSS (20B and 120B) and Qwen3-MoE
GitHub · C · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.05
-
16
cl-vv-h/SGLang_Tutor
Bilingual SGLang tutorials and source-code walkthroughs for LLM inference, AI infrastructure, scheduling, KV cache, attention, Ascend NPU, and kernel optimization. | SGLang 中英双语教程与源码解析
GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.03
-
17
amitshekhariitbhu/llm-inference-engineering
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
GitHub · Markdown · 405 stars · 50 forks · 0 stars in 24 hours · Trending score 0.02
-
18
pleasedodisturb/awesome-llm-token-optimization
A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.
GitHub · Shell · 119 stars · 39 forks · 0 stars in 24 hours · Trending score 0.02
-
19
Scottcjn/ram-coffers
LLM infrastructure cost reduction via NUMA-aware weight banking: 147 t/s (8.8x stock llama.cpp) on refurbished enterprise POWER8. Self-hosted inference, no cloud APIs. Part of the Proof of Physical AI stack.
GitHub · Python · 173 stars · 59 forks · 0 stars in 24 hours · Trending score 0.01
-
20
htobty/ReadyLLM
Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.
GitHub · Python · 100 stars · 12 forks · 0 stars in 24 hours · Trending score 0.00
-
21
brontoguana/krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
GitHub · C++ · 522 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00
-
22
tanavc1/local-llm-autotune
Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.
GitHub · Python · 37 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
23
qr0n/p100-llama-cpp
Patches for running llama.cpp fast on Tesla P100 (GP100/sm_60)
GitHub · C++ · 15 stars · 3 forks · 0 stars in 24 hours · Trending score 0.00
-
24
andrei-borcea/liyab
🔥 Run large open language models on your phone. A C++20 GGUF inference engine that streams models larger than RAM from flash, paces itself by the device's thermal forecast, and powers a private on-device assistant. Android · iOS · macOS · Linux.
GitHub · C++ · 15 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00