quantization repositories
Trending repositories tagged quantization, ranked by trending score. Updated daily.
-
1
FareedKhan-dev/kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
GitHub · C · 12,485 stars · 1,932 forks · +181 stars in 24 hours · Trending score 4.37
-
2
hyperionaidev/smolmaxxing
maxxing the ability to run different LLMs on the smallest devices possible
GitHub · C · 1,035 stars · 54 forks · +65 stars in 24 hours · Trending score 3.88
-
3
mw00/project-maya
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.
GitHub · C++ · 3,094 stars · 430 forks · +69 stars in 24 hours · Trending score 3.63
-
4
synchopate/llama-modchip
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
GitHub · Python · 1,387 stars · 134 forks · +38 stars in 24 hours · Trending score 3.55
-
5
JakeATX/llamAmpere
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
GitHub · C++ · 403 stars · 50 forks · +87 stars in 24 hours · Trending score 3.35
-
6
hiyouga/LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
GitHub · Python · 76,148 stars · 9,394 forks · +48 stars in 24 hours · Trending score 3.32
-
7
huawei-csl/MACaQ
SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.
GitHub · C++ · 380 stars · 22 forks · +22 stars in 24 hours · Trending score 2.98
-
8
animede/Diffusers_minimax-h3
MiniMax-H3 (video + synchronized audio, 33B) on diffusers — 9.7s per still, 44.2s per 5s clip on 48GB, with every speedup verified byte-identical
GitHub · Python · 61 stars · 6 forks · +25 stars in 24 hours · Trending score 2.61
-
9
prabanta-dev/janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
GitHub · C · 735 stars · 73 forks · +11 stars in 24 hours · Trending score 2.41
-
10
Indras-Mirror/Xolotl
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
GitHub · C++ · 545 stars · 67 forks · +12 stars in 24 hours · Trending score 2.18
-
11
Anionex/h3-image-mlx
Faster local MiniMax H3 Image generation and editing on Apple Silicon with MLX, with ultra-low ram
GitHub · Python · 80 stars · 11 forks · +6 stars in 24 hours · Trending score 2.17
-
12
hiruhiru2005/sync-strata
A one-click local LLM app for Windows and Linux: Qwen3.8-Flash-Next with the Sync quantizations (XS, S, M), based on Strata.
GitHub · C++ · 69 stars · 5 forks · +48 stars in 24 hours · Trending score 2.14
-
13
amitshekhariitbhu/ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
GitHub · Markdown · 3,679 stars · 656 forks · +7 stars in 24 hours · Trending score 2.10
-
14
vllm-project/llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
GitHub · Python · 3,977 stars · 739 forks · +7 stars in 24 hours · Trending score 1.86
-
15
TandemLLM/TandemEngine
TandemEngine: local inference that drafts and verifies its own tokens, behind an OpenAI-compatible API.
GitHub · Python · 82 stars · 17 forks · +8 stars in 24 hours · Trending score 1.73
-
16
warpfront/hipfire
RDNA-native LLM inference engine in Rust.
GitHub · Rust · 819 stars · 109 forks · +5 stars in 24 hours · Trending score 1.64
-
17
AI-Hypercomputer/tpu-codesign
Hardware-grounded model architecture design principles, MoE co-design patterns, low-precision numerics, and parallelism guides for training and serving on Google Cloud TPUs.
GitHub · 90 stars · 7 forks · +25 stars in 24 hours · Trending score 1.58
-
18
RyanCodrai/turbovec
A vector index built on TurboQuant, written in Rust with Python bindings
GitHub · Rust · 17,503 stars · 1,483 forks · +6 stars in 24 hours · Trending score 1.51
-
19
Tencent/AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
GitHub · Python · 1,806 stars · 199 forks · +4 stars in 24 hours · Trending score 1.51
-
20
jaedync/crucible-strix-halo
Run Qwen3.8-Flash-Next on one AMD Strix Halo (Ryzen AI Max+ 395, gfx1151). EXL3 4.05 and 3.05 bpw packs, own ROCm/HIP kernels, OpenAI-compatible server, 4 sessions of 262K context, setup guide included.
GitHub · Python · 35 stars · 7 forks · +2 stars in 24 hours · Trending score 1.45
-
21
qdrant/skills
Agent skills for Qdrant vector search: scaling, performance optimization, search quality, monitoring, deployment, model migration, version upgrades, and SDK usage across Python, TypeScript, Rust, Go, .NET, Java
GitHub · Python · 271 stars · 31 forks · +7 stars in 24 hours · Trending score 1.44
-
22
IST-DASLab/GSQ
Gumbel-Softmax post-training quantization for LLMs (1–3 bit scalar, INT/GGUF-compatible).
GitHub · Python · 143 stars · 21 forks · 0 stars in 24 hours · Trending score 1.38
-
23
datawhalechina/llm-algo-leetcode
Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。
GitHub · Jupyter Notebook · 797 stars · 162 forks · 0 stars in 24 hours · Trending score 1.38
-
24
whitecircle/halo
Halo is an open-source framework built by White Circle for training large language and multimodal models
GitHub · Python · 868 stars · 50 forks · +2 stars in 24 hours · Trending score 1.32