consumer-gpu repositories
Trending repositories tagged consumer-gpu, ranked by trending score. Updated daily.
-
1
MakazhanAlpamys/Soup
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
GitHub · Python · 10,713 stars · 1,777 forks · +56 stars in 24 hours · Trending score 3.47
-
2
syv-ai/HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
GitHub · Python · 2,779 stars · 349 forks · +83 stars in 24 hours · Trending score 3.22
-
3
JakeATX/llamAmpere
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
GitHub · C++ · 406 stars · 51 forks · +6 stars in 24 hours · Trending score 2.24
-
4
halt95/merlin-engine
vLLM 0.33 fork for consumer GPUs: 3 x 262K sessions on Ampere,Ada and more! Qwen 3.8 Flash-Next at 192 tok/s at full 262K context, up to 17 sessions on bigger cards. Eight checkpoints incl. Qwen3.8-27B, Nemotron 3.5 Super VL, Ornith 1.5 and Hy4 Flash (new). One command: merlin serve.
GitHub · Python · 53 stars · 8 forks · +11 stars in 24 hours · Trending score 1.83
-
5
alexturkey/wan22-pascal-4gb
Wan 2.2 TI2V-5B video generation on a 4 GB GTX 1050 Ti (Pascal): stable-diffusion.cpp patches, byte-identical speedups, recipe and agent-readable install guide
GitHub · Python · 33 stars · 13 forks · +9 stars in 24 hours · Trending score 1.10
-
6
DominikBucko/qwen38-flash-next-3090
Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with hot MoE experts on the GPU and cold experts computed on the CPU, INT8 KV cache, Docker, OpenAI-compatible API.
GitHub · Python · 16 stars · 3 forks · +3 stars in 24 hours · Trending score 1.02
-
7
chenyumo166/moziAI-35B-A3B-MOE-MTP
MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.
GitHub · Jinja · 12 stars · 2 forks · +2 stars in 24 hours · Trending score 0.73
-
8
smouj/kimari-local-ai
🦁 Local AI for Consumer GPUs — Run powerful LLMs on GTX 1060/1080. No cloud. No subscriptions. Built on llama.cpp + CUDA.
GitHub · Python · 10 stars · 1 forks · 0 stars in 24 hours · Trending score 0.22
-
9
yuuki-net/FreeToken-Kai
Unofficial FreeToken fork: Flash-Next 125B at 38 tok/s on two RTX 3060s with 2-bit experts, gpt-oss-120b on one; a benchmark tunes your settings. Qwen3.8-Flash-Next, 12 GB, MTP, 250k context, image input, WSL2. Runs on Turing: RTX 2060, RTX 20 series, sm_75.
GitHub · Python · 11 stars · 2 forks · 0 stars in 24 hours · Trending score 0.07
-
10
lvyufeng/PocketLLM
Inference engine for large language models on consumer GPUs, with deep optimization for older hardware
GitHub · Python · 55 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00