cpu-inference repositories

Trending repositories tagged cpu-inference, ranked by trending score. Updated daily.

  1. 1
    FareedKhan-dev/kimi-k3-in-c

    A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

    GitHub · C · 12,510 stars · 1,935 forks · +127 stars in 24 hours · Trending score 4.17

  2. 2
    synchopate/llama-modchip

    Zero-GPU CPU inference for 30B-284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.

    GitHub · Python · 1,393 stars · 135 forks · +33 stars in 24 hours · Trending score 3.35

  3. 3
    saurabhv749/supertonic3-voice-clone

    Train voice styles for Supertone/supertonic-3 model.

    GitHub · Python · 89 stars · 25 forks · +13 stars in 24 hours · Trending score 1.95

  4. 4
    shyringo/qwen3.8-flash-next-in-c

    The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic 8 GB RAM path. Native C, no GPU or Python. | 200B 以下最强模型,单颗笔记本 CPU 精确批处理接近 10 token/s,最低 8 GB 内存自动运行。原生 C 语言,无需 GPU 或 Python。

    GitHub · C · 24 stars · 8 forks · +6 stars in 24 hours · Trending score 1.45

  5. 5
    prabanta-dev/janas

    37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

    GitHub · C · 735 stars · 73 forks · 0 stars in 24 hours · Trending score 1.28

  6. 6
    DaggerSlicer34/koboldcpp

    KoboldCpp — local llm cpu.

    GitHub · 21 stars · 0 forks · +13 stars in 24 hours · Trending score 1.20

  7. 7
    yanfei0725/dsh-qwen-paint

    DSH 插件 / DeepSeek Harness:视频 / 生图 / TTS 全能 · 本机运行(QQ 交流群 454707710)。★ 0.33.0 起许可★ 0.33.0 起许可改为 BSL 1.1(BUSL-1.1,非 OSI 开源许可):除托管/代管服务与直接竞品捆绑两类「竞争性使用」外免费生产使用,2030-11-15 转 Apache-2.0;0.32.1 及更早仍 MIT、不追溯。★ 1.0.0:选模型「假成功」根治(拒绝时如实回未生效 + ★ 1.0.0:选模型「假成功」根治(拒绝时如实回未生效 + 原因,不再无条件报已选)、优化面板与素材库白条同源修复(持久化缓存

    GitHub · JavaScript · 45 stars · 0 forks · +1 stars in 24 hours · Trending score 0.87

  8. 8
    shyringo/qwen3.8-27b-in-c

    Qwen3.8-27B on one laptop CPU: up to 2.52 token/s, 8 GB tested, no runtime accuracy loss. Native OpenAI-compatible API with function tools; no GPU or Python. | 单颗笔记本 CPU 运行 Qwen3.8-27B:最快 2.52 token/s,最低 8 GB 内存可运行,推理加速不牺牲准确性。原生 OpenAI 兼容接口支持函数工具;无需 GPU 或 Python。

    GitHub · C · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.42

  9. 9
    sauravsingla/MemVanta

    Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU — mmap-backed weights, paged KV cache, Q4/Q8 kernels, and reproducible llama.cpp benchmarks.

    GitHub · C++ · 41 stars · 40 forks · 0 stars in 24 hours · Trending score 0.37

  10. 10
    maheshvaikri-code/snapllm

    🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp

    GitHub · C++ · 44 stars · 4 forks · 0 stars in 24 hours · Trending score 0.32

  11. 11
    DavidTorro/gdg-uam-ml-contest-2026

    1D CNN for 8-class multichannel time-series classification, optimized for CPU inference with ONNX Runtime and Numba.

    GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.26

  12. 12
    siris9476/pulsarforge

    GitHub · C · 24 stars · 4 forks · 0 stars in 24 hours · Trending score 0.14

  13. 13
    Dicklesworthstone/franken_nlp

    Pure-Rust, CPU-optimized local NLP toolbox built around one model (Nanbeige4.2-3B): schema-valid structured extraction, NER and entity resolution, sentiment, zero-shot classification, PII redaction, faithfulness judging, QA, and text generation. One library plus the fnlp CLI; no Python, CUDA, or GPU. Pre-release.

    GitHub · Rust · 7 stars · 2 forks · 0 stars in 24 hours · Trending score 0.10

  14. 14
    Classevelabs/rai

    Run language models on your own computer's CPU: one-step installers for Windows, macOS and Linux, a local chat page and API, 4-bit models, pure Rust. No GPU, no Python, nothing leaves your machine.

    GitHub · Rust · 16 stars · 1 forks · 0 stars in 24 hours · Trending score 0.08

  15. 15
    ranjithrajv/awesome-cpu-first-ai

    Curated, evidence-backed list of runtimes, formats & tools for running AI inference on CPU — start with CPU, justify the GPU.

    GitHub · Python · 5 stars · 4 forks · 0 stars in 24 hours · Trending score 0.07

  16. 16
    Endgame-Labs/understudy

    A drop-in, CPU-only proxy for System One models (Jev, Rune, Clef). It learns each repeated question from the lead, then answers it itself.

    GitHub · Rust · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.05

  17. 17
    benmaster82/picchio

    A streaming Mixture-of-Experts (MoE) inference engine written in pure C, that runs models larger than your RAM on ordinary consumer hardware - MiniMax-M2 (230B) - GPT-OSS (20B and 120B) and Qwen3-MoE

    GitHub · C · 11 stars · 3 forks · 0 stars in 24 hours · Trending score 0.04

  18. 18
    ronitgupta138/Local-Jev

    ⚡ 100% offline, zero-cost drop-in alternative to TypeSafe Jev System One decision models in Rust (<18ms on CPU)

    GitHub · Rust · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.03

  19. 19
    mcpware/IntentProbe

    CPU activation probes for AI-agent guardrails and classification. One frozen local model pass, many independent heads. Open models, code and reproducible benchmarks.

    GitHub · Python · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    brontoguana/krasis

    Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

    GitHub · C++ · 522 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    feder-cr/jev

    jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.

    GitHub · C++ · 1,345 stars · 163 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    shyringo/deepseek-v4-flash-0731-in-c

    Run the native 284B-A13B DeepSeek-V4-Flash-0731 LLM locally on one laptop CPU: pure C, 8 GB RAM minimum, no GPU, best TPOT 0.892 s/token, resident OpenAI-compatible API with function tools. | 在笔记本单颗 CPU 上本地运行原生 284B-A13B DeepSeek-V4-Flash-0731 大模型:纯 C,最低 8 GB 内存,无需 GPU,最优 TPOT 0.892 秒/token,支持模型常驻的 OpenAI 兼容接口与函数工具。

    GitHub · C · 29 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    Tahirpathan-AiLab/Evaan_Personal_Intelligence_Engine

    Evaan — a fully local, offline-capable AI companion chatbot built with Python & Hugging Face Transformers, running on Qwen2.5-0.5B-Instruct (CPU-only, no fine-tuning). Features a persona-driven system prompt, rule-based mood/tone detection, and persistent JSON memory across sessions — no API key or internet needed after setup.

    GitHub · Python · 23 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    IronHpc/UVR-rs

    群友让我优化一下UVR5的4个常用的模型的推理速度,以及运行时环境的大小

    GitHub · Rust · 5 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00