local-inference repositories

Trending repositories tagged local-inference, ranked by trending score. Updated daily.

  1. 1
    huawei-csl/MACaQ

    SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.

    GitHub · C++ · 381 stars · 22 forks · +22 stars in 24 hours · Trending score 2.98

  2. 2
    Anionex/h3-image-mlx

    Faster local MiniMax H3 Image generation and editing on Apple Silicon with MLX, with ultra-low ram

    GitHub · Python · 80 stars · 11 forks · +6 stars in 24 hours · Trending score 2.17

  3. 3
    NeuroBrix/neurobrix

    Run real AI models on the GPU you already own: language, speech, image and video models from one engine and one container format, with or without PyTorch. NVIDIA GPUs, Mac GPUs through Metal, CPU; other GPUs experimental. Apache 2.0.

    GitHub · Python · 67 stars · 0 forks · +8 stars in 24 hours · Trending score 1.62

  4. 4
    ollaya-dev/ollaya

    Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

    GitHub · Rust · 1,473 stars · 93 forks · +3 stars in 24 hours · Trending score 1.44

  5. 5
    ardanlabs/kronk

    Go With Your Own Intelligence! Use Go for hardware accelerated local inference with llama.cpp, whisper.cpp, and stable-diffusion.cpp directly integrated into your Go applications. Kronk provides a high-level API and production ready model server.

    GitHub · Go · 871 stars · 65 forks · +3 stars in 24 hours · Trending score 1.04

  6. 6
    san-digital/tokspur

    Private chat and code on your Mac. Run a 177B model on a 128 GB M5 Max: local Qwen3.8 Flash-Next inference with a 65,536-token default or optional 131,072-token context, Tokspur's own Metal engine with native MTP or DFlash2 drafting, and an OpenAI-compatible API. Apache 2.0.

    GitHub · C++ · 26 stars · 2 forks · +1 stars in 24 hours · Trending score 0.94

  7. 7
    Magnvs7/Promethevs

    PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.

    GitHub · 193 stars · 12 forks · −1 stars in 24 hours · Trending score 0.84

  8. 8
    ahwurm/localharness

    An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harness (detailed event steams, logs, and performance data) or quickly replace closed source one (featuring out of the box plugins for web search, image gen, memory, auto research and more in dev).

    GitHub · Python · 74 stars · 7 forks · 0 stars in 24 hours · Trending score 0.81

  9. 9
    lna-lab/flash-next-8gb

    Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on disk; the experts run on CPU.

    GitHub · Python · 36 stars · 10 forks · 0 stars in 24 hours · Trending score 0.59

  10. 10
    siamcafe/bomllm

    Self-hosted Thai AI infrastructure — Mac Mini M4 Pro running TURBO-Fable 27B at 18.5 tok/s for $0.54/day. Fleet of 9 machines, GOD CONFIG v7.0, 40 IRON safety rules. Production-grade Open WebUI + LiteLLM + Ollama MLX stack.

    GitHub · Python · 48 stars · 9 forks · 0 stars in 24 hours · Trending score 0.52

  11. 11
    c1t1zen1/markgitup-research-desk

    Autonomous hourly research desk: a local LLM picks a fresh angle from 24 standing AI/policy/markets topic families, SearXNG gathers the evidence, and nothing publishes without 2+ independent sources. Static site, zero dependencies, built with Hermes Agent.

    GitHub · Python · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10

  12. 12
    KyaniteLabs/qwen38-27b-strix-halo

    Qwen 3.8 27B on a $1,400 Strix Halo: frozen optimal config, 262k needle-exact retrieval, vision fixed, delegation-card certified

    GitHub · Python · 14 stars · 0 forks · 0 stars in 24 hours · Trending score 0.07

  13. 13
    launcher-rs/audio-cpp-rs

    audio.cpp(基于 ggml 的本地音频推理引擎)的 Rust 安全绑定与 FFI 封装:TTS / ASR / VAD / 说话人分离 / 音源分离 / 声音克隆 / 音频编解码 / 音乐生成,覆盖 135 个模型族,后端支持 CPU、CUDA、HIP、Vulkan、Metal,提供离线与流式 API 与预编译库自动下载。

    GitHub · Rust · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.07

  14. 14
    loomax-labs/InferrLM

    InferrLM - On-device AI for iOS & Android

    GitHub · TypeScript · 113 stars · 24 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    alesha-pro/tools

    Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig

    GitHub · HTML · 687 stars · 77 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    faelion/BehaviorLLM

    Open-source Unity package for local LLM-driven NPC and game-system decisions, with configurable actions and playable samples.

    GitHub · C# · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    sahibzada-allahyar/gliner2-ultrafast

    Real browser automation with local GLiNER2 inference and open weights. Work in progress. Contributions welcome.

    GitHub · Python · 81 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    yzfly/edgejev

    离线可用的本地类型化决策:4 核 CPU 单题 15.6ms。Local & offline Jev / System One inference on CPU — ONNX + INT8, no torch at runtime. 支持 laya / kev / PlayJev

    GitHub · Python · 34 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00

  19. 19
    bladedevoff/stuntd

    Local proxy that learns your app's typed LLM decisions and answers them with a Laya head. Jev and OpenAI compatible.

    GitHub · Python · 73 stars · 5 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    Phocinae/Phocinae-Largha-150M-v1

    斑海豹 — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions. 小海豹,大决断。

    GitHub · Python · 572 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00