gpu-inference repositories

Trending repositories tagged gpu-inference, ranked by trending score. Updated daily.

  1. 1
    peonist-ai/halogen-flash-server

    Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

    GitHub · Shell · 1,403 stars · 101 forks · +17 stars in 24 hours · Trending score 2.66

  2. 2
    aws-samples/sample-genai-on-eks-starter-kit

    A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

    GitHub · JavaScript · 102 stars · 64 forks · +3 stars in 24 hours · Trending score 1.19

  3. 3
    shinbunbun/llama-cpp-p100-patches

    llama.cpp performance patches developed and measured on Tesla P100 (GP100, sm_60). Most are not Pascal-specific.

    GitHub · Nix · 64 stars · 5 forks · 0 stars in 24 hours · Trending score 0.76

  4. 4
    infercrane/infercrane

    Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind one endpoint.

    GitHub · Go · 32 stars · 0 forks · −8 stars in 24 hours · Trending score 0.08

  5. 5
    Raytorin/triton-openai-gateway

    OpenAI-compatible gateway for NVIDIA Triton and vLLM with tools, multimodal inputs, embeddings, reranking, and observability.

    GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.08

  6. 6
    brontoguana/krasis

    Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

    GitHub · C++ · 522 stars · 33 forks · 0 stars in 24 hours · Trending score 0.00

  7. 7
    redaER7/qwen3.8-27b-self-hosted

    Self-hosted Qwen3.8-27B (FP8) inference with vLLM, KServe and Envoy AI Gateway on RTX 6000 PRo or 2× RTX 4080 Super

    GitHub · Shell · 18 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  8. 8
    peonist-ai/halogen-server

    The fastest way to run Qwen3.8 27B on Strix Halo (gfx1151)

    GitHub · Python · 137 stars · 7 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    janishar/h3c-studio

    Local web UI for h3.c — native Metal video/audio generation with MiniMax-H3 on Apple Silicon, no PyTorch or MLX required.

    GitHub · Go · 12 stars · 4 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    yeungtuzi/Xiaotu-LLM

    XTU = X Transformers Unity — hybrid (CPU-expert + GPU) inference acceleration for vLLM: mainline plugin + CPU MoE engine (AVX512-VNNI/BF16 MXFP4/FP8), long-prefill GPU streaming, SM80 (A100) port.

    GitHub · Python · 16 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00