prefix-caching repositories

Trending repositories tagged prefix-caching, ranked by trending score. Updated daily.

  1. 1
    OmniJev/Flide

    🔥🔥 The extremely fast inference engine for decision model

    GitHub · Python · 546 stars · 37 forks · +9 stars in 24 hours · Trending score 2.36

  2. 2
    Dami-wq-k03/nano-lmcache

    📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.

    GitHub · Python · 43 stars · 13 forks · +1 stars in 24 hours · Trending score 0.96

  3. 3
    OrbitInfra/OrbitKV

    OrbitKV is a High Performance KV Cache System for LLM Inference.

    GitHub · Rust · 284 stars · 19 forks · +1 stars in 24 hours · Trending score 0.82

  4. 4
    Prajwal7-2002/mini-infer

    Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.

    GitHub · Python · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  5. 5
    alphaparkinc/genpark-speculative-decoding-draft-verifier-skill

    GenPark AI Agent Skill - Speculative decoding simulation engine coordinating small draft model speculative token generation and large target model parallel rejection sampling verification.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  6. 6
    alphaparkinc/genpark-prefix-caching-kv-state-reuse-skill

    GenPark AI Agent Skill - Radix tree prefix caching simulator for prompt templates and agent system instructions, maximizing KV-cache hit rates and cutting TTFT latency.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  7. 7
    alphaparkinc/genpark-cascading-router-cost-quality-optimizer-skill

    GenPark AI Agent Skill - Multi-tier model cascading router dynamically selecting optimal models (e.g. Small vs Medium vs Large) based on task complexity, cost budgets, and SLA constraints.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  8. 8
    alphaparkinc/genpark-prompt-compression-token-pruning-skill

    GenPark AI Agent Skill - Task-aware prompt token pruning using information density scoring and semantic stop-phrase elimination to compress long context prompts up to 60%.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    alphaparkinc/genpark-dynamic-mixture-of-agents-moa-layer-skill

    GenPark AI Agent Skill - Mixture-of-Agents (MoA) multi-layer aggregator synthesizing diverse candidate proposals from heterogeneous sub-agents into high-consensus outputs.

    GitHub · Python · 8 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    gauravapiscean/agentic-kv-cache

    Reproducing agentic KV-cache policy claims on real traces. 68k requests from 393 Claude Code sessions. LRU is harder to beat than the papers suggest.

    GitHub · Python · 24 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  11. 11
    athrael-soju/Narwhal

    Adaptive Disaggregated Inference on a Role-free Fleet!

    GitHub · Python · 8 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  12. 12
    alphaparkinc/genpark-semantic-prompt-cache-key-normalizer-and-hit-predictor-skill

    Scrubs volatile request UUIDs and timestamps from prompts to maximize deterministic prefix cache hit rates | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

    GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00