GitHub · Python
awesome-pro/hiqcache
INT8-quantized hierarchical KV cache for SGLang HiCache. Compresses KV evicted from GPU L1 into the CPU L2 tier, holding 1.78x more tokens in the same host memory. Measured on Qwen3-8B against a BF16 baseline.
- Stars
- 11
- Forks
- 0
- Stars gained in 24 hours
- 0
- Forks gained in 24 hours
- 0
Topics: cuda, hicache, int8, kv-cache, llm-serving, memory-efficiency, pytorch, quantization, sglang
Latest data point: Nov 22, 2026 · 11 stars · 0 forks