Prajwal7-2002/mini-infer
Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.
GitHub repository with 5 stars and 0 forks.
Language: Python
Topics: continuous-batching, kv-cache, llm-inference, llm-serving, paged-attention, prefix-caching, pytorch, qwen2-5, speculative-decoding, vllm