ahtx/kindi
A custom LLM inference engine for low-cost GPUs: NVIDIA cards from a 1 GB Quadro K420 to an RTX 3080, the AMD Radeon 780M and the CPU. Runs the Bonsai 2B to 27B models and Nemotron with an OpenAI-compatible server.
GitHub repository with 114 stars and 29 forks.
Language: Python
Topics: amd, ampere, bitnet, cuda, kepler, llm-inference, local-llm, low-cost-gpu, maxwell, openai-compatible