affifuddin-hazam/ghost-infer
Fast local LLM inference for the GPU you already have: one Rust binary, GGUF models, an OpenAI-compatible server and a chat page with a ghost.
GitHub repository with 237 stars and 15 forks.
Language: Rust
Topics: cuda, desktop-app, gguf, llm, llm-inference, local-llm, mixture-of-experts, openai-compatible-api, rust, windows