davanstrien/saturate
Batch inference for datasets: rows in, any OpenAI-compatible endpoint, resumable parquet out. Congestion-aware concurrency — no tuning.
GitHub repository with 54 stars and 2 forks.
Language: Python
Topics: batch-inference, huggingface, llm-inference, openai-api, parquet, sglang, vllm