evaluation repositories
Trending repositories tagged evaluation, ranked by trending score. Updated daily.
-
1
NVIDIA/SkillEvaluator
Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
GitHub · Python · 4,195 stars · 546 forks · +794 stars in 24 hours · Trending score 5.61
-
2
Tencent/WeKnora
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
GitHub · Go · 37,864 stars · 5,280 forks · +151 stars in 24 hours · Trending score 4.32
-
3
langfuse/langfuse
🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
GitHub · TypeScript · 37,664 stars · 4,277 forks · +85 stars in 24 hours · Trending score 3.85
-
4
promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
GitHub · TypeScript · 27,483 stars · 2,800 forks · +65 stars in 24 hours · Trending score 3.79
-
5
mlflow/mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
GitHub · Python · 28,905 stars · 6,640 forks · +40 stars in 24 hours · Trending score 3.23
-
6
alchaincyf/darwin-skill
达尔文.skill —— 一个让你的Skill无限进化的系统:评估→改进→测试→保留或回滚 | Autoresearch-inspired autonomous skill optimization for Claude Code. Evaluate, improve, test, keep or revert.
GitHub · HTML · 7,004 stars · 725 forks · +39 stars in 24 hours · Trending score 3.08
-
7
comet-ml/opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
GitHub · Java · 23,460 stars · 1,964 forks · +10 stars in 24 hours · Trending score 2.61
-
8
zaxiu88888/issue-rag
RAG over GitHub issues, with a hand-reviewed eval set and an audit of the judge itself. Reports the interval, not just the best number.
GitHub · Python · 483 stars · 6 forks · +16 stars in 24 hours · Trending score 2.33
-
9
BUAASubnet/UBAA
更好的智慧北航. Make BUAA Great Again!
GitHub · Dart · 368 stars · 26 forks · +13 stars in 24 hours · Trending score 2.29
-
10
open-compass/opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
GitHub · Python · 7,601 stars · 915 forks · +17 stars in 24 hours · Trending score 2.26
-
11
modelscope/evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
GitHub · Python · 3,714 stars · 585 forks · +5 stars in 24 hours · Trending score 2.01
-
12
didi/helixharness
HelixHarness 提供完整的 AI Coding 工程化基础能力,还支持不同企业和团队结合自身的业务知识、工程规范和研发流程,持续构建和演进属于自己的专属 Harness
GitHub · Python · 453 stars · 62 forks · +8 stars in 24 hours · Trending score 1.99
-
13
lmnr-ai/lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
GitHub · TypeScript · 3,510 stars · 262 forks · +8 stars in 24 hours · Trending score 1.85
-
14
Valen-Team/Roller
Turn agent runs into training data. Roller rolls out Claude Code, Codex, OpenClaw and four other Harnesses in disposable Docker containers, grades benchmark tasks in place, and exports SFT-ready trajectories that keep reasoning, images and compacted-context segments.
GitHub · Python · 106 stars · 0 forks · +37 stars in 24 hours · Trending score 1.78
-
15
run-llama/ParseBench
ParseBench - A Document Parsing Benchmark for AI Agents
GitHub · Python · 648 stars · 140 forks · +5 stars in 24 hours · Trending score 1.76
-
16
NVIDIA-NeMo/Gym
Evaluate and improve models and agents using environments
GitHub · Python · 1,346 stars · 485 forks · +3 stars in 24 hours · Trending score 1.72
-
17
MCPJam/inspector
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
GitHub · TypeScript · 2,331 stars · 305 forks · +3 stars in 24 hours · Trending score 1.57
-
18
langwatch/langwatch
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
GitHub · TypeScript · 5,047 stars · 434 forks · +3 stars in 24 hours · Trending score 1.55
-
19
strands-agents/evals
A comprehensive evaluation framework for AI agents and LLM applications.
GitHub · Python · 261 stars · 87 forks · +3 stars in 24 hours · Trending score 1.47
-
20
intel/dokimi
Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.
GitHub · Python · 39 stars · 13 forks · +1 stars in 24 hours · Trending score 1.30
-
21
beipo-ai/NorthCUA
Turn intent into work. Open-source Desktop/Web agent harness with long-horizon memory, live evaluation and recoverable batches.
GitHub · Python · 128 stars · 7 forks · +4 stars in 24 hours · Trending score 1.28
-
22
Helicone/helicone
🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
GitHub · TypeScript · 6,278 stars · 704 forks · 0 stars in 24 hours · Trending score 1.25
-
23
Kiln-AI/Kiln
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
GitHub · Python · 5,286 stars · 406 forks · +1 stars in 24 hours · Trending score 1.23
-
24
awslabs/uaef
Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.
GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22