evaluation-framework repositories
Trending repositories tagged evaluation-framework, ranked by trending score. Updated daily.
-
1
promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
GitHub · TypeScript · 27,483 stars · 2,800 forks · +65 stars in 24 hours · Trending score 3.79
-
2
confident-ai/deepeval
The LLM Evaluation Framework
GitHub · Python · 19,534 stars · 2,219 forks · +84 stars in 24 hours · Trending score 3.50
-
3
future-agi/future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
GitHub · Python · 2,288 stars · 725 forks · +2 stars in 24 hours · Trending score 1.55
-
4
microsoft/FrontierWeekHack
Hands-on Microsoft Foundry hackathon labs to build, monitor, evaluate, and deploy agentic AI workflows.
GitHub · Python · 64 stars · 681 forks · 0 stars in 24 hours · Trending score 1.54
-
5
Kiln-AI/Kiln
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
GitHub · Python · 5,286 stars · 406 forks · +1 stars in 24 hours · Trending score 1.23
-
6
awslabs/uaef
Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.
GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22
-
7
ktwu01/benchmark-radar
Track 50,331+ AI benchmark, eval, dataset, and data-quality records from 41 public sources, with linked evidence and daily updates.
GitHub · Python · 409 stars · 65 forks · 0 stars in 24 hours · Trending score 1.16
-
8
open-compass/AgentCompass
[EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.
GitHub · Python · 197 stars · 47 forks · +1 stars in 24 hours · Trending score 1.16
-
9
cxcscmu/IdeaTaste
IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.
GitHub · Python · 33 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45
-
10
HPAI-BSC/TuRTLe
TuRTLe: A Unified Evaluation of LLMs for RTL Generation 🐢 (MLCAD 2025, ACM TODAES 2026)
GitHub · Python · 53 stars · 12 forks · 0 stars in 24 hours · Trending score 0.42
-
11
jfrog/agent-belt
Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox.
GitHub · Python · 19 stars · 5 forks · 0 stars in 24 hours · Trending score 0.39
-
12
openbenchmarks-labs/company-news
Open, independent benchmark on company news APIs for AI agents - web search APIs vs dedicated news indexes on 300 company news questions, ranked by cost per 1,000 correct answers. Exa, Parallel, Perplexity, Brave, Firecrawl, Linkup, You, TinyFish, Tavily, Google SERP, Seltz, PredictLeads, Autobound, Datahyena.
GitHub · Python · 5 stars · 11 forks · 0 stars in 24 hours · Trending score 0.27
-
13
AcruxCore/AcruxCore
Open-source LLMOps platform with prompt management, an OpenAI-compatible AI gateway, OpenTelemetry tracing, a versioned tool catalog, and LLM evaluation. Use hosted or self-host with TypeScript and Python SDKs.
GitHub · TypeScript · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.19
-
14
EuroEval/EuroEval
The robust European language model benchmark.
GitHub · Python · 205 stars · 69 forks · 0 stars in 24 hours · Trending score 0.18
-
15
alexgreensh/eval-genius
Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.
GitHub · Python · 27 stars · 4 forks · 0 stars in 24 hours · Trending score 0.14
-
16
openbenchmarks-labs/document-processing
Open, independent benchmark on document parsing APIs and frontier VLMs for AI agents and RAG: does a struck-out clause in a redlined contract PDF survive the parse? Open source code + open data. LlamaParse, Reducto, Datalab, Extend, Pulse, Mistral OCR, GPT-6 Astra, Claude Fable 5.1.
GitHub · Python · 7 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10
-
17
lizhiyao/oh-my-knowledge
OMK helps people and agents build, maintain, and reuse valuable knowledge from real work and collaboration, with readable content, traceable sources, and revision history for future tasks.
GitHub · TypeScript · 25 stars · 5 forks · 0 stars in 24 hours · Trending score 0.09
-
18
MSKazemi/aobench
Role-aware, permission-enforced benchmark for LLM agents that operate HPC systems — SLURM, telemetry, RBAC, docs, facility. A policy violation hard-fails the task. 88 tasks (10 categories x 5 roles), 29 reproducible environments with 6 rebuilt from real Marconi100 ExaData, scored on 7 weighted dimensions.
GitHub · Python · 9 stars · 14 forks · −2 stars in 24 hours · Trending score 0.08
-
19
techtheist/knowledgedrift
KnowledgeDrift - Second Generation AI Memory benchmark
GitHub · Rust · 17 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07
-
20
dokimos-dev/dokimos
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
GitHub · Java · 61 stars · 5 forks · 0 stars in 24 hours · Trending score 0.05
-
21
letta-ai/letta-evals
Evaluation kit for testing stateful agents
GitHub · Python · 86 stars · 15 forks · 0 stars in 24 hours · Trending score 0.01
-
22
MrBinnacle/skill-harness
How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”
GitHub · Python · 16 stars · 0 forks · 0 stars in 24 hours · Trending score 0.01
-
23
pyrddlgym-project/pyRDDLGym
A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.
GitHub · Python · 94 stars · 24 forks · 0 stars in 24 hours · Trending score 0.01
-
24
danielle34/cavewoman
Caveman prompting, measured. A two-channel evaluation protocol scoring what input and output compression actually cost LLMs in dollars, accuracy, and surface-text fidelity across seven models and five benchmarks.
GitHub · Python · 16 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00