evaluation-framework repositories

Trending repositories tagged evaluation-framework, ranked by trending score. Updated daily.

  1. 1
    promptfoo/promptfoo

    Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

    GitHub · TypeScript · 27,483 stars · 2,800 forks · +65 stars in 24 hours · Trending score 3.79

  2. 2
    confident-ai/deepeval

    The LLM Evaluation Framework

    GitHub · Python · 19,534 stars · 2,219 forks · +84 stars in 24 hours · Trending score 3.50

  3. 3
    future-agi/future-agi

    Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

    GitHub · Python · 2,288 stars · 725 forks · +2 stars in 24 hours · Trending score 1.55

  4. 4
    microsoft/FrontierWeekHack

    Hands-on Microsoft Foundry hackathon labs to build, monitor, evaluate, and deploy agentic AI workflows.

    GitHub · Python · 64 stars · 681 forks · 0 stars in 24 hours · Trending score 1.54

  5. 5
    Kiln-AI/Kiln

    Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

    GitHub · Python · 5,286 stars · 406 forks · +1 stars in 24 hours · Trending score 1.23

  6. 6
    awslabs/uaef

    Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.

    GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22

  7. 7
    ktwu01/benchmark-radar

    Track 50,331+ AI benchmark, eval, dataset, and data-quality records from 41 public sources, with linked evidence and daily updates.

    GitHub · Python · 409 stars · 65 forks · 0 stars in 24 hours · Trending score 1.16

  8. 8
    open-compass/AgentCompass

    [EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.

    GitHub · Python · 197 stars · 47 forks · +1 stars in 24 hours · Trending score 1.16

  9. 9
    cxcscmu/IdeaTaste

    IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.

    GitHub · Python · 33 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45

  10. 10
    HPAI-BSC/TuRTLe

    TuRTLe: A Unified Evaluation of LLMs for RTL Generation 🐢 (MLCAD 2025, ACM TODAES 2026)

    GitHub · Python · 53 stars · 12 forks · 0 stars in 24 hours · Trending score 0.42

  11. 11
    jfrog/agent-belt

    Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox.

    GitHub · Python · 19 stars · 5 forks · 0 stars in 24 hours · Trending score 0.39

  12. 12
    openbenchmarks-labs/company-news

    Open, independent benchmark on company news APIs for AI agents - web search APIs vs dedicated news indexes on 300 company news questions, ranked by cost per 1,000 correct answers. Exa, Parallel, Perplexity, Brave, Firecrawl, Linkup, You, TinyFish, Tavily, Google SERP, Seltz, PredictLeads, Autobound, Datahyena.

    GitHub · Python · 5 stars · 11 forks · 0 stars in 24 hours · Trending score 0.27

  13. 13
    AcruxCore/AcruxCore

    Open-source LLMOps platform with prompt management, an OpenAI-compatible AI gateway, OpenTelemetry tracing, a versioned tool catalog, and LLM evaluation. Use hosted or self-host with TypeScript and Python SDKs.

    GitHub · TypeScript · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.19

  14. 14
    EuroEval/EuroEval

    The robust European language model benchmark.

    GitHub · Python · 205 stars · 69 forks · 0 stars in 24 hours · Trending score 0.18

  15. 15
    alexgreensh/eval-genius

    Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.

    GitHub · Python · 27 stars · 4 forks · 0 stars in 24 hours · Trending score 0.14

  16. 16
    openbenchmarks-labs/document-processing

    Open, independent benchmark on document parsing APIs and frontier VLMs for AI agents and RAG: does a struck-out clause in a redlined contract PDF survive the parse? Open source code + open data. LlamaParse, Reducto, Datalab, Extend, Pulse, Mistral OCR, GPT-6 Astra, Claude Fable 5.1.

    GitHub · Python · 7 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10

  17. 17
    lizhiyao/oh-my-knowledge

    OMK helps people and agents build, maintain, and reuse valuable knowledge from real work and collaboration, with readable content, traceable sources, and revision history for future tasks.

    GitHub · TypeScript · 25 stars · 5 forks · 0 stars in 24 hours · Trending score 0.09

  18. 18
    MSKazemi/aobench

    Role-aware, permission-enforced benchmark for LLM agents that operate HPC systems — SLURM, telemetry, RBAC, docs, facility. A policy violation hard-fails the task. 88 tasks (10 categories x 5 roles), 29 reproducible environments with 6 rebuilt from real Marconi100 ExaData, scored on 7 weighted dimensions.

    GitHub · Python · 9 stars · 14 forks · −2 stars in 24 hours · Trending score 0.08

  19. 19
    techtheist/knowledgedrift

    KnowledgeDrift - Second Generation AI Memory benchmark

    GitHub · Rust · 17 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07

  20. 20
    dokimos-dev/dokimos

    LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.

    GitHub · Java · 61 stars · 5 forks · 0 stars in 24 hours · Trending score 0.05

  21. 21
    letta-ai/letta-evals

    Evaluation kit for testing stateful agents

    GitHub · Python · 86 stars · 15 forks · 0 stars in 24 hours · Trending score 0.01

  22. 22
    MrBinnacle/skill-harness

    How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”

    GitHub · Python · 16 stars · 0 forks · 0 stars in 24 hours · Trending score 0.01

  23. 23
    pyrddlgym-project/pyRDDLGym

    A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

    GitHub · Python · 94 stars · 24 forks · 0 stars in 24 hours · Trending score 0.01

  24. 24
    danielle34/cavewoman

    Caveman prompting, measured. A two-channel evaluation protocol scoring what input and output compression actually cost LLMs in dollars, accuracy, and surface-text fidelity across seven models and five benchmarks.

    GitHub · Python · 16 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00