evaluation-metrics repositories

Trending repositories tagged evaluation-metrics, ranked by trending score. Updated daily.

  1. 1
    confident-ai/deepeval

    The LLM Evaluation Framework

    GitHub · Python · 19,534 stars · 2,219 forks · +84 stars in 24 hours · Trending score 3.50

  2. 2
    awslabs/uaef

    Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.

    GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22

  3. 3
    cxcscmu/IdeaTaste

    IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.

    GitHub · Python · 33 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45

  4. 4
    opensiro/vsm-harness-index

    An evidence-based index of AI agent harnesses mapped by control function and autonomy.

    GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.23

  5. 5
    k4black/codebleu

    Pip compatible CodeBLEU metric implementation available for linux/macos/win

    GitHub · Python · 141 stars · 27 forks · 0 stars in 24 hours · Trending score 0.06

  6. 6
    dokimos-dev/dokimos

    LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.

    GitHub · Java · 61 stars · 5 forks · 0 stars in 24 hours · Trending score 0.05

  7. 7
    IBM/unitxt

    🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

    GitHub · Python · 215 stars · 70 forks · 0 stars in 24 hours · Trending score 0.01

  8. 8
    proycon/pynlpl

    PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

    GitHub · Python · 474 stars · 66 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    gil-son/language-ai-engineering-lab

    Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.

    GitHub · Jupyter Notebook · 129 stars · 27 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    vectara/open-rag-eval

    RAG evaluation without the need for "golden answers"

    GitHub · Python · 420 stars · 30 forks · 0 stars in 24 hours · Trending score 0.00

  11. 11
    huggingface/lighteval

    Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

    GitHub · Python · 2,582 stars · 611 forks · 0 stars in 24 hours · Trending score 0.00

  12. 12
    MiXaiLL76/faster_coco_eval

    Continuation of an abandoned project fast-coco-eval

    GitHub · Python · 148 stars · 15 forks · 0 stars in 24 hours · Trending score 0.00

  13. 13
    hparreao/Awesome-AI-Evaluation-Guide

    A comprehensive, implementation-focused guide to evaluating Large Language Models, RAG systems, and Agentic AI in production environments.

    GitHub · Python · 21 stars · 8 forks · 0 stars in 24 hours · Trending score 0.00

  14. 14
    frontier-harness-eval/eval

    Public results and task definitions for FrontierHarness Eval

    GitHub · JavaScript · 356 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    TrustifAI/typed_evals

    Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

    GitHub · Python · 16 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    Vvkmnn/awesome-ai-eval

    Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

    GitHub · 127 stars · 45 forks · 0 stars in 24 hours · Trending score 0.00