evaluation-metrics repositories
Trending repositories tagged evaluation-metrics, ranked by trending score. Updated daily.
-
1
confident-ai/deepeval
The LLM Evaluation Framework
GitHub · Python · 19,534 stars · 2,219 forks · +84 stars in 24 hours · Trending score 3.50
-
2
awslabs/uaef
Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.
GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22
-
3
cxcscmu/IdeaTaste
IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.
GitHub · Python · 33 stars · 3 forks · +1 stars in 24 hours · Trending score 0.45
-
4
opensiro/vsm-harness-index
An evidence-based index of AI agent harnesses mapped by control function and autonomy.
GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.23
-
5
k4black/codebleu
Pip compatible CodeBLEU metric implementation available for linux/macos/win
GitHub · Python · 141 stars · 27 forks · 0 stars in 24 hours · Trending score 0.06
-
6
dokimos-dev/dokimos
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
GitHub · Java · 61 stars · 5 forks · 0 stars in 24 hours · Trending score 0.05
-
7
IBM/unitxt
🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking
GitHub · Python · 215 stars · 70 forks · 0 stars in 24 hours · Trending score 0.01
-
8
proycon/pynlpl
PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).
GitHub · Python · 474 stars · 66 forks · 0 stars in 24 hours · Trending score 0.00
-
9
gil-son/language-ai-engineering-lab
Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.
GitHub · Jupyter Notebook · 129 stars · 27 forks · 0 stars in 24 hours · Trending score 0.00
-
10
vectara/open-rag-eval
RAG evaluation without the need for "golden answers"
GitHub · Python · 420 stars · 30 forks · 0 stars in 24 hours · Trending score 0.00
-
11
huggingface/lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
GitHub · Python · 2,582 stars · 611 forks · 0 stars in 24 hours · Trending score 0.00
-
12
MiXaiLL76/faster_coco_eval
Continuation of an abandoned project fast-coco-eval
GitHub · Python · 148 stars · 15 forks · 0 stars in 24 hours · Trending score 0.00
-
13
hparreao/Awesome-AI-Evaluation-Guide
A comprehensive, implementation-focused guide to evaluating Large Language Models, RAG systems, and Agentic AI in production environments.
GitHub · Python · 21 stars · 8 forks · 0 stars in 24 hours · Trending score 0.00
-
14
frontier-harness-eval/eval
Public results and task definitions for FrontierHarness Eval
GitHub · JavaScript · 356 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00
-
15
TrustifAI/typed_evals
Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.
GitHub · Python · 16 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00
-
16
Vvkmnn/awesome-ai-eval
Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.
GitHub · 127 stars · 45 forks · 0 stars in 24 hours · Trending score 0.00