Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
The LLM Evaluation Framework
A framework for few-shot evaluation of language models.
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
☁️ 🚀 📊 📈 Evaluating state of the art in AI
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
Repo for AI Agents The Definitive Guide
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
Data-Driven Evaluation for LLM-Powered Applications
PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection
Moonshot - A simple and modular tool to evaluate and red-team any LLM application.
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
Python SDK for running evaluations on LLM generated responses
The official evaluation suite and dynamic data release for MixEval.
MedEvalKit: A Unified Medical Evaluation Framework
build and benchmark deep research
A research library for automating experiments on Deep Graph Networks
AI Data Management & Evaluation Platform
WritingBench: A Comprehensive Benchmark for Generative Writing