Here are 70 public repositories matching this topic...

coze-dev / coze-loop

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

agent open-source playground ai monitoring evaluation openai observability agentops coze langchain llmops prompt-management llm-observability agent-evaluation eino agent-observability

Updated Mar 17, 2026
Go

Giskard-AI / giskard-oss

Sponsor

🐢 Open-Source Evaluation & Testing library for LLM Agents

ai-security mlops fairness-ai responsible-ai ml-validation red-team-tools trustworthy-ai ml-testing llm ai-red-team ai-testing llmops llm-security llm-eval llm-evaluation rag-evaluation agent-evaluation

Updated Mar 17, 2026
Python

truera / trulens

Evaluation and Tracking for LLM Experiments and AI Agents

machine-learning neural-networks ai-agents explainable-ml agentops ai-monitoring ai-observability llms llmops llm-eval evals llm-evaluation agent-evaluation

Updated Mar 10, 2026
Python

mozilla-ai / any-agent

A single interface to use and evaluate different agent frameworks

ai mcp agents a2a agent-evaluation

Updated Mar 3, 2026
Python

rungalileo / agent-leaderboard

Ranking LLMs on agentic tasks

ai evaluation ai-agents synthetic-data ai-evaluation llms ai-benchmark agent-evaluation

Updated Nov 18, 2025
Jupyter Notebook

hidai25 / eval-view

Regression testing for AI agents. Snapshot behavior, diff tool calls, catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

python testing cli mcp evaluation pytest regression-testing ai-agents autogen llm anthropic langchain-agent openai-assistants crewai langgraph agentic-ai agent-evaluation agent-benchmark

Updated Mar 16, 2026
Python

Cre4T3Tiv3 / ai-agents-reality-check

Sponsor

Mathematical benchmark exposing the massive performance gap between real agents and LLM wrappers. Rigorous multi-dimensional evaluation with statistical validation (95% CI, Cohen's h) and reproducible methodology. Separates architectural theater from real systems through stress testing, network resilience, and failure analysis.

python open-source benchmarking reproducible-research statistical-analysis performance-testing network-resilience llm-agent llm-tools agent-architecture agentic-workflow agentic-ai agent-performance agent-evaluation ai-benchmarking agent-benchmark reality-check-ai-agent architectural-evaluation ensemble-coordination

Updated Aug 8, 2025
Python

microsoft / ignite25-PREL13-observe-manage-and-scale-agentic-ai-apps-with-microsoft-foundry

Learn How To Observe, Manage, and Scale, Agentic AI Apps Using Azure AI Foundry - with this hands-on workshop

observability quality-evaluation aiops distillation-model azure-openai azure-ai-search safety-evaluation azure-ai-foundry supervised-fine-tuning agent-evaluation azure-ai-foundry-models

Updated Feb 5, 2026
Jupyter Notebook

SparkBeyond / agentune

Tune your AI Agent to best meet its KPI with a cyclic process of analyze, improve and simulate

customer-support customer-service conversational-agents ai-agents chatbot-evaluation agent-simulator kpi-analysis agent-evaluation agent-optimization sales-agents customer-facing-agents kpi-optimization

Updated Jan 14, 2026
Python

solana8800 / langeval

Sponsor

Evaluation Infrastructure for AI Agents

ai-evaluation agent-evaluation ai-evals

Updated Feb 25, 2026
TypeScript

chaosync-org / awesome-ai-agent-testing

🤖 A curated list of resources for testing AI agents - frameworks, methodologies, benchmarks, tools, and best practices for ensuring reliable, safe, and effective autonomous AI systems

testing qa benchmark machine-learning evaluation chaos artificial-intelligence chaos-monkey testing-tools awesome-list quality-assurance ai-safety ai-agents chaos-engineering llm llm-evaluation agentic-ai ai-benchmark agent-evaluation

Updated May 28, 2025

Arc-Computer / CL-Bench

Benchmark framework for evaluating LLM agent continual learning in stateful environments. Features production-realistic CRM workflows with multi-turn conversations, state mutations, and cross-entity relationships. Extensible to additional domains

benchmark continual-learning agent-evaluation

Updated Nov 14, 2025
Python

dokimos-dev / dokimos

Evaluation Framework for LLM applications in Java and Kotlin

Updated Mar 8, 2026
Java

alepot55 / agentrial

Statistical evaluation framework for AI agents

python testing ci-cd pytest confidence-intervals quality-assurance non-deterministic ai-agents mlops statistical-testing llm ai-testing llm-evaluation agent-evaluation

Updated Feb 6, 2026
Python

shiragannavar / Testing-RAG

evaluation ground-truth llm generative-ai agent-evaluation

Updated May 12, 2025
Python

eljaplacido / projectcarfcynepic

bayesian-inference complexity-analysis symbolic-ai transparency-enhancing-technologies complex-adaptive-systems cynefin causalinference neurosymbolic-ai agentic-rag agentic-ai data-driven-decision-making self-healing-systems agent-evaluation euaiact causalai

Updated Mar 14, 2026
Python

najeed / ai-agent-eval-harness

The open-source MultiAgentOps evaluation harness for any industry scenario.

Updated Mar 16, 2026
HTML

lml2468 / ContextOptimizer

Intelligent Context Engineering Assistant for Multi-Agent Systems. Analyze, optimize, and enhance your AI agent configurations with AI-powered insights

multi-agent-systems prompt-engineering agent-evaluation context-engineering agent-optimizer

Updated Jul 5, 2025
Python

diorwave / agent-playground

A minimal sandbox to run, score, and compare AI agent outputs locally.

python experimental minimal deterministic ai-agents local-tools research-tools agent-evaluation agent-comparsion agent-playground

Updated Dec 19, 2025
Python

JetBrains / teamcity-ai-agent-testing-demo

End-to-end TeamCity framework to run AI agents on SWE-Bench Lite. Spin up isolated Docker images per task, extract patches, score with the official harness, and aggregate success rates. As an example, we'll look at Junie and Google Gemini CLI

ai evaluation eval evaluation-framework agentic-ai agent-evaluation evaluation-tools

Updated Aug 13, 2025
Kotlin