Updated today
by relai-ai
agent-eval-loop
Build evals from traces, code, and coding sessions, then measure and improve AI agent behavior, including your coding agents
37 GitHub stars (37)
+37 in 6 days
| Period | New stars |
|---|---|
| Nov 14 | +16 |
| Nov 15 | +6 |
| Nov 16 | +4 |
| Nov 17 | 0 |
- 7 skills (7 skills found in its SKILL.md files)
- 2 forks (2 forks)
- Today last update (Last updated today)
Install
Skills CLI
npx skills add relai-ai/agent-eval-loop
Asks which skills and agents to install. Just one: npx skills add relai-ai/agent-eval-loop --skill coding-evals — add -g for every project.
Claude Code
git clone https://github.com/relai-ai/agent-eval-loopcp -r "agent-eval-loop/skills/coding-evals" ~/.claude/skills/
Copies the coding-evals skill into ~/.claude/skills/ for every project (use .claude/skills/ for one project). Repeat for the other skills listed below.
Plugin
/plugin marketplace add relai-ai/agent-eval-loop
Run inside Claude Code if the repository ships a plugin marketplace, then pick plugins from /plugin.
Skills can run code on your machine — read the SKILL.md first.
What it does
In the author's own words, from each skill's SKILL.md.
-
coding-evals
Use to build evals for the coding agent a team uses (Claude Code, Codex, OpenHands, or the team's own setup of one) from real coding sessions, pull requests and git history, and to run them to measure how that agent does on the team's code, check whether a change helps (a model, an agent version, CLAUDE.md, AGENTS.md, a skill), find which parts of a setup help, or compare agents ("turn this sessi…
skills/coding-evals/ on GitHub (opens in a new tab)SKILL.md -
discover
Use to learn what the AI agent (or LLM app, chatbot or pipeline) in this repository must do and where it can fail - requirements, risks and open questions - from its code, prompts and docs, and from traces, transcripts or logs of real runs when there are any ("what could go wrong with this agent", "why did this run fail", "ingest these traces", "build the wiki", "record this requirement for the a…
skills/discover/ on GitHub (opens in a new tab)SKILL.md -
eval
Use to build and run evals for the AI agent (or LLM app, chatbot or pipeline) in this repository - the harness that runs the real agent (first time), hard realistic tests from risks, requirements, failed runs or traces, harder or combined tests, benchmarks, and shared evaluators ("create evals for this agent", "turn this failure into a test", "set up the harness", "bring in this benchmark"); or t…
skills/eval/ on GitHub (opens in a new tab)SKILL.md -
improve
Use to fix the AI agent's code, improve the agent (or LLM app, chatbot or pipeline) against its evaluations, compare prompts, models or settings for it, reduce its cost or latency while preserving behavior, or check a change to it for regressions ("fix this crash in the agent", "make the agent pass X", "compare these two prompts for the agent", "does this change help the agent", "make it cheaper"…
skills/improve/ on GitHub (opens in a new tab)SKILL.md -
loop
Use to improve the AI agent in this repository continuously from traces, feedback and evaluations ("improve this agent from ./traces", "keep improving it as new traces come in", "run the improvement loop", "make it better and cheaper with as little of my time as possible"). Not for a one-off test run (use the eval skill) or a single fix (use the improve skill), and not for running a prompt on a s…
skills/loop/ on GitHub (opens in a new tab)SKILL.md -
start
Use when the user asks how to use RELAI, where to start, what it can do for their agent, or what the status of their evals is and what to do next ("how do I use RELAI", "where do I start", "what's the status of my evals"); also to set up RELAI records when a selected workflow needs them. Routes an already clear request directly to the appropriate skill, reusing existing project infrastructure.
skills/start/ on GitHub (opens in a new tab)SKILL.md -
trace-review
Build or adapt a project-owned UI app for reviewing agent traces, with full interactions, keyboard navigation, search, filters and persistent human feedback ("build a trace viewer", "let me label traces by hand", "a review app for these runs"). Use when the user wants a trace viewer or review app; use discover for trace analysis without app building.
skills/trace-review/ on GitHub (opens in a new tab)SKILL.md
No skill matches that filter.
Review before you install
A skill is a set of instructions and scripts your agent will follow, and it can run code on your machine. Read the SKILL.md and any scripts first — this listing is not a security review.