Updated today
by Calibre-Labs
reforge-ai-evals
Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.
74 GitHub stars (74)
- 8 skills (8 skills found in its SKILL.md files)
- 8 forks (8 forks)
- Today last update (Last updated today)
Install
Skills CLI
npx skills add Calibre-Labs/reforge-ai-evals
Asks which skills and agents to install. Just one: npx skills add Calibre-Labs/reforge-ai-evals --skill eval-code — add -g for every project.
Claude Code
git clone https://github.com/Calibre-Labs/reforge-ai-evalscp -r "reforge-ai-evals/skills/eval-code" ~/.claude/skills/
Copies the eval-code skill into ~/.claude/skills/ for every project (use .claude/skills/ for one project). Repeat for the other skills listed below.
Plugin
/plugin marketplace add Calibre-Labs/reforge-ai-evals
Run inside Claude Code if the repository ships a plugin marketplace, then pick plugins from /plugin.
Skills can run code on your machine — read the SKILL.md first.
What it does
In the author's own words, from each skill's SKILL.md.
-
eval-code
Write deterministic, code-based evaluator functions for an AI product. Use this skill whenever you need to write evaluators that check structural properties of AI outputs (format, count, presence, schema compliance) without calling another LLM. Also use when auditing existing code-based evaluators for brittleness, coverage gaps, or false positives.
skills/eval-code/SKILL.md on GitHub (opens in a new tab) -
eval-guide
Entry point for eval work. Use when someone asks for help with evals in general, does not know where to start, asks "what should I do next", or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches the request. Load that skill directly.
skills/eval-guide/SKILL.md on GitHub (opens in a new tab) -
eval-llm-judge
Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner fu…
skills/eval-llm-judge/SKILL.md on GitHub (opens in a new tab) -
eval-rubric
Write or update the eval rubric for an AI agent or AI feature: the pass/fail criteria that define good behavior, in four groups (outcome, trajectory, governance, experience), plus the release thresholds that belong in the PRD. Use this skill whenever someone asks to "write an eval rubric", "define what good looks like", "add evals to my PRD", "what should we evaluate", "turn this PRD into a rubri…
skills/eval-rubric/SKILL.md on GitHub (opens in a new tab) -
llm-align
Analyze alignment between LLM judge scores and human labels in an eval dataset. Use this skill whenever someone wants to evaluate how well an LLM judge agrees with human reviewers, calculate TPR/TNR, investigate disagreements, or improve a scorer prompt. Trigger on phrases like: "calculate TPR/TNR", "judge alignment", "where does the LLM disagree with humans", "improve my scorer", "why is the LLM…
skills/llm-align/SKILL.md on GitHub (opens in a new tab) -
ticket-to-eval
Strip PII from a customer support ticket or eval trace and convert it into eval dataset rows. Produces two outputs: a regression dataset row (close to the original input, tagged with failure_mode) and a generalized dataset row (abstracted for broader coverage). Works with local CSV files or any eval platform. Use this skill whenever someone shares a support ticket, user complaint, or raw trace an…
skills/ticket-to-eval/SKILL.md on GitHub (opens in a new tab) -
trace-codes
Turn free-form notes from reading agent traces into trace codes: named, yes/no failure or quality patterns with definitions, examples and counts. Use this skill whenever someone has annotated traces or review notes and asks to "cluster my notes", "create trace codes", "find the failure modes", "do error analysis on these annotations", "what patterns are in these traces", or "what should we fix fi…
skills/trace-codes/SKILL.md on GitHub (opens in a new tab) -
uig
Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query matrix" or "SQM", improve dataset coverage, or assess whether their offline evals reflect what real users will actually s…
skills/uig/SKILL.md on GitHub (opens in a new tab)
No skill matches that filter.
Review before you install
A skill is a set of instructions and scripts your agent will follow, and it can run code on your machine. Read the SKILL.md and any scripts first — this listing is not a security review.
Similar skills
-
reaby morluto Reverse engineer anything with agents, from app behavior down to native binaries.
-
ponytailby DietrichGebert Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote.
-
agent-skillsby addyosmani Production-grade engineering skills for AI coding agents.
-
taste-skillby Leonxlnx Agent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
-
archifyby tt-a1i Turn any idea, plan, or codebase into a beautiful interactive diagram. An agent skill for Claude Code, Codex, and more.
-
awesome-claude-skillsby ComposioHQ A curated list of awesome Claude Skills, resources, and tools for customizing Claude AI workflows