evals-skills

In the author's own words, from each skill's SKILL.md.

• build-review-interface

  Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.

  skills/build-review-interface/SKILL.md on GitHub (opens in a new tab)

• error-discovery

  Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

  skills/error-discovery/SKILL.md on GitHub (opens in a new tab)

• eval-audit

  Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

  skills/eval-audit/SKILL.md on GitHub (opens in a new tab)

• evals-start

  Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.

  skills/evals-start/SKILL.md on GitHub (opens in a new tab)

• evaluate-rag

  Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

  skills/evaluate-rag/SKILL.md on GitHub (opens in a new tab)

• generate-synthetic-data

  Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.

  skills/generate-synthetic-data/SKILL.md on GitHub (opens in a new tab)

• validate-evaluator

  Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).

  skills/validate-evaluator/SKILL.md on GitHub (opens in a new tab)

• write-code-eval

  Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.

  skills/write-code-eval/SKILL.md on GitHub (opens in a new tab)

• write-judge-prompt

  Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.

  skills/write-judge-prompt/SKILL.md on GitHub (opens in a new tab)

No skill matches that filter.

Similar skills

• superpowersby obra An agentic skills framework & software development methodology that works. 300k stars · 15 skills
• skillsby mattpocock Skills for Real Engineers. Straight from my .agents directory. 293k stars · 30 skills
• ECCby affaan-m The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Cod… 280k stars · 30 skills
• reaby morluto Reverse engineer anything with agents, from app behavior down to native binaries. 270k stars · 1 skill
• andrej-karpathy-skillsby multica-ai A single CLAUDE.md file to improve Claude Code behavior, derived from Andrej Karpathy's observations on LLM coding pitfalls. 220k stars · 1 skill
• skillsby anthropics Public repository for Agent Skills 181k stars · 19 skills