Evaluating Skills with skill-creator

The skill that seems to work

You write a skill, invoke it a few times, it does roughly what you wanted. Done? Probably not. Seeing a skill trigger tells you Claude found it — it does not tell you whether the output matches what you intended. The description might be triggering too often, or too rarely, or the skill body might be producing outputs that look right at a glance but miss important cases.

Measuring a skill properly requires running it against a set of test cases with and without it, then comparing the results. Doing that by hand is tedious enough that most people do not bother. The skill-creator plugin automates the loop.

Installing skill-creator

Install the plugin from the official marketplace:

/plugin install skill-creator@claude-plugins-official

If Claude Code reports that the plugin is not found, refresh the marketplace first:

/plugin marketplace update claude-plugins-official

Then run /reload-plugins to make the plugin's skills available in the current session.

Running an evaluation

Ask Claude to evaluate any of your skills by name:

evaluate my summarize-changes skill with skill-creator

skill-creator walks you through writing test cases for the skill, then runs the evaluation loop. Each test case specifies a prompt, an optional set of input files, and the expected behavior — what the output should contain or avoid.

Test cases are stored in evals/evals.json inside the skill directory:

~/.claude/skills/summarize-changes/
├── SKILL.md
└── evals/
    └── evals.json

The format captures the prompt, any context to provide, and the assertions to check against the output. skill-creator spawns a subagent per test case so each run starts with a clean context — isolated from your conversation history and from each other.

Reading the output

After running, skill-creator writes three files inside evals/:

grading.json records pass or fail for each assertion in each test case, along with evidence — the relevant excerpt from the actual output that was evaluated against the assertion. Reading the failed assertions tells you exactly where the skill is falling short.

benchmark.json aggregates the pass rate, token count, and time for sessions with the skill versus without it. This is the core comparison: does the skill actually improve outcomes, and what does it cost in tokens and time? A skill that raises the pass rate by 20% but doubles the token cost may not be worth it for every task. One that raises the pass rate by 60% at minimal cost is clearly worth keeping.

skill-creator also opens an HTML review viewer where you can inspect each output side-by-side, record qualitative feedback (things a pass/fail assertion cannot capture), and flag cases where the skill produced technically correct output but missed the intent.

Description tuning

A common problem: a skill triggers on prompts it should not, or fails to trigger on prompts it should. The description field drives this behavior, and tuning it by intuition is guesswork.

skill-creator has a dedicated description-tuning mode. It generates a set of should-trigger prompts (requests that match the skill's intended use case) and should-not-trigger prompts (requests that look similar but do not), measures the hit rate against the current description, then proposes description edits that improve the rate.

This is useful when a skill auto-invokes too aggressively — appearing when you ask about unrelated topics — or not aggressively enough, requiring you to always invoke it manually even for requests that clearly match.

Version A/B comparison

Before committing an edit to a skill's body or description, skill-creator can run a blind A/B comparison between the current version and a proposed version. Both versions run against the same test cases, and the results are compared side-by-side in the benchmark output.

This turns skill iteration from "does this feel better?" into a measured decision. If version B raises the pass rate, reduces token cost, and the HTML report shows better qualitative outputs — commit it. If the metrics are similar but the instructions in version B are simpler to maintain, that is also a good reason to switch.

When evals are worth it

Running evals on every skill you create is overkill. The overhead pays off for:

• High-frequency skills — a commit or code-review skill that runs dozens of times per week benefits from a few hours of tuning to eliminate the most common failure modes
• Shared team skills — a skill that everyone on the team uses is worth measuring before it goes into the shared repo
• Skills with side effects — anything that triggers a deploy, sends a message, or creates a PR is worth verifying against test cases before it runs autonomously
• Skills with complex descriptions — if a skill is triggering in unexpected situations, description tuning will tell you why

For simple personal skills — a daily standup format, a quick diff summarizer — intuition-based iteration is fine. When a skill becomes part of the team's workflow, add a test case set and run it before committing changes.