Testing Your Skills

Why Fresh Sessions Matter

Authoring-context bleed is the most common reason a skill appears to work during development and fails in real use. When you write a skill, you know exactly what it says. A fresh session does not. If you test by invoking the skill in the same session you used to write it, your prompts are already shaped by the skill's content — you naturally phrase requests in ways the skill expects.

Always test in a fresh session: no prior conversation, no accumulated context from authoring. The agent in a fresh session is the user you are writing the skill for.

The Baseline A/B Test

The simplest valid test for any skill on either platform:

• Collect prompts. Write 5–10 realistic prompts that should trigger the skill — use phrasing real users would type, not prompts that mirror the skill's description or procedure text.
• Test with skill enabled. Run each prompt in a fresh session with the skill active. Record the outputs.
• Test with skill disabled. Run each prompt in a fresh session with the skill removed from the active list. Record the outputs.
• Compare. Did the skill produce better output? Did it fire at all? Did it fire on prompts where it should not have?
• Tune and repeat. Adjust the description, when_to_use, or skill body based on what you observed, then run the comparison again.

This procedure takes 30–60 minutes for a well-defined skill and catches the two most common failures: non-invocation (the skill exists but never fires) and false-positive invocation (the skill fires on unrelated requests).

skill-creator Plugin (Claude Code)

The skill-creator plugin automates the eval workflow with structured assertions, isolated session runs, and side-by-side output comparison. Install it:

/plugin install skill-creator@claude-plugins-official

Write test cases in evals/evals.json. Each case pairs a prompt with graded assertions about the output:

{
  "tests": [
    {
      "prompt": "I need to submit a PR for review",
      "assertions": [
        {"type": "skill-invoked", "skill": "review-pr"},
        {"type": "output-contains", "value": "checklist"}
      ]
    }
  ]
}

skill-creator runs each test case in an isolated session with clean context — no history bleed between cases. It generates three output files:

• grading.json — pass/fail per assertion across every test case
• benchmark.json — side-by-side output comparison with and without the skill enabled
• review.html — a human-readable viewer for all outputs, suitable for manual review

Description hit-rate testing: skill-creator can vary the description field and measure how often the agent auto-invokes the skill across a prompt set. Use this when tuning when_to_use — iterate the wording and measure invocation rate rather than guessing which phrase performs better.

A/B version comparison: Before publishing an update, run skill-creator's A/B mode to compare old and new SKILL.md side-by-side across the same prompt set. This catches regressions before they affect users.

Testing with Hermes

Hermes provides a command-line testing pattern without a plugin:

hermes chat --toolsets skills -q "Submit a PR for review using the team workflow"

The --toolsets skills flag loads only the skills toolset, isolating skill discovery from the full Hermes context. The -q flag runs a single-turn query without an interactive session. Together, they verify that the agent retrieves the correct skill at the correct disclosure level.

Verifying progressive disclosure with skill_view: Hermes loads skills in three levels. Verify each level is wired correctly:

• Level 0 — names and short descriptions only (the catalog pass). Verify the description is concise and accurately matches the use case.
• Level 1 — full SKILL.md content, activated when a task matches the description. Verify the procedure is complete and correct.
• Level 2 — reference sections pulled via skill_view paths only when explicitly needed. Verify reference files are not loading on every invocation.

Testing conditional activation: Run the same prompt with and without the required toolset to confirm requires_toolsets gates the skill correctly. The skill should not be visible when its required toolset is absent.

What a Passing Skill Looks Like

A skill passes when all three conditions hold:

• Fires on relevant prompts — auto-invocation or correct user invocation on every prompt in your trigger set.
• Does not fire on irrelevant prompts — a skill that activates on unrelated requests wastes context and degrades session quality.
• Measurably improves output — the agent's response with the skill is concretely better (more accurate, more structured, fewer errors) than without it.

All three conditions are independent requirements. A skill that fires reliably but produces no quality improvement is not passing. A skill that improves output when it fires but activates on irrelevant prompts half the time is not passing either.