Measuring Effectiveness and Tuning Auto-Invocation

/skill-doctor: Claude Code Diagnostics

The /skill-doctor command (Claude Code v2.1.252 or later) produces a per-skill diagnostic report in the terminal. It shows:

• Context cost per skill — the token count each installed skill loads into the session
• Invocation frequency — how often each skill has been invoked in recent sessions
• Unused skills — skills that have never been invoked, flagged for review or removal
• Disable recommendations — where and how to turn off expensive unused skills in your settings
• Unused plugins — plugins installed but not recently used

In an interactive session, /skill-doctor opens the plugin manager's Stats tab. In non-interactive mode (the -p flag), it prints as plain text. It is not available over Remote Control from a phone or browser — the command returns an error on those connections. Run it in the terminal on the machine where the session is running.

Run /skill-doctor monthly as a maintenance pass. Skills showing zero invocations over several weeks are candidates for retirement before they continue accumulating Level 0 scan cost.

Hermes Curator: Lifecycle Automation

The Hermes Curator tracks last-used timestamps per skill and runs on a configurable interval (default: 7 days). It moves skills through a three-stage lifecycle automatically:

• Active — used within the Curator's interval. Loads at Level 0 on every session start.
• Stale — not used within the interval. Still loads, but flagged in Curator reports for review.
• Archived — unused for an extended period. Stops loading entirely, eliminating its Level 0 token cost.

Review Curator reports before skills auto-archive. A skill entering the stale state often signals that its description no longer matches current usage patterns — a tuning opportunity before the skill disappears. A skill that auto-archives unreviewed may be one you still need but whose description drifted.

Auto-Compaction Token Budgets

When Claude Code's context fills and auto-compaction runs, invoked skills re-attach to the compacted session within two token budgets:

• Per-skill limit: 5,000 tokens. Only the most recent invocation of each skill re-attaches, and only up to 5,000 tokens of that invocation.
• Combined budget: 25,000 tokens. All re-attached skills share this pool. Most-recently-invoked skills fill the budget first; older skills may be dropped entirely if the pool fills.

The symptom of budget exhaustion is subtle: a skill that was guiding behavior earlier in the session stops influencing responses mid-session, with no error or warning. It was dropped during compaction. Two fixes:

• Re-invoke the skill — moves it to the top of the recency order for the next compaction cycle.
• Keep SKILL.md under 5,000 tokens — ensures the full body survives re-attachment even when the combined budget is tight.

Tuning when_to_use for Invocation Precision

The when_to_use field supplements description with additional context for Claude's auto-invocation decision. Both fields are truncated together at 1,536 combined characters in the skill listing. Put the primary use case in description and reserve when_to_use for discriminating trigger phrases and negative examples:

---
name: review-pr
description: Submit a structured PR review following team conventions
when_to_use: Use when the user asks to submit, merge, or review a pull request.
  Not for general git questions or branch management.
---

Negative examples — what the skill should not trigger on — are often the highest-value content in when_to_use. They prevent the skill from firing on adjacent requests that share vocabulary with the trigger case but belong to a different workflow.

Tuning paths for File-Scoped Invocation

The paths field limits auto-invocation to sessions where files matching the specified globs are in scope. A noisy skill that loads on every session becomes a precise one that loads only when relevant:

paths: "src/**/*.tsx,src/**/*.jsx"   # React components only
paths: "Dockerfile*,docker-compose*" # Docker config only
paths: "*.test.ts,*.spec.ts"         # Test files only

paths does not restrict manual invocation — /skill-name still works from anywhere in the session. It filters auto-invocation from the Level 0 catalog pass only. A Docker skill with paths: "Dockerfile*" stops loading in sessions that never touch Docker files, eliminating its scan cost in unrelated workflows.

Description Engineering: The Routing Label Principle

On both platforms, the skill description is the primary routing signal. Treat it as a routing label rather than a tagline — concrete and action-oriented:

• Weak: "Helps with git work" — matches almost anything involving git, causing false-positive invocations across unrelated git workflows.
• Strong: "Submit a pull request following team conventions" — specific action, specific context, clear boundary.

Run description A/B tests using skill-creator's hit-rate mode (Claude Code) or by varying the description and re-testing with hermes chat -q (Hermes). Measure invocation rate changes rather than guessing which phrasing performs better.

When to Retire vs. Tune

Not every underperforming skill is worth tuning. Apply this decision framework:

• Low invocation AND low output delta after tuning — retire the skill. It does not earn its Level 0 token cost.
• Low invocation but high output delta when it fires — tune when_to_use and description to improve discoverability.
• High invocation but low output delta — the skill fires but does not help; rewrite the procedure or retire it.