skill tester

Engineering Community

Skill notes

1a. Read the target skill

User provides skill name or path Read the target skill's SKILL.md + ALL referenced files completely Map out: Skill type: procedural / informational Input: what does the skill expect? (user message, task file, structured data)

1b. Design test prompts

Design test prompts applying criteria from test-design-guide.md (realistic prompts, assertion design, persona setup): Propose 2-3 test prompts: 1 happy-path: the most common, standard use case 1-2 edge cases: where the skill might break or behave unexpectedly

1c. Design trigger eval queries

Design trigger eval queries applying patterns from trigger-eval-guide.md: Generate 15-20 trigger eval queries: 8-10 should-trigger: varied phrasings of the skill's intended use 8-10 should-not-trigger: near-misses that share keywords but need something different

1d. Confirm with user

Present the full test plan: Test prompts with assertions Trigger eval queries Proposed model for runners Persona (use default, modify only if user requests)

2a. Setup

Create workspace: ~/.claude/skill-tests/{skill-name}/iteration-{N}/ (N = 1 for first run, increment for re-runs) Save test plan to workspace: evals.json with all prompts and assertions trigger-evals.json with all trigger queries

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Engineering
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@pavel-molyanov · no license declared
Token usage
Moderate
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview skill-tester-pavel-molyanov.preview
# 3b. Compile results per scenario

1. Build results table (assertions × runners)
2. Cross-runner consistency: where did skill-runners diverge?
   - Divergence on a criterion = ambiguous instruction in the skill
3. Baseline comparison:
   - Passed by skill-runners ONLY → skill adds value
   - Passed by ALL → criterion too easy or skill doesn't help here

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.