skill eval

Engineering Community

Skill notes

Reference Loading Table

Signal · Load These Files · Why tasks related to this reference · schemas.md · Loads detailed guidance from schemas.md. tasks related to this reference · self-improve-loop.md · Loads detailed guidance from self-improve-loop.md. "bake-off", "head-to-head", "compare implementations", "grade two versions", "which Feynman skill is better" · bake-off-methodology.md · Loads the bake-off rubric, anti-rationalization gate, fold-filter, and worked Feynman example.

Phase 1: ASSESS — Determine what to evaluate

Step 1: Identify the skill This checks: SKILL.md exists, valid frontmatter, required fields (name, description), kebab-case naming, description under 1024 chars, no angle brackets. Step 2: Choose evaluation mode based on user intent Intent · Mode · Script "Test if description triggers correctly" · Trigger eval · run_eval.py

Phase 2: EVALUATE — Run the appropriate evaluation

Mode A: Trigger Evaluation Test whether a skill's description causes Claude to invoke it for the right queries. Step 1: Create eval set (or use existing) Create a JSON file with 8-20 test queries. Eval set quality matters — use realistic prompts with detail (file paths, context, casual phrasing), not abstract one-liners. Focus on edge cases where the skill competes with adjacent skills. Example of good eval queries:

Phase 3: IMPROVE — Apply results

Step 1: Review results For trigger eval / description optimization: Show the best description vs original Show per-query results (which queries improved, which regressed) Show train vs test scores

Error: "No SKILL.md found"

Cause: Skill path doesn't point to a valid skill directory Solution: Verify path contains a SKILL.md file. Skills must follow the skill-name/SKILL.md structure.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Engineering
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@notque · no license declared
Token usage
Heavy
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
  • Env read
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview skill-eval.preview
# Phase 3: IMPROVE — Apply results

- Show the best description vs original
- Show per-query results (which queries improved, which regressed)
- Show train vs test scores
- Show pass rate delta (with-skill vs without-skill)
- Show timing and token cost delta
- Highlight assertions that only pass with the skill (value-add)

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.