Skill Evals

Engineering Community

Skill notes

1. Load inputs

skills/skill-evals/evals.json — assertion manifest (read with jq; if parse fails, retry once after 5s, then exit SKILL_EVALS_ERROR). aeon.yml — registered skills, enabled flags, cron schedules. memory/cron-state.json — total_runs, success_rate, last_quality_score, last_failed, last_success per skill. memory/issues/INDEX.md — currently open issues (used to dedupe issue filing in step 5b).

2. Run coverage audit (delegated)

Call ./scripts/eval-audit --json and parse: summary.coverage_pct, summary.covered, summary.uncovered_enabled, summary.uncovered_disabled uncovered_enabled[].skill and .inferred_pattern — these are the spec-gap candidates surfaced in the Action Queue. Do not re-implement coverage detection in prose. If the script fails, fall back to the in-memory check (compare evals.json keys to aeon.yml enabled skills) and mark eval-audit=fail in the source-status footer.

3. Determine eval scope

If ${var} is set → evaluate only that one skill (skip if not in evals.json and notify "skill-evals: ${var} has no spec — add an entry to evals.json"). Otherwise evaluate every skill in evals.json.

4. For each skill in scope, run checks

a. Find latest output: glob output_pattern, sort descending by filename. If empty → status NO_OUTPUT, root_cause no_file_match, skip remaining checks. b. Empty/stale: if file size is 0 bytes → FAIL, root_cause empty_file. If file mtime is older than 2× expected_cadence (derived from the skill's cron in aeon.yml; fall back to 14 days if cron is workflow_dispatch or unparseable) → STALE, root_cause stale_file. Stale outputs still run their assertions but are reported as STALE so dashboard noise is correct.

5. Diff vs prior eval (the lede)

Parse the prior eval article's results table. For each skill produce one of: NEW_FAIL — was PASS/STALE/WARN, now FAIL/QUALITY_DEGRADED/NO_OUTPUT FIXED — was failing, now PASS STILL_FAIL — was failing, still failing (carry the issue ID forward) NEW_PASS — wasn't in prior (newly added to evals.json)

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Engineering
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
94 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@aaronjmars · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
Linux
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
Network behavior
External requests
Install commands
None (reference only)
Output preview skill-evals.preview
# Full Results

| Skill | Status | Diff | Root cause | Quality | Words | Last output | |-------|--------|------|------------|---------|-------|-------------|

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.