skill evolution

Engineering Community

Skill notes

The 5 Scoring Dimensions

Each skill execution is scored 0-100 on five dimensions: Dimension · Weight · What It Measures Accuracy · 25% · Did the skill produce the correct result for the task? Relevance · 20% · Was the skill content applicable to the actual use case? Token Efficiency · 20% · Did the skill guide the agent without bloat or repetition?

Draft

New skills enter as Draft. They receive no special protection and are evaluated critically on first use. A Draft skill that scores below 30 on its very first run is discarded rather than repaired.

Active

Skills in regular use. Scores are tracked in ~/.claude/skill-scores.jsonl. No action unless scores trend below 30 or above 90 over a rolling window of 5 runs.

Crystallized

A skill that maintains an average composite score above 90 over 5 or more consecutive runs is crystallized: Git tag applied: skill/<name>/crystallized-v<N> Read-only flag added to frontmatter: locked: true Skill is excluded from auto-repair Changes require explicit human unlock + PR

Archived

A skill that fails auto-repair 3 times is archived: Moved to skills/_archived/<name>/ Git tag applied: skill/<name>/archived Replacement skill drafted by catalyst agent if the capability is still needed

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Engineering
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@vibeeval · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
macOS · Linux · Windows
Runtime requirements
Node.js
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Author source file SKILL.md

# Skill Evolution

Darwinian selection for skills. Skills that produce good outcomes are crystallized and protected. Skills that produce poor outcomes are repaired or archived. Every execution generates a score that drives the next generation of the skill.

## The 5 Scoring Dimensions

Each skill execution is scored 0-100 on five dimensions:

| Dimension | Weight | What It Measures |
|-----------|--------|-----------------|
| Accuracy | 25% | Did the skill produce the correct result for the task? |
| Relevance | 20% | Was the skill content applicable to the actual use case? |
| Token Efficiency | 20% | Did the skill guide the agent without bloat or repetition? |
| User Satisfaction | 20% | Did the outcome meet or exceed user expectations? |
| Reusability | 15% | Could another agent use this skill in a similar situation? |

**Composite score** = weighted average of all five dimensions (0-100).

### Scoring Rubric

```
90-100: Excellent -- candidate for crystallization
70-89:  Good -- active skill, no action needed
50-69:  Adequate -- flag for review after 3 more runs
30-49:  Poor -- schedule auto-repair attempt
0-29:   Critical -- immediate auto-repair or archive
```

## Skill Lifecycle

```
DRAFT          ACTIVE         CRYSTALLIZED      ARCHIVED
  |               |                |                |
New skill   In regular use   Proven stable    Deprecated/replaced
  |               |                |                |
  +-- first run ->+-- score >90   ++-- score <30    |
                  |   for 5+ runs  |   (3 attempts)  |
                  +-- score <30 -->+ auto-repair      |
                  |   auto-repair  |   fails 3x -->--+
                  +-- score >90 -->+
```

### Draft
New skills enter as Draft. They receive no special protection and are evaluated critically on first use. A Draft skill that scores below 30 on its very first run is discarded rather than repaired.

### Active
Skills in regular use. Scores are tracked in `~/.claude/skill-scores.jsonl`. No action unless scores trend below 30 or above 90 over a rolling window of 5 runs.

### Crystallized
A skill that maintains an average composite score above 90 over 5 or more consecutive runs is crystallized:
- Git tag applied: `skill/<name>/crystallized-v<N>`
- Read-only flag added to frontmatter: `locked: true`
- Skill is excluded from auto-repair
- Changes require explicit human unlock + PR

### Archived
A skill that fails auto-repair 3 times is archived:
- Moved to `skills/_archived/<name>/`
- Git tag applied: `skill/<name>/archived`
- Replacement skill drafted by `catalyst` agent if the capability is still needed

## Score Storage Format

Append one record per execution to `~/.claude/skill-scores.jsonl`:

```jsonl
{"skill":"experiment-loop","ts":"2026-04-07T10:00:00Z","session":"abc123","scores":{"accuracy":88,"relevance":92,"token_efficiency":75,"user_satisfaction":90,"reusability":85},"composite":86.5,"feedback":"Loop ran 4 iterations successfully, target nearly met"}
{"skill":"experiment-loop","ts":"2026-04-07T14:30:00Z","session":"def456","scores":{"accuracy":95,"relevance":90,"token_efficiency":82,"user_satisfaction":95,"reusability":88},"composite":90.4,"feedback":"Bundle size reduced 28%, target exceeded"}
```

### Score CLI (quick check)

```bash
# Average scores for a skill (last 10 runs)
cat ~/.claude/skill-scores.jsonl | python3 -c "
import sys, json, statistics
skill = '$1'
runs = [json.loads(l) for l in sys.stdin if json.loads(l).get('skill') == skill][-10:]
if runs:
    avg = statistics.mean(r['composite'] for r in runs)
    print(f'{skill}: {avg:.1f} avg over {len(runs)} runs')
"
```

## Crystallization Protocol

When a skill reaches 90+ composite score over 5+ consecutive runs:

1. Verify scores in `~/.claude/skill-scores.jsonl` -- confirm no outliers inflating the average
2. Add `locked: true` to the skill's frontmatter
3. Apply git tag:
   ```bash
   git tag skill/<name>/crystallized-v1 -m "Crystallized: avg score 92.3 over 7 runs"
   git push origin skill/<name>/crystallized-v1
   ```
4. Log the crystallization in `thoughts/SKILL-EVOLUTION.md`
5. Notify via canavar cross-training so all agents know this skill is stable

## Auto-Repair Protocol

When a skill's composite score drops below 30:

### Diagnosis
1. Identify the lowest-scoring dimension (the primary failure mode)
2. Read the last 3 session feedback notes from `~/.claude/skill-scores.jsonl`
3. Summarize what went wrong (specific, not vague)

### Repair
The `catalyst` agent rewrites the failing section(s) of the skill:
- Only the sections relevant to the low-scoring dimension
- Preserve all high-scoring sections unchanged
- Add a concrete example for the repaired section

### Validation
After repair, the skill is re-scored on a synthetic test case by the `verifier` agent:
- Synthetic score must be 50+ to proceed to Active state
- If synthetic score < 50, attempt 2 of 3 repairs begins

### Escalation
After 3 failed auto-repairs:
- Archive the skill
- Alert via `thoughts/SKILL-EVOLUTION.md`
- Spawn `catalyst` to draft a replacement from scratch

## Evolution Log Format

Append events to `thoughts/SKILL-EVOLUTION.md`:

```markdown
## 2026-04-07

### skill: experiment-loop
- Status change: Active -> Crystallized
- Trigger: avg composite 91.2 over 6 consecutive runs
- Git tag: skill/experiment-loop/crystallized-v1
- Notable strength: Token Efficiency dimension consistently 85+

### skill: legacy-deploy-helper
- Status change: Active -> Auto-Repair (attempt 1/3)
- Trigger: composite 24 on last run
- Lowest dimension: Relevance (12) -- skill referenced outdated Heroku patterns
- Repair: catalyst rewrote "Deployment Targets" section with Vercel/Railway focus
- Post-repair synthetic score: 71 -- promoted back to Active
```

## Integration with Canavar Cross-Training

Skill evolution data feeds into canavar's cross-training pipeline:

- A crystallized skill is injected into canavar's `skill-matrix.json` with `trust: locked`
- An archived skill is marked `trust: deprecated` -- agents stop referencing it
- Auto-repair failures are logged to `error-ledger.jsonl` with `source: skill-evolution`
- The canavar leaderboard tracks which agents most frequently produce high-scoring skill executions

```bash
# View crystallized skills
node ~/.claude/hooks/dist/canavar-cli.mjs leaderboard --filter crystallized

# View skills needing repair
cat ~/.claude/skill-scores.jsonl | python3 -c "
import sys, json, collections
runs = [json.loads(l) for l in sys.stdin]
low = {r['skill'] for r in runs if r['composite'] < 30}
print('Skills needing repair:', low)
"
```

## Activation

This skill activates automatically when:
- A skill completes an execution (PostToolUse hook)
- A skill is referenced in a session that ends with user dissatisfaction
- The `verifier` agent reports a skill-guided task as failed

Agents involved: `catalyst` (repair), `verifier` (validation), `self-learner` (feedback extraction), `canavar` (cross-training propagation).

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.