技能评估
- 作者仓库星标 386
- 作者仓库 vexjoy-agent
- 领域
- 工程开发
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @notque · 未声明许可证
- 上下文用量评级
- 较高消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- 未检出明确要求
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 读取环境变量
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
第 1 阶段:评估 — 确定要评估的内容
步骤 1:识别技能 这会检查:SKILL.md 是否存在、有效的 frontmatter、必填字段(名称、描述)、短横线命名、描述少于 1024 个字符、无尖括号。步骤2:根据用户意图选择评估模式 Intent · 模式 · 脚本“测试描述是否正确触发” · 触发评估 · `run_eval.py`
第 2 阶段:评估 — 进行适当的评估
模式 A:触发评估 测试技能的描述是否会导致 Claude 调用它来进行正确的查询。步骤 1:创建评估集(或使用现有的) 创建一个包含 8-20 个测试查询的 JSON 文件。评估集质量很重要——使用带有细节的现实提示(文件路径、上下文、随意的措辞),而不是抽象的俏皮话。重点关注该技能与相邻技能竞争的边缘情况。良好的评估查询示例:
第 3 阶段:改进 — 应用结果
第 1 步:查看结果 对于触发器评估/描述优化:显示最佳描述与原始描述 显示每个查询结果(哪些查询改进,哪些查询退化) 显示训练与测试分数
# Skill Evaluation & Improvement
Measure and improve skill quality through empirical testing — because structure doesn't guarantee behavior, and measurement beats assumption. Also covers head-to-head bake-offs of two peer implementations of the same artifact (Mode F).
## Reference Loading Table
| Signal | Load These Files | Why |
|---|---|---|
| tasks related to this reference | `schemas.md` | Loads detailed guidance from `schemas.md`. |
| tasks related to this reference | `self-improve-loop.md` | Loads detailed guidance from `self-improve-loop.md`. |
| "bake-off", "head-to-head", "compare implementations", "grade two versions", "which Feynman skill is better" | `bake-off-methodology.md` | Loads the bake-off rubric, anti-rationalization gate, fold-filter, and worked Feynman example. |
## Instructions
### Phase 1: ASSESS — Determine what to evaluate
**Step 1: Identify the skill**
```bash
# Validate skill structure first
python3 -m scripts.skill_eval.quick_validate <path/to/skill>
```
This checks: SKILL.md exists, valid frontmatter, required fields (name, description), kebab-case naming, description under 1024 chars, no angle brackets.
**Step 2: Choose evaluation mode based on user intent**
| Intent | Mode | Script |
|--------|------|--------|
| "Test if description triggers correctly" | Trigger eval | `run_eval.py` |
| "Optimize/improve the description through autoresearch" | Route to `agent-comparison` | `optimize_loop.py` |
| "Compare skill vs no-skill output" | Output benchmark | Manual + `aggregate_benchmark.py` |
| "Validate skill structure" | Quick validate | `quick_validate.py` |
| "Self-improve skill" / "optimize skill" / "improve skill with A/B" | Self-improvement loop | `references/self-improve-loop.md` |
| "Bake-off" / "head-to-head grade these two" / "compare X vs Y implementation" | Head-to-head bake-off | `references/bake-off-methodology.md` |
**GATE**: Skill path confirmed, mode selected.
### Phase 2: EVALUATE — Run the appropriate evaluation
#### Mode A: Trigger Evaluation
Test whether a skill's description causes Claude to invoke it for the right queries.
**Step 1: Create eval set** (or use existing)
Create a JSON file with 8-20 test queries. **Eval set quality matters** — use realistic prompts with detail (file paths, context, casual phrasing), not abstract one-liners. Focus on edge cases where the skill competes with adjacent skills.
Example of good eval queries:
```json
[
{"query": "ok so my boss sent me this xlsx file (Q4 sales final FINAL v2.xlsx) and she wants profit margin as a percentage", "should_trigger": true},
{"query": "Format this data", "should_trigger": false}
]
```
**Why**: Real users write detailed, specific prompts. Abstract queries don't test real triggering behavior. Overfitting descriptions to abstract test cases bloats the description and fails on real usage.
**Step 2: Run evaluation**
```bash
python3 -m scripts.skill_eval.run_eval \
--eval-set evals.json \
--skill-path <path/to/skill> \
--runs-per-query 3 \
--verbose
```
This spawns `claude -p` for each query, checking whether it invokes the skill. Runs each query 3 times for reliability. Output includes pass/fail per query with trigger rates. Default 30s timeout; increase with `--timeout 60` if needed for complex queries.
**Constraints applied**:
- Always run baseline eval before making improvements
- 3 runs per query ensures statistical reliability
- Verbose output shows per-query pass/fail during eval runs
**GATE**: Eval results available. Proceed to improvement if failures found.
#### Mode B: Description Optimization
Automated loop that tests, improves, and re-tests descriptions using Claude with extended thinking.
```bash
python3 -m scripts.skill_eval.run_loop \
--eval-set evals.json \
--skill-path <path/to/skill> \
--max-iterations 5 \
--verbose
```
This will:
1. Split eval set 60/40 train/test (stratified by should_trigger) — prevents overfitting to test cases
2. Evaluate current description on all queries (3 runs each for reliability)
3. Use `claude -p` to propose improvements based on training failures
4. Re-evaluate the new description
5. Repeat until all pass or max iterations reached
6. Select best description by **test** score (not train score — prevents overfitting)
7. Open an HTML report in the browser
**Why 60/40 split**: Improvements should help across many prompts, not just test cases. Training on failures, validating on holdout ensures generalization.
**Why report HTML**: Visual reports enable quick review of which queries improved, which regressed, and what the new description looks like.
**GATE**: Loop complete. Best description identified.
#### Mode C: Output Benchmark
Compare skill quality by running prompts with and without the skill.
**Step 1: Create test prompts** — 2-3 realistic user prompts
**Step 2: Run with-skill and without-skill** in parallel subagents:
For each test prompt, spawn two agents:
- **With skill**: Load the skill, run the prompt, save outputs
- **Without skill** (baseline): Same prompt, no skill, save outputs
**Why baseline matters**: Can't prove the skill adds value without a baseline. Maybe Claude handles it fine without the skill. The delta is what matters.
**Step 3: Grade outputs**
Spawn a grader subagent using `agents/grader.md`. It evaluates assertions against the outputs.
**Step 4: Aggregate**
```bash
python3 -m scripts.skill_eval.aggregate_benchmark <workspace>/iteration-1 --skill-name <name>
```
Produces `benchmark.json` and `benchmark.md` with pass rates, timing, and token usage.
**Step 5: Analyze** (optional)
For blind comparison, use `agents/comparator.md` to judge outputs without knowing which skill produced them. Then use `agents/analyzer.md` to understand why the winner won.
**GATE**: Benchmark results available.
#### Mode D: Quick Validate
```bash
python3 -m scripts.skill_eval.quick_validate <path/to/skill>
```
Checks: SKILL.md exists, valid frontmatter, required fields (name, description), kebab-case naming, description under 1024 chars, no angle brackets.
#### Mode E: Self-Improvement Loop
Automatically generate variants of a skill, A/B test them against the original, and promote winners. This is a closed-loop pipeline — baseline, hypothesize, generate, test, promote.
Read the full protocol: `${CLAUDE_SKILL_DIR}/references/self-improve-loop.md`
The loop runs 5 phases: BASELINE (establish metrics with 3+ test cases), HYPOTHESIZE (2-3 single-variable changes), GENERATE VARIANTS (minimal diffs), BLIND A/B TEST (paired comparison via `agents/comparator.md`), PROMOTE OR KEEP (60%+ win rate required, no regressions). All outcomes — wins and losses — are recorded to the learning DB to prevent re-testing failed hypotheses.
**GATE**: Self-improvement protocol loaded from reference. Proceed through the 5 phases.
#### Mode F: Head-to-Head Bake-Off
Score two peer implementations of the same artifact (e.g., toolkit `voice-feynman` vs an external Feynman voice profile) on a numeric rubric and declare a decisive winner. Use when the user says "bake-off", "head-to-head", "compare implementations", "grade these two", or "which X is better".
Read the full protocol: `${CLAUDE_SKILL_DIR}/references/bake-off-methodology.md`
The protocol runs 5 phases: PREPARE (read both artifacts in full, pick a verifier that built neither side), RUBRIC (define 5–12 criteria scored 0–10, pre-state the loser-of-each-criterion before reading evidence), GRADE (every score cites a path/line range or quote; build the matrix; apply anti-rationalization gate), FOLD (filter loser-wins through `docs/PHILOSOPHY.md` before recommending any folds into the winner), REPORT (output to `tmp/<topic>-bakeoff-report.md`, gitignored).
The Feynman bake-off (toolkit 86 vs external 74 across 11 criteria, 12-point margin) is the canonical worked example carried in the reference.
**GATE**: Bake-off protocol loaded from reference. Proceed through the 5 phases.
### Phase 3: IMPROVE — Apply results
**Step 1: Review results**
For trigger eval / description optimization:
- Show the best description vs original
- Show per-query results (which queries improved, which regressed)
- Show train vs test scores
For output benchmark:
- Show pass rate delta (with-skill vs without-skill)
- Show timing and token cost delta
- Highlight assertions that only pass with the skill (value-add)
**Step 2: Apply changes** (with user confirmation)
If description optimization found a better description:
1. Show before/after with scores
2. Ask user to confirm
3. Update the skill's SKILL.md frontmatter
4. Re-run quick_validate to confirm the update is valid
**Constraint**: Always show results before/after with metrics. This enables informed decisions.
**GATE**: Changes applied and validated, or user chose to keep original.
---
## Error Handling
### Error: "No SKILL.md found"
**Cause**: Skill path doesn't point to a valid skill directory
**Solution**: Verify path contains a `SKILL.md` file. Skills must follow the `skill-name/SKILL.md` structure.
### Error: "claude: command not found"
**Cause**: Claude CLI not available for trigger evaluation
**Solution**: Install Claude Code CLI. Trigger eval requires `claude -p` to test skill invocation.
### Error: "legacy SDK dependency"
**Cause**: Outdated instructions or an old checkout still expects a direct SDK client
**Solution**: Update to the current scripts. Description optimization now runs through `claude -p`.
### Error: "CLAUDECODE environment variable"
**Cause**: Running eval from inside a Claude Code session blocks nested instances
**Solution**: The scripts automatically strip the `CLAUDECODE` env var. If issues persist, run from a separate terminal.
### Error: "All queries timeout"
**Cause**: Default 30s timeout too short for complex queries
**Solution**: Increase with `--timeout 60`. Simple trigger queries should complete in <15s.
---
## References
### Scripts (in `scripts/skill_eval/`)
- `run_eval.py` — Trigger evaluation: tests description against query set
- `run_loop.py` — Eval+improve loop: automated description optimization
- `improve_description.py` — Single-shot description improvement via Claude API
- `generate_report.py` — HTML report from loop output
- `aggregate_benchmark.py` — Benchmark aggregation from grading results
- `quick_validate.py` — Structural validation of SKILL.md
### Bundled Agents (in `skills/meta/skill-eval/agents/`)
- `grader.md` — Evaluates assertions against execution outputs
- `comparator.md` — Blind A/B comparison of two outputs
- `analyzer.md` — Post-hoc analysis of why one version beat another
### Reference Files
- `${CLAUDE_SKILL_DIR}/references/schemas.md` — JSON schemas for evals.json, grading.json, benchmark.json
- `${CLAUDE_SKILL_DIR}/references/self-improve-loop.md` — Self-improvement loop protocol: variant generation, blind A/B testing, promotion criteria
- `${CLAUDE_SKILL_DIR}/references/bake-off-methodology.md` — Head-to-head bake-off protocol: rubric construction, anti-rationalization gate, philosophy-filtered fold-list, Feynman worked example
章节与文件索引
按说明文件中的顺序排列。
参考装载表
第 1 阶段:评估 — 确定要评估的内容
第 2 阶段:评估 — 进行适当的评估
第 3 阶段:改进 — 应用结果
错误:“未找到 SKILL.md”
文件/命令 -> schemas.md · self-improve-loop.md · bake-off-methodology.md · run_eval.py · agent-comparison · optimize_loop.py · aggregate_benchmark.py · quick_validate.py
内容校验值 -> e95b3a5a0a34
当前原文的自动摘译,尚未人工审校。
参考装载表
信号 · 加载这些文件 · 为什么任务与此参考相关 ·
schemas.md· 从schemas.md加载详细指导。与此参考相关的任务 ·self-improve-loop.md· 从self-improve-loop.md加载详细指导。 “烘焙”、“头对头”、“比较实现”、“两个版本”、“哪个费曼技能更好” ·bake-off-methodology.md· 加载烘焙标题、反合理化门、折叠过滤器和工作费曼示例。错误:“未找到 SKILL.md”
原因:技能路径未指向有效的技能目录解决方案:验证路径包含
SKILL.md文件。技能必须遵循skill-name/SKILL.md结构。