技能蒸馏师
- 作者仓库星标 236
- 作者仓库 armory
- 领域
- 通用
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @Mathews-Tom · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- Python
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
对源 SKILL.md 的每个部分进行推理难度评分: 复杂性信号 · 得分 · 具有 3 个以上分支的蒸馏操作决策树 · 高 · 转换为显式 if/then 查找表“使用判断”或“考虑上下文” · 高 · 用具体的启发式规则替换 多步推理链 · 高 · 分解为编号的原子步骤
使用 Opus 执行 5 个代表性任务的源技能:从`evals/cases.yaml`(正面案例)中选择任务或生成新任务 对于每个任务,捕获完整的执行跟踪: 进行了工具调用(哪些工具,按什么顺序) 输出中可见的中间推理
从收集的跟踪中,提取确定性模式: 决策路径 - 对于每个高复杂性部分,找到 Opus 在 5 个任务中做出的实际决策。如果 Opus 在 4/5 的情况下选择相同的路径,则该路径将成为默认规则 查找表 - Opus 应用领域知识的地方,构建显式查找表(例如,“如果输入包含 SQL,则使用这些模式;如果输入包含 Python,则使用这些模式”)
重写 SKILL.md,应用第 1 阶段中的所有蒸馏操作:源模式·蒸馏替换“分析代码并确定...”·“检查这 5 个特定模式:[列表]”“使用适当的格式”·“输出为包含列的降价表:[A、B、C]”“考虑上下文来决定...”·“如果 [条件 A]:执行 X。如果 [条件 B]:执行 Y。默认值:Z”
在目标模型(Haiku 或 Sonnet)上运行蒸馏技能:在加载蒸馏技能的情况下执行第 2 阶段中相同的 5 个任务 使用`surrogate-verifier` 为每个任务输出生成断言 比较通过率:指标 · 源(作品 + 原始) · 目标(俳句 + 蒸馏) · Delta
产生最终比较:
# Skill Distiller
Transform skills authored for high-capability models (Opus) into deterministic workflows
that execute reliably on lower-cost models (Sonnet, Haiku). The core insight from
EvoSkills: skills encode reusable task structure, not model-specific artifacts. A skill
evolved on Opus transfers with +35-45pp gains to other models — but only when the
instructions are sufficiently deterministic that lower-capability models can follow them
without improvising.
## Reference Files
| File | Contents | Load When |
| -------------------------------------- | -------------------------------------------------- | ------------------------------------ |
| `references/distillation-patterns.md` | Pattern catalog for converting reasoning to rules | Always |
## Prerequisites
- The source skill must exist and pass `package-evaluator` at >= 70%
- Access to both the source model (Opus) and target model (Haiku/Sonnet) for validation
- The `surrogate-verifier` skill for cross-model assertion checking
## Workflow
### Phase 1: Complexity Analysis
Score each section of the source SKILL.md for reasoning difficulty:
| Complexity Signal | Score | Distillation Action |
| ------------------------------------- | ----- | -------------------------------------------- |
| Decision tree with 3+ branches | HIGH | Convert to explicit if/then lookup table |
| "Use judgment" or "consider context" | HIGH | Replace with concrete heuristic rules |
| Multi-step inference chain | HIGH | Break into numbered atomic steps |
| Reference to domain expertise | MED | Add explicit reference file with knowledge |
| Clear enumerated steps | LOW | Keep as-is |
| Concrete examples with expected output| LOW | Keep as-is |
Produce a complexity map: section name -> complexity score -> planned action.
### Phase 2: Trace Collection
Execute the source skill with Opus on 5 representative tasks:
1. Select tasks from `evals/cases.yaml` (positive cases) or generate new ones
2. For each task, capture the full execution trace:
- Tool calls made (which tools, in what order)
- Intermediate reasoning visible in output
- Final output structure and content
- Time taken and token usage
3. Store traces as structured data for pattern extraction
### Phase 3: Pattern Extraction
From the collected traces, extract deterministic patterns:
1. **Decision paths** — For each HIGH-complexity section, find the actual decisions Opus made across
the 5 tasks. If Opus chose the same path in 4/5 cases, that path becomes the default rule
2. **Lookup tables** — Where Opus applied domain knowledge, build explicit lookup tables
(e.g., "if input contains SQL, use these patterns; if input contains Python, use those")
3. **Concrete examples** — Extract representative input/output pairs from traces to serve as
few-shot examples in the distilled skill
4. **Tool sequences** — Identify the common tool invocation pattern and make it explicit
("Step 1: Read the file. Step 2: Grep for pattern X. Step 3: Write output.")
### Phase 4: Distilled Rewrite
Rewrite the SKILL.md applying all distillation actions from Phase 1:
| Source Pattern | Distilled Replacement |
| -------------------------------------- | ------------------------------------------------------------ |
| "Analyze the code and determine..." | "Check for these 5 specific patterns: [list]" |
| "Use appropriate formatting" | "Output as a markdown table with columns: [A, B, C]" |
| "Consider the context to decide..." | "If [condition A]: do X. If [condition B]: do Y. Default: Z" |
| "Apply best practices for..." | Reference file with explicit best practices enumerated |
| Multi-paragraph reasoning instruction | Numbered step list with single-sentence steps |
Rules for the rewrite:
- Every instruction must be actionable by a model with no domain expertise
- No step should require inference — each step's input and output must be explicit
- Replace all "consider", "analyze", "determine" verbs with "check", "count", "list", "output"
- Add concrete examples for any step that could be ambiguous
- Keep the SKILL.md under 500 lines (distillation should reduce, not expand)
### Phase 5: Target Model Validation
Run the distilled skill on the target model (Haiku or Sonnet):
1. Execute the same 5 tasks from Phase 2 with the distilled skill loaded
2. Use the `surrogate-verifier` to generate assertions for each task output
3. Compare pass rates:
| Metric | Source (Opus + original) | Target (Haiku + distilled) | Delta |
| ------------------------------- | ------------------------ | -------------------------- | ----- |
| Assertions passed | N/M | N/M | ± |
| Weighted score | X.XX | X.XX | ± |
| Output completeness | % | % | ± |
| Format compliance | % | % | ± |
4. If target model score < 80% of source model score, iterate:
- Identify which assertions the target model fails
- Add more explicit instructions for those specific failure points
- Re-run validation (max 3 iterations)
### Phase 6: Cross-Model Report
Produce the final comparison:
```markdown
# Skill Distillation Report: <skill-name>
## Complexity Reduction
- Sections distilled: N/M (HIGH → LOW)
- Instruction word count: original X → distilled Y (Z% reduction)
- Decision points replaced with lookup tables: N
## Cross-Model Performance
| Model | Assertions Passed | Weighted Score | Format Compliance |
|---------|-------------------|----------------|-------------------|
| Opus | 7/7 | 1.00 | 100% |
| Sonnet | 6/7 | 0.92 | 100% |
| Haiku | 5/7 | 0.85 | 85% |
## Changes Made
1. [Section] "Analyze complexity" → explicit 5-item checklist
2. [Section] "Apply formatting" → fixed markdown table template
...
## Recommendation
[SHIP | ITERATE | MANUAL_REVIEW_NEEDED]
```
## Error Handling
| Error | Resolution |
| ----------------------------------- | ------------------------------------------------------------- |
| Source skill scores below 70% | Refuse distillation; recommend evolution via test-engineer |
| No execution traces available | Generate synthetic tasks and collect traces before proceeding |
| Target model fails all assertions | Skill may be too complex for target model; report with detail |
| Distilled skill longer than source | Review distillation; patterns may need consolidation |
## Limitations
- Cannot distill skills that rely on open-ended adaptive reasoning at many decision points or multi-turn reasoning
- Visual/interactive skills (HTML generation, browser automation) may not distill well
- Distillation optimizes for determinism, not creativity — skills requiring open-ended generation
(writing, brainstorming) are poor candidates
- Trace collection requires actual model execution, incurring API costs
按说明文件中的顺序排列。
参考文件
先决条件
第一阶段:复杂性分析
第 2 阶段:痕迹收集
第三阶段:模式提取
文件/命令 -> references/distillation-patterns.md · package-evaluator · surrogate-verifier · evals/cases.yaml
内容校验值 -> 5c06a8d50963
当前原文的自动摘译,尚未人工审校。
参考文件
文件·内容·加载时间
references/distillation-patterns.md·用于将推理转换为规则的模式目录·始终先决条件
源技能必须存在并通过
package-evaluator>= 70% 可以访问源模型 (Opus) 和目标模型 (Haiku/Sonnet) 进行验证surrogate-verifier跨模型断言检查技能第 2 阶段:痕迹收集
使用 Opus 执行 5 个代表性任务的源技能:从
evals/cases.yaml(正面案例)中选择任务或生成新任务 对于每个任务,捕获完整的执行跟踪:进行的工具调用(哪些工具,按什么顺序) 输出中可见的中间推理