技能化
- 作者仓库星标 18,413
- 作者仓库 gbrain
- 领域
- 通用
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 92 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @garrytan · v1.1.0 · 未声明许可证
- 上下文用量评级
- 中等消耗
- 接入复杂程度
- 需手动接入
- 是否需要外部接口密钥
- 需要 · OpenAI / Anthropic
- 兼容的系统
- macOS · Linux · Windows
- 运行环境要求
- 未检出明确要求
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 读取环境变量
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
第 0 阶段:这应该是一项技能吗?
在技能化之前,检查:这个会被调用2次以上吗? (一次性工作≠技能)是否有超过 20 行逻辑? (简单的助手不需要完整的基础设施)它是否有用户实际上会说的明确的触发短语?如果这三个都不是,那就是脚本,而不是技能。继续前行。
第 1 步:选择有代表性的输入
选择能够练习该技能最难记录的用例的输入。如果不确定:使用 SKILL.md 中的主要触发器示例,或最近 7 天的内存文件中最复杂的现实世界输入。
第2步:运行技能,捕获输出
在代表性输入上运行技能。输出文件是被评估的内容。
第 3 步:运行评估门
该命令并行运行来自 3 个不同提供商的 3 个前沿模型,根据 5 个记录维度上的任务对输出进行评分,并在 `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` 下写入收据(sha-8 将收据绑定到当前 SKILL.md 内容 — 编辑后重新运行写入新收据)。
第 4 阶段:测试(第 4-6 项)
现在 eval 已经证明了质量,编写测试将其锁定:单元测试 - 确定性逻辑的每个分支。模拟外部调用。集成测试——达到真正的端点。捕捉错误模拟隐藏。 LLM 评估 — LLM 步骤的质量/正确性。比跨模式评估更轻——测试特定行为。
阶段 5:解析器 + 检查-可解析(第 7-9 项)
使用触发短语添加到 Skills/RESOLVER.md 用户实际类型 Resolver eval:馈送触发器,断言正确的路由 检查可解析:可从 Skills/RESOLVER.md 访问技能(不是孤立的) MECE 与其他技能不重叠
阶段 6:E2E + 大脑归档(第 10-11 项)
E2E烟雾:从触发到副作用的完整流程大脑归档:如果技能写入大脑页面,则添加到brain/RESOLVER.md
# Skillify — The Meta Skill
> **Relationship to `/cross-modal-review`:** That skill is the manual mid-flow
> "second opinion" gate (one model reviews work product before commit). This
> skill's Phase 3 below uses `gbrain eval cross-modal` instead — three
> different-provider frontier models score-and-iterate on a documented
> dimension list *before* tests cement behavior. Use `/cross-modal-review`
> for ad-hoc second opinions; use Phase 3 here when skillifying a feature.
## Contract
A feature is "properly skilled" when all 11 checklist items pass. Item 3
(cross-modal eval) is informational in v1.1.0 — it does not gate the
skillpack-check audit, but a missing or stale receipt is surfaced so the
user knows where the gate stands.
## The Checklist
```
□ 1. SKILL.md — skill file with frontmatter + contract + phases
□ 2. Code — deterministic script if applicable
□ 3. Cross-modal eval — 3 frontier models from 3 providers; informational
□ 4. Unit tests — cover every branch of deterministic logic
□ 5. Integration tests — exercise live endpoints
□ 6. LLM evals — quality/correctness cases for LLM-involving steps
□ 7. Resolver trigger — entry in skills/RESOLVER.md with real user trigger phrases
□ 8. Resolver eval — test that triggers route to this skill
□ 9. Check-resolvable — DRY + MECE audit, no orphans
□ 10. E2E test — smoke test: trigger → side effect
□ 11. Brain filing — if it writes pages, entry in brain/RESOLVER.md
```
## Phase 0: Should This Be a Skill?
Before skillifying, check:
- Will this be invoked 2+ times? (One-off work ≠ skill)
- Is there >20 lines of logic? (Trivial helpers don't need full infrastructure)
- Does it have a clear trigger phrase a user would actually say?
If no to all three, it's a script, not a skill. Move on.
## Phase 1: Audit
```
Feature: [name]
Code: [path]
Missing items: [check each of the 11]
```
## Phase 2: Write SKILL.md + Code (items 1-2)
### SKILL.md frontmatter template (copy-paste):
```yaml
---
name: my-skill
version: 1.0.0
description: |
One paragraph. What it does, when to use it.
triggers:
- "trigger phrase users actually say"
- "another real trigger"
tools:
- exec
- read
- write
mutating: false # true if it writes to brain/disk
---
```
Body must include: **Contract** (what it guarantees), **Phases** (step-by-step), **Output Format** (what it produces).
Extract deterministic code into `scripts/*.ts`.
## Phase 3: Cross-Modal Eval (item 3) — THE QUALITY GATE
### Why this comes before tests
Tests lock in behavior. If the behavior is mediocre, tests lock in mediocrity.
Cross-modal eval proves the quality bar FIRST, then tests cement it.
### Step 1: Pick a representative input
Choose the input that exercises the skill's hardest documented use case. If
unsure: use the primary trigger example from SKILL.md, or the most complex
real-world input from the last 7 days of memory files.
### Step 2: Run the skill, capture output
Run the skill on the representative input. The OUTPUT FILE is what gets
evaluated.
### Step 3: Run the eval gate
```bash
gbrain eval cross-modal \
--task "What this skill is supposed to accomplish" \
--output skills/<slug>/SKILL.md
```
The command runs 3 frontier models from 3 different providers in parallel,
scores the OUTPUT against the TASK on 5 documented dimensions, and writes a
receipt under `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` (the
sha-8 binds the receipt to the current SKILL.md content — re-running after
edits writes a new receipt).
**Default models** (override per slot via `--slot-a-model`, `--slot-b-model`,
`--slot-c-model`):
| Slot | Default | Provider |
|------|---------|----------|
| A | `openai:gpt-4o` | OpenAI |
| B | `anthropic:claude-opus-4-7` | Anthropic |
| C | `google:gemini-1.5-pro` | Google |
**These MUST be frontier models from DIFFERENT providers.** Using a single
provider's family or budget models defeats the purpose — different families
have less correlated blind spots. Refresh the list when a new model
generation ships.
**Pass criteria (BOTH must be true):**
1. Every dimension's mean across successful models ≥ 7.
2. No single model scored any dimension < 5 (the floor).
**Inconclusive:** fewer than 2 of 3 models returned parseable scores.
Receipt is still written (forensics) but the gate is not authoritative.
Exit code 2; CI wrappers should treat this as "did not run cleanly", not
"failed quality gate".
### Step 4: Cycle until you pass (≤3 cycles)
```
CYCLE 1:
Eval → scores + top 10 improvements
IF pass: → done, write tests
ELSE:
Apply top 10 improvements to the actual file
Log: which improvements applied, what changed
CYCLE 2:
Re-eval the FIXED output (same 3 models, same dimensions)
Compare: before/after scores per dimension (track delta)
IF pass: → done, write tests
ELSE: apply remaining improvements + new ones
CYCLE 3 (final):
Re-eval
IF pass: → ship
ELSE: → ship with KNOWN_GAPS section listing:
- Which dimensions are still below 7
- Which improvements couldn't be resolved
- Why (e.g., "would require architectural change")
```
### Cycles + cost guardrails
- Default `--cycles 3` in TTY, `--cycles 1` in non-TTY (limits scripted
bulk spend in CI loops).
- The command prints an estimated max-cost-per-cycle from a small pricing
constant before each run. Real cost varies with prompt size; treat the
estimate as a ceiling for default `--max-tokens 4000`.
- A `--budget-usd N` hard cap is a v0.27.x follow-up TODO.
### Provider configuration
Models resolve through the gbrain AI gateway. Configure once with:
```bash
gbrain providers test # see what's configured
gbrain config # set keys
```
Or set env vars: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
`GOOGLE_GENERATIVE_AI_API_KEY`, `TOGETHER_API_KEY`, etc. The gateway reads
from `~/.gbrain/config.json` plus `process.env`.
### Cost expectations
3 cycles × 3 models = 9 frontier calls max per run. With Opus-class +
GPT-4o-class + Gemini-1.5-Pro, expect $1–3 per full run on default
`--max-tokens 4000`. Receipts include the per-call model identifiers so
you can audit retroactively.
### Skip cross-modal eval when:
- Output is < 200 tokens (trivial — not worth 9 API calls).
- The skill is a thin wrapper around a single API call (one cycle is enough).
## Phase 4: Tests (items 4-6)
NOW that eval has proven quality, write tests that lock it in:
**Unit tests** — every branch of deterministic logic. Mock external calls.
**Integration tests** — hit real endpoints. Catch bugs mocks hide.
**LLM evals** — quality/correctness for LLM steps. Lighter than cross-modal eval — test specific behaviors.
## Phase 5: Resolver + Check-Resolvable (items 7-9)
1. Add to skills/RESOLVER.md with trigger phrases users ACTUALLY type
2. Resolver eval: feed triggers, assert correct routing
3. Check-resolvable:
- Skill reachable from skills/RESOLVER.md (not orphaned)
- No MECE overlap with other skills
- No DRY violations (shared logic in lib/, not copy-pasted)
- No ambiguous trigger routing
## Phase 6: E2E + Brain Filing (items 10-11)
- E2E smoke: full pipeline from trigger to side effect
- Brain filing: add to brain/RESOLVER.md if the skill writes brain pages
## Phase 7: Verify
```bash
bun test test/<skill>.test.ts # unit tests
gbrain skillify check skills/<slug>/scripts/<slug>.mjs --json | \
jq '.[] | .items[] | select(.name | contains("Cross-modal"))'
ls ~/.gbrain/.gbrain/eval-receipts/ # receipt landed
gbrain check-resolvable --json | jq .ok # resolver clean
```
## Worked Example: Skillifying a "summarize-pr" Feature
```
Phase 0: Yes — invoked weekly, 50+ lines, clear trigger "summarize this PR"
Phase 1: Audit → SKILL.md missing, no tests, no resolver entry. Score: 1/11
Phase 2: Write SKILL.md + extract script to scripts/summarize-pr.ts
Phase 3: Cross-modal eval cycle 1 →
GPT-4o: goal=6, depth=5, specificity=4 → "misses file-level diffs"
Opus 4.7: goal=7, depth=6, specificity=5 → "no test plan in summary"
Gemini 1.5 Pro: goal=6, depth=5, specificity=5 → "template feels generic"
Aggregate: goal=6.3 FAIL, depth=5.3 FAIL
Top improvements: add file-level changes, include test plan, use PR context
→ Apply fixes → Cycle 2: goal=8, depth=7.5, specificity=7 → PASS
Phase 4: Write 12 unit tests locking in the improved behavior
Phase 5: Add "summarize this PR" trigger to skills/RESOLVER.md
Phase 6: E2E test: feed a real PR URL → verify brain page created
Phase 7: All green. Score: 11/11
```
## Quality Gates
NOT properly skilled until:
- All required items pass (1-2, 4-10; 11 only when applicable).
- Cross-modal eval (item 3) has a current receipt OR is explicitly waived
with rationale (item 3 is informational; not blocking, but a missing
receipt is visible in the audit).
- All tests pass (unit + integration + LLM evals).
- Resolver entry exists with real trigger phrases.
- Check-resolvable shows no orphans, overlaps, or DRY violations.
- Brain filing if applicable.
## Output Format
Skillify produces three durable artifacts per skill:
1. **The skill tree on disk.** `skills/<slug>/SKILL.md`, `scripts/<slug>.mjs`,
`routing-eval.jsonl`, plus a `test/<slug>.test.ts` skeleton. Generated by
`gbrain skillify scaffold <name>` and refined by the human/agent into a
real implementation.
2. **A cross-modal eval receipt** at
`~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json`. The sha-8 binds the
receipt to the current `SKILL.md` content. `gbrain skillify check`
surfaces the status (`found` / `stale` / `missing`) as informational.
3. **An audit verdict** from `gbrain skillify check`: `properly skilled` |
`close — create: <missing items>` | `needs skillify — run /skillify on
<target>`. Score is `<passed>/<total>`. Required items gate the verdict;
item 11 (cross-modal eval) is informational and never blocks PASS.
JSON output (`gbrain skillify check --json`) includes the same fields plus
the per-item detail string, so agents can route on the structured envelope
without parsing prose.
## Anti-Patterns
- ❌ Writing tests before cross-modal eval (locks in mediocrity)
- ❌ Using budget models for eval (C student grading A student)
- ❌ Using a single provider's family for all 3 slots (correlated blind spots)
- ❌ Skipping eval "because the output looks fine" (your judgment isn't 3 models)
- ❌ Eval without fix cycle (vanity metrics)
- ❌ Code with no SKILL.md (invisible to resolver)
- ❌ Tests that reimplement production code (masks real bugs)
- ❌ Resolver entry with internal jargon (must mirror real user language)
- ❌ Two skills doing the same thing (merge or kill one)
- ❌ Running cross-modal eval on trivial outputs (< 200 tokens, not worth 9 API calls)
章节与文件索引
按说明文件中的顺序排列。
合同
第 0 阶段:这应该是一项技能吗?
SKILL.md frontmatter 模板(复制粘贴):
为什么这发生在测试之前
第 1 步:选择有代表性的输入
文件/命令 -> /cross-modal-review · gbrain eval cross-modal · scripts/*.ts · ~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json · --slot-a-model · --slot-b-model · --slot-c-model · openai:gpt-4o
内容校验值 -> bdcf6dfb55db
当前原文的自动摘译,尚未人工审校。
合同
当所有 11 个清单项目都通过时,某个功能就被“熟练掌握”。第 3 项(跨模式评估)在 v1.1.0 中是信息性的 — 它不会对技能包检查审核进行门控,但会显示丢失或过时的收据,以便用户知道门的位置。
SKILL.md frontmatter 模板(复制粘贴):
正文必须包括:合同(它保证什么)、阶段(逐步)、输出格式(它产生什么)。将确定性代码提取到
scripts/*.ts中。为什么这发生在测试之前
测试锁定行为。如果行为平庸,测试就会锁定平庸。跨模式评估首先证明质量标准,然后进行测试巩固它。