技能等级设置

写作 社区

技能解读

当前原文的自动摘译,尚未人工审校。

程序

第 1 步:安装 Skillgrade 验证 Node.js 20+ 和 Docker 是否可用。运行 npm i -g skillgrade 以全局安装 CLI。步骤 2:初始化评估配置 导航到技能目录(必须包含SKILL.md)。

错误处理

如果 skillgrade init 失败并显示“未找到 SKILL.md”,请验证当前目录是否包含有效的 SKILL.md 文件。如果评估挂起,请检查 Docker 是否正在运行以及容器是否具有 API 调用的网络访问权限。如果所有试验均因“无 API 密钥”而失败,请确保导出环境变量,而不仅仅是为不同的命令设置内联变量。

流狐档案 运行与权限由流狐静态检查推断,不代表作者声明或实际测试结果。
领域
写作
作者声明的智能体
未找到明确声明;不据此推断已兼容或已测试
静态检查
88 / 100 · 启发式扫描,不代表运行安全
运行环境证据
未执行;当前没有隔离运行测试证据
作者 / 版本 / 许可证
@mgechev · 未声明许可证
上下文用量评级
低消耗
接入复杂程度
需手动接入
是否需要外部接口密钥
需要 · OpenAI / Anthropic / Gemini
兼容的系统
Docker
运行环境要求
Node.js · Docker
检测到的文件与系统行为
  • 只读
  • 允许写入 / 修改
  • 读取环境变量
网络行为
允许外网请求
安装命令数
无(仅作为资料)

需要注意: 未声明工具权限范围;实际权限由运行环境决定。

作者说明文件(原文) SKILL.md

# Skillgrade Evaluation Setup

## Procedures

**Step 1: Install Skillgrade**
1. Verify Node.js 20+ and Docker are available.
2. Run `npm i -g skillgrade` to install the CLI globally.

**Step 2: Initialize an Eval Configuration**
1. Navigate to the skill directory (must contain a `SKILL.md`).
2. Set the appropriate API key environment variable (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`).
3. Run `skillgrade init` to generate an `eval.yaml` with AI-powered tasks and graders.
4. If an `eval.yaml` already exists, pass `--force` to overwrite: `skillgrade init --force`.
5. Without an API key, a well-commented template is generated instead.

**Step 3: Configure eval.yaml**
1. Read `references/eval-yaml-spec.md` for the full configuration schema.
2. Define one or more tasks under the `tasks:` key. Each task requires:
   - `name`: unique task identifier
   - `instruction`: what the agent should accomplish
   - `workspace`: files to copy into the evaluation container
   - `graders`: one or more scoring mechanisms (see the `skillgrade-graders` skill)
3. Optionally configure `defaults:` for agent, provider, trials, timeout, and threshold.

**Step 4: Run Evaluations**
1. Select an appropriate preset based on the evaluation goal:
   - `--smoke` (5 trials): Quick capability check.
   - `--reliable` (15 trials): Reliable pass rate estimate.
   - `--regression` (30 trials): High-confidence regression detection.
2. Run the evaluation: `skillgrade --smoke`.
3. Run a specific eval by name: `skillgrade --eval=fix-linting`.
4. Run multiple evals: `skillgrade --eval=fix-linting,write-tests`.
5. Run only deterministic graders (skip LLM calls): `skillgrade --grader=deterministic`.
6. Run only LLM rubric graders: `skillgrade --grader=llm_rubric`.
7. The agent is auto-detected from the API key. Override with `--agent=gemini|claude|codex`.
8. Override the provider with `--provider=docker|local`.

**Step 5: Review Results**
1. Run `skillgrade preview` for a CLI report.
2. Run `skillgrade preview browser` to open the web UI at `http://localhost:3847`.
3. Reports are saved to `$TMPDIR/skillgrade/<skill-name>/results/`. Override with `--output=DIR`.

**Step 6: Integrate with CI**
1. Add a GitHub Actions step that installs skillgrade, navigates to the skill directory, and runs with `--regression --ci --provider=local`.
2. Use `--provider=local` in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.
3. The `--ci` flag causes a non-zero exit code if the pass rate falls below `--threshold` (default: 0.8).
4. Read `references/ci-example.md` for a complete workflow template.

## Error Handling
* If `skillgrade init` fails with "No SKILL.md found," verify the current directory contains a valid `SKILL.md` file.
* If evaluation hangs, check Docker is running and the container has network access for API calls.
* If all trials fail with "No API key," ensure the environment variable is exported, not just set inline for a different command.

讨论

基于 GitHub Discussions。登录 GitHub 即可参与讨论、点赞、订阅更新。