技能等级设置
- 作者仓库星标 490
- 作者仓库 skillgrade
- 领域
- 写作
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @mgechev · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需手动接入
- 是否需要外部接口密钥
- 需要 · OpenAI / Anthropic / Gemini
- 兼容的系统
- Docker
- 运行环境要求
- Node.js · Docker
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 读取环境变量
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
# Skillgrade Evaluation Setup
## Procedures
**Step 1: Install Skillgrade**
1. Verify Node.js 20+ and Docker are available.
2. Run `npm i -g skillgrade` to install the CLI globally.
**Step 2: Initialize an Eval Configuration**
1. Navigate to the skill directory (must contain a `SKILL.md`).
2. Set the appropriate API key environment variable (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`).
3. Run `skillgrade init` to generate an `eval.yaml` with AI-powered tasks and graders.
4. If an `eval.yaml` already exists, pass `--force` to overwrite: `skillgrade init --force`.
5. Without an API key, a well-commented template is generated instead.
**Step 3: Configure eval.yaml**
1. Read `references/eval-yaml-spec.md` for the full configuration schema.
2. Define one or more tasks under the `tasks:` key. Each task requires:
- `name`: unique task identifier
- `instruction`: what the agent should accomplish
- `workspace`: files to copy into the evaluation container
- `graders`: one or more scoring mechanisms (see the `skillgrade-graders` skill)
3. Optionally configure `defaults:` for agent, provider, trials, timeout, and threshold.
**Step 4: Run Evaluations**
1. Select an appropriate preset based on the evaluation goal:
- `--smoke` (5 trials): Quick capability check.
- `--reliable` (15 trials): Reliable pass rate estimate.
- `--regression` (30 trials): High-confidence regression detection.
2. Run the evaluation: `skillgrade --smoke`.
3. Run a specific eval by name: `skillgrade --eval=fix-linting`.
4. Run multiple evals: `skillgrade --eval=fix-linting,write-tests`.
5. Run only deterministic graders (skip LLM calls): `skillgrade --grader=deterministic`.
6. Run only LLM rubric graders: `skillgrade --grader=llm_rubric`.
7. The agent is auto-detected from the API key. Override with `--agent=gemini|claude|codex`.
8. Override the provider with `--provider=docker|local`.
**Step 5: Review Results**
1. Run `skillgrade preview` for a CLI report.
2. Run `skillgrade preview browser` to open the web UI at `http://localhost:3847`.
3. Reports are saved to `$TMPDIR/skillgrade/<skill-name>/results/`. Override with `--output=DIR`.
**Step 6: Integrate with CI**
1. Add a GitHub Actions step that installs skillgrade, navigates to the skill directory, and runs with `--regression --ci --provider=local`.
2. Use `--provider=local` in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.
3. The `--ci` flag causes a non-zero exit code if the pass rate falls below `--threshold` (default: 0.8).
4. Read `references/ci-example.md` for a complete workflow template.
## Error Handling
* If `skillgrade init` fails with "No SKILL.md found," verify the current directory contains a valid `SKILL.md` file.
* If evaluation hangs, check Docker is running and the container has network access for API calls.
* If all trials fail with "No API key," ensure the environment variable is exported, not just set inline for a different command.
章节与文件索引
按说明文件中的顺序排列。
程序
错误处理
文件/命令 -> npm i -g skillgrade · GEMINI_API_KEY · ANTHROPIC_API_KEY · OPENAI_API_KEY · skillgrade init · eval.yaml · --force · skillgrade init --force
内容校验值 -> 16ebf8cdf65f
当前原文的自动摘译,尚未人工审校。
程序
第 1 步:安装 Skillgrade 验证 Node.js 20+ 和 Docker 是否可用。运行
npm i -g skillgrade以全局安装 CLI。步骤 2:初始化评估配置 导航到技能目录(必须包含SKILL.md)。错误处理
如果
skillgrade init失败并显示“未找到 SKILL.md”,请验证当前目录是否包含有效的SKILL.md文件。如果评估挂起,请检查 Docker 是否正在运行以及容器是否具有 API 调用的网络访问权限。如果所有试验均因“无 API 密钥”而失败,请确保导出环境变量,而不仅仅是为不同的命令设置内联变量。