技能等级评分者
- 作者仓库星标 490
- 作者仓库 skillgrade
- 领域
- 写作
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @mgechev · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 需要 · OpenAI / Anthropic / Gemini
- 兼容的系统
- macOS · Linux · Windows
- 运行环境要求
- Node.js
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
# Skillgrade Grader Authoring
## Procedures
**Step 1: Identify the Grading Strategy**
1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).
**Step 2: Write a Deterministic Grader**
1. Create a script in the skill's `graders/` directory (bash or TypeScript).
2. The script must output a JSON object to stdout with the following structure:
```json
{"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]}
```
3. `score` (0.0–1.0) and `details` are required. `checks` is optional but recommended.
4. Read `references/grader-output-schema.md` for the full output specification.
5. Use `awk` for arithmetic in bash scripts — `bc` is not available in `node:20-slim`.
6. Reference the grader in eval.yaml:
```yaml
- type: deterministic
run: bash graders/check.sh
weight: 0.7
```
**Step 3: Write an LLM Rubric Grader**
1. Draft a rubric with explicit scoring criteria and point allocations.
2. Structure the rubric into weighted sections that sum to 1.0:
```
Workflow Compliance (0-0.5):
- Did the agent follow the mandatory workflow steps?
Efficiency (0-0.5):
- Completed in ≤5 commands without trial-and-error?
```
3. Reference the rubric in eval.yaml:
```yaml
- type: llm_rubric
rubric: |
[rubric text or file path]
weight: 0.3
provider: gemini # optional: gemini (default) | anthropic | openai
model: gemini-3-flash-preview # optional, each provider has a default model
```
4. For long rubrics, store in a separate file and reference by path: `rubric: rubrics/quality.md`.
**Step 4: Combine Multiple Graders**
1. Assign weights to each grader based on importance. Weights are normalized automatically.
2. Final reward is calculated as: `Σ (grader_score × weight) / Σ weight`.
3. Example configuration:
```yaml
graders:
- type: deterministic
run: bash graders/check.sh
weight: 0.7
- type: llm_rubric
rubric: rubrics/quality.md
weight: 0.3
```
**Step 5: Validate Graders**
1. Create a reference solution script that produces the expected output.
2. Run `skillgrade --validate` to verify graders score the reference solution correctly.
3. Test only deterministic graders: `skillgrade --grader=deterministic` (skips LLM calls, faster iteration).
4. Test only LLM rubric graders: `skillgrade --grader=llm_rubric`.
5. Run a specific eval with a specific grader type: `skillgrade --eval=my-eval --grader=deterministic`.
6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.
## Error Handling
* If a deterministic grader outputs non-JSON, ensure all `echo`/`console.log` statements except the final JSON result are redirected to stderr.
* If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: `GEMINI_API_KEY` (provider: gemini), `ANTHROPIC_API_KEY` (provider: anthropic), or `OPENAI_API_KEY` (provider: openai).
* To use a custom/self-hosted LLM endpoint, set `ANTHROPIC_BASE_URL` (for provider: anthropic) or `OPENAI_BASE_URL` (for provider: openai) — e.g. for Ollama or vLLM.
* If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.
章节与文件索引
按说明文件中的顺序排列。
程序
错误处理
文件/命令 -> graders/ · score · details · checks · references/grader-output-schema.md · awk · node:20-slim · rubric: rubrics/quality.md
内容校验值 -> 03f65f0912f8
当前原文的自动摘译,尚未人工审校。
程序
步骤 1:确定评分策略 确定任务是否需要客观验证(确定性)或定性评估(法学硕士标题)。对于大多数任务,将两者结合起来:确定性评分者验证结果(权重 0.7),LLM 评分标准评估方法质量(权重 0.3)。步骤 2:编写确定性评分器 在技能的
graders/目录中创建一个脚本(bash 或 TypeScript)。错误处理
如果确定性评分器输出非 JSON,请确保除最终 JSON 结果之外的所有
echo/console.log语句都重定向到 stderr。如果 LLM 评分标准返回 0.00 且缺少 API 密钥消息,请为您的提供商设置适当的密钥:GEMINI_API_KEY(提供商:gemini)、ANTHROPIC_API_KEY(提供商:anthropic)或OPENAI_API_KEY(提供商:openai)。要使用自定义/自托管 LLM 端点,请设置ANTHROPIC_BASE_URL(对于提供商:anthropic)或OPENAI_BASE_URL(对于提供商:openai) - 例如奥拉玛 (Ollama) 或 vLLM。