技能等级评分者

写作 社区

技能解读

当前原文的自动摘译,尚未人工审校。

程序

步骤 1:确定评分策略 确定任务是否需要客观验证(确定性)或定性评估(法学硕士标题)。对于大多数任务,将两者结合起来:确定性评分者验证结果(权重 0.7),LLM 评分标准评估方法质量(权重 0.3)。步骤 2:编写确定性评分器 在技能的 graders/ 目录中创建一个脚本(bash 或 TypeScript)。

错误处理

如果确定性评分器输出非 JSON,请确保除最终 JSON 结果之外的所有 echo/console.log 语句都重定向到 stderr。如果 LLM 评分标准返回 0.00 且缺少 API 密钥消息,请为您的提供商设置适当的密钥:GEMINI_API_KEY(提供商:gemini)、ANTHROPIC_API_KEY(提供商:anthropic)或OPENAI_API_KEY(提供商:openai)。要使用自定义/自托管 LLM 端点,请设置 ANTHROPIC_BASE_URL (对于提供商:anthropic)或 OPENAI_BASE_URL (对于提供商:openai) - 例如奥拉玛 (Ollama) 或 vLLM。

流狐档案 运行与权限由流狐静态检查推断,不代表作者声明或实际测试结果。
领域
写作
作者声明的智能体
未找到明确声明;不据此推断已兼容或已测试
静态检查
88 / 100 · 启发式扫描,不代表运行安全
运行环境证据
未执行;当前没有隔离运行测试证据
作者 / 版本 / 许可证
@mgechev · 未声明许可证
上下文用量评级
低消耗
接入复杂程度
需简单配置
是否需要外部接口密钥
需要 · OpenAI / Anthropic / Gemini
兼容的系统
macOS · Linux · Windows
运行环境要求
Node.js
检测到的文件与系统行为
  • 只读
  • 允许写入 / 修改
网络行为
允许外网请求
安装命令数
无(仅作为资料)

需要注意: 未声明工具权限范围;实际权限由运行环境决定。

作者说明文件(原文) SKILL.md

# Skillgrade Grader Authoring

## Procedures

**Step 1: Identify the Grading Strategy**
1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).

**Step 2: Write a Deterministic Grader**
1. Create a script in the skill's `graders/` directory (bash or TypeScript).
2. The script must output a JSON object to stdout with the following structure:
   ```json
   {"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]}
   ```
3. `score` (0.0–1.0) and `details` are required. `checks` is optional but recommended.
4. Read `references/grader-output-schema.md` for the full output specification.
5. Use `awk` for arithmetic in bash scripts — `bc` is not available in `node:20-slim`.
6. Reference the grader in eval.yaml:
   ```yaml
   - type: deterministic
     run: bash graders/check.sh
     weight: 0.7
   ```

**Step 3: Write an LLM Rubric Grader**
1. Draft a rubric with explicit scoring criteria and point allocations.
2. Structure the rubric into weighted sections that sum to 1.0:
   ```
   Workflow Compliance (0-0.5):
   - Did the agent follow the mandatory workflow steps?
   Efficiency (0-0.5):
   - Completed in ≤5 commands without trial-and-error?
   ```
3. Reference the rubric in eval.yaml:
   ```yaml
   - type: llm_rubric
     rubric: |
       [rubric text or file path]
     weight: 0.3
     provider: gemini               # optional: gemini (default) | anthropic | openai
     model: gemini-3-flash-preview  # optional, each provider has a default model
   ```
4. For long rubrics, store in a separate file and reference by path: `rubric: rubrics/quality.md`.

**Step 4: Combine Multiple Graders**
1. Assign weights to each grader based on importance. Weights are normalized automatically.
2. Final reward is calculated as: `Σ (grader_score × weight) / Σ weight`.
3. Example configuration:
   ```yaml
   graders:
     - type: deterministic
       run: bash graders/check.sh
       weight: 0.7
     - type: llm_rubric
       rubric: rubrics/quality.md
       weight: 0.3
   ```

**Step 5: Validate Graders**
1. Create a reference solution script that produces the expected output.
2. Run `skillgrade --validate` to verify graders score the reference solution correctly.
3. Test only deterministic graders: `skillgrade --grader=deterministic` (skips LLM calls, faster iteration).
4. Test only LLM rubric graders: `skillgrade --grader=llm_rubric`.
5. Run a specific eval with a specific grader type: `skillgrade --eval=my-eval --grader=deterministic`.
6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.

## Error Handling
* If a deterministic grader outputs non-JSON, ensure all `echo`/`console.log` statements except the final JSON result are redirected to stderr.
* If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: `GEMINI_API_KEY` (provider: gemini), `ANTHROPIC_API_KEY` (provider: anthropic), or `OPENAI_API_KEY` (provider: openai).
* To use a custom/self-hosted LLM endpoint, set `ANTHROPIC_BASE_URL` (for provider: anthropic) or `OPENAI_BASE_URL` (for provider: openai) — e.g. for Ollama or vLLM.
* If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.

讨论

基于 GitHub Discussions。登录 GitHub 即可参与讨论、点赞、订阅更新。