skillgrade graders
- Repo stars 490
- Author repo skillgrade
- Domain
- Writing
- Author-declared agents
- No explicit declaration found; this is not inferred or tested compatibility
- Static check
- 88 / 100 · heuristic scan, not runtime safety proof
- Runtime evidence
- Not run; no isolated runtime test evidence is currently recorded
- Author / version / license
- @mgechev · no license declared
- Token usage
- Lean
- Setup complexity
- Guided setup
- External API key
- Required · OpenAI / Anthropic / Gemini
- Operating systems
- macOS · Linux · Windows
- Runtime requirements
- Node.js
- Detected file/system behavior
-
- Read-only
- Write / modify
- Network behavior
- External requests
- Install commands
- None (reference only)
Heads up: Tool permissions are not declared; actual access depends on the runtime.
# Skillgrade Grader Authoring
## Procedures
**Step 1: Identify the Grading Strategy**
1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).
**Step 2: Write a Deterministic Grader**
1. Create a script in the skill's `graders/` directory (bash or TypeScript).
2. The script must output a JSON object to stdout with the following structure:
```json
{"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]}
```
3. `score` (0.0–1.0) and `details` are required. `checks` is optional but recommended.
4. Read `references/grader-output-schema.md` for the full output specification.
5. Use `awk` for arithmetic in bash scripts — `bc` is not available in `node:20-slim`.
6. Reference the grader in eval.yaml:
```yaml
- type: deterministic
run: bash graders/check.sh
weight: 0.7
```
**Step 3: Write an LLM Rubric Grader**
1. Draft a rubric with explicit scoring criteria and point allocations.
2. Structure the rubric into weighted sections that sum to 1.0:
```
Workflow Compliance (0-0.5):
- Did the agent follow the mandatory workflow steps?
Efficiency (0-0.5):
- Completed in ≤5 commands without trial-and-error?
```
3. Reference the rubric in eval.yaml:
```yaml
- type: llm_rubric
rubric: |
[rubric text or file path]
weight: 0.3
provider: gemini # optional: gemini (default) | anthropic | openai
model: gemini-3-flash-preview # optional, each provider has a default model
```
4. For long rubrics, store in a separate file and reference by path: `rubric: rubrics/quality.md`.
**Step 4: Combine Multiple Graders**
1. Assign weights to each grader based on importance. Weights are normalized automatically.
2. Final reward is calculated as: `Σ (grader_score × weight) / Σ weight`.
3. Example configuration:
```yaml
graders:
- type: deterministic
run: bash graders/check.sh
weight: 0.7
- type: llm_rubric
rubric: rubrics/quality.md
weight: 0.3
```
**Step 5: Validate Graders**
1. Create a reference solution script that produces the expected output.
2. Run `skillgrade --validate` to verify graders score the reference solution correctly.
3. Test only deterministic graders: `skillgrade --grader=deterministic` (skips LLM calls, faster iteration).
4. Test only LLM rubric graders: `skillgrade --grader=llm_rubric`.
5. Run a specific eval with a specific grader type: `skillgrade --eval=my-eval --grader=deterministic`.
6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.
## Error Handling
* If a deterministic grader outputs non-JSON, ensure all `echo`/`console.log` statements except the final JSON result are redirected to stderr.
* If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: `GEMINI_API_KEY` (provider: gemini), `ANTHROPIC_API_KEY` (provider: anthropic), or `OPENAI_API_KEY` (provider: openai).
* To use a custom/self-hosted LLM endpoint, set `ANTHROPIC_BASE_URL` (for provider: anthropic) or `OPENAI_BASE_URL` (for provider: openai) — e.g. for Ollama or vLLM.
* If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.
Sections and files
Listed in source order.
sections -> Procedures → Error Handling
terms -> Step 1: Identify the Grading Strategy · Step 2: Write a Deterministic Grader · Step 3: Write an LLM Rubric Grader · Step 4: Combine Multiple Graders · Step 5: Validate Graders
files/cmd -> graders/ · score · details · checks · references/grader-output-schema.md · awk · node:20-slim · rubric: rubrics/quality.md
body sha256 -> 03f65f0912f8
Procedures
Step 1: Identify the Grading Strategy Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric). For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3). Step 2: Write a Deterministic Grader Create a script in the skill's
graders/directory (bash or TypeScript).Error Handling
If a deterministic grader outputs non-JSON, ensure all
echo/console.logstatements except the final JSON result are redirected to stderr. If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider:GEMINI_API_KEY(provider: gemini),ANTHROPIC_API_KEY(provider: anthropic), orOPENAI_API_KEY(provider: openai). To use a custom/self-hosted LLM endpoint, setANTHROPIC_BASE_URL(for provider: anthropic) orOPENAI_BASE_URL(for provider: openai) — e.g. for Ollama or vLLM.