skillgrade graders

Writing Community

Skill notes

Procedures

Step 1: Identify the Grading Strategy Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric). For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3). Step 2: Write a Deterministic Grader Create a script in the skill's graders/ directory (bash or TypeScript).

Error Handling

If a deterministic grader outputs non-JSON, ensure all echo/console.log statements except the final JSON result are redirected to stderr. If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: GEMINI_API_KEY (provider: gemini), ANTHROPIC_API_KEY (provider: anthropic), or OPENAI_API_KEY (provider: openai). To use a custom/self-hosted LLM endpoint, set ANTHROPIC_BASE_URL (for provider: anthropic) or OPENAI_BASE_URL (for provider: openai) — e.g. for Ollama or vLLM.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Writing
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@mgechev · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Required · OpenAI / Anthropic / Gemini
Operating systems
macOS · Linux · Windows
Runtime requirements
Node.js
Detected file/system behavior
  • Read-only
  • Write / modify
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Author source file SKILL.md

# Skillgrade Grader Authoring

## Procedures

**Step 1: Identify the Grading Strategy**
1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).

**Step 2: Write a Deterministic Grader**
1. Create a script in the skill's `graders/` directory (bash or TypeScript).
2. The script must output a JSON object to stdout with the following structure:
   ```json
   {"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]}
   ```
3. `score` (0.0–1.0) and `details` are required. `checks` is optional but recommended.
4. Read `references/grader-output-schema.md` for the full output specification.
5. Use `awk` for arithmetic in bash scripts — `bc` is not available in `node:20-slim`.
6. Reference the grader in eval.yaml:
   ```yaml
   - type: deterministic
     run: bash graders/check.sh
     weight: 0.7
   ```

**Step 3: Write an LLM Rubric Grader**
1. Draft a rubric with explicit scoring criteria and point allocations.
2. Structure the rubric into weighted sections that sum to 1.0:
   ```
   Workflow Compliance (0-0.5):
   - Did the agent follow the mandatory workflow steps?
   Efficiency (0-0.5):
   - Completed in ≤5 commands without trial-and-error?
   ```
3. Reference the rubric in eval.yaml:
   ```yaml
   - type: llm_rubric
     rubric: |
       [rubric text or file path]
     weight: 0.3
     provider: gemini               # optional: gemini (default) | anthropic | openai
     model: gemini-3-flash-preview  # optional, each provider has a default model
   ```
4. For long rubrics, store in a separate file and reference by path: `rubric: rubrics/quality.md`.

**Step 4: Combine Multiple Graders**
1. Assign weights to each grader based on importance. Weights are normalized automatically.
2. Final reward is calculated as: `Σ (grader_score × weight) / Σ weight`.
3. Example configuration:
   ```yaml
   graders:
     - type: deterministic
       run: bash graders/check.sh
       weight: 0.7
     - type: llm_rubric
       rubric: rubrics/quality.md
       weight: 0.3
   ```

**Step 5: Validate Graders**
1. Create a reference solution script that produces the expected output.
2. Run `skillgrade --validate` to verify graders score the reference solution correctly.
3. Test only deterministic graders: `skillgrade --grader=deterministic` (skips LLM calls, faster iteration).
4. Test only LLM rubric graders: `skillgrade --grader=llm_rubric`.
5. Run a specific eval with a specific grader type: `skillgrade --eval=my-eval --grader=deterministic`.
6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.

## Error Handling
* If a deterministic grader outputs non-JSON, ensure all `echo`/`console.log` statements except the final JSON result are redirected to stderr.
* If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: `GEMINI_API_KEY` (provider: gemini), `ANTHROPIC_API_KEY` (provider: anthropic), or `OPENAI_API_KEY` (provider: openai).
* To use a custom/self-hosted LLM endpoint, set `ANTHROPIC_BASE_URL` (for provider: anthropic) or `OPENAI_BASE_URL` (for provider: openai) — e.g. for Ollama or vLLM.
* If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.