skillgrade-setup

Name: skillgrade-setup
Author: mgechev

Writing Community

Repo stars 490
Author updated Live
Author repo skillgrade

Fluxly profile Facts only: domain, agents, trust score, runtime, permissions and network

Domain: Writing
Compatible agents: Claude Code
Cursor
Cline
Codex
Windsurf
Gemini CLI

+20
Trust score: 88 / 100 · community maintained
Author / version / license: @mgechev · no license declared
Token usage: Lean
Setup complexity: Manual integration
External API key: Required · OpenAI / Anthropic / Gemini
Operating systems: Docker
Runtime requirements: Node.js · Docker
Permissions: Read-only
Write / modify
Env read
Network behavior: External requests
Install commands: 26 variants

Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.

Heads up: 未限定 allowed-tools，默认拥有全部工具权限。

Output preview skillgrade-setup.preview

---
name: skillgrade-setup
description: Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval co…
category: writing
runtime: Node.js / Docker
---

# skillgrade-setup output preview

## PART A: Task fit
- Use case: Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval configurations, running trials, reviewing results, or integrating with CI. Don't use for writing grader scripts, general test authoring, or non-agentic documentation..
- Inputs: target material, constraints, expected output, and acceptance criteria.
- Evidence boundary: follow “Procedures / Error Handling” and do not present inference as author intent.

## PART B: Execution result
- **01** The card summarizes the use case; runtime output centers on “Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval configurations, running trials, reviewing results, or integrating with CI. Don't use for writing grader scripts, general test authoring, or non-agentic documentation.”.
- **02** When the source has headings, the agent prioritizes “Procedures / Error Handling” so the result follows the author’s structure.
- **03** Typical output includes task judgment, concrete steps, required commands or file edits, validation, and follow-up options.
- **04** Risk context follows the fingerprint: read files, write/modify files, read environment variables; may access external network resources; requires OpenAI / Anthropic / Gemini API keys.

## Running Rules
- read files, write/modify files, read environment variables; may access external network resources; requires OpenAI / Anthropic / Gemini API keys.
- Validate with a small sample before expanding scope.
- Return the result, validation criteria, and next iteration options.

SKILL.md SKILL.md

---
name: skillgrade-setup
description: Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval co…
category: writing
source: mgechev/skillgrade
---

# skillgrade-setup

## When to use
- Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval configurations, running…
- Use it when the task has clear inputs, repeatable steps, and validation criteria.

## What to provide
- Target material, scope, expected result, and forbidden changes.
- Whether network, commands, file writes, or external services are allowed.

## Execution rules
- Organize steps around “Procedures / Error Handling” and keep inference separate from source facts.
- read files, write/modify files, read environment variables; may access external network resources; requires OpenAI / Anthropic / Gemini API keys.
- Validate with a small sample before expanding the task.

## Output requirements
- Return the deliverable, key evidence, validation method, and next action.
- Mark missing information as unknown; do not invent commands, platforms, or dependencies.

Evidence boundary and execution chain

The author source anchors workflow facts; repository files anchor sources and commands; Fluxly only adds fit, limitations, and quality judgment.

skill "skillgrade-setup" {
  input      -> user goal + target files + boundaries + acceptance criteria
  context    -> Procedures / Error Handling
  rules      -> SKILL.md triggers / order / output contract
  runtime    -> Node.js / Docker | read files, write/modify files, read environment variables | may access external network resources
  guardrails -> requires OpenAI / Anthropic / Gemini API keys + small-sample validation + diff/log review
  output     -> copyable result + checklist + next iteration
}

Interpretation is structured for decision-making; original keeps the upstream SKILL.md unchanged.

Decide Fit First

Core job: Sets up and runs skillgrade evaluation pipelines for Agent Skills. Use when initializing eval configurations, running trials, re…
Best fit: Use it when the task has reusable inputs, steps, and validation criteria rather than a one-off answer.
Avoid forcing it: If the source lacks commands, platform support, or external-service evidence, keep those fields unknown instead of guessing.

Design Intent

Structure: The skill is organized around “Procedures”, “Error Handling”, showing how the author expects the agent to judge fit, collect context, and produce verifiable output.
Trigger evidence: Prioritize the author’s wording around when to use it, what context to collect, and what output shape to produce.
Evidence boundary: Author text states facts, repository files prove commands and paths, and Fluxly only adds fit, limits, and usage judgment.

How To Use It

Inputs: Provide target material, scope, expected result, forbidden changes, and validation method.
Invocation: Name skillgrade-setup directly; if the source includes slash commands, start with the command and then add task context.
Validation: Start small and check whether the result follows “Procedures / Error Handling” before expanding.

Boundaries And Review

Dependencies: Prepare OpenAI / Anthropic / Gemini API keys before running a full task.
Permissions: Declared permissions include read / write / env-read; ask the agent to state file, command, and rollback boundaries before acting.
Quality bar: A useful result names the deliverable, evidence, and next action. Generic prose means the task needs tighter context.

Skillgrade Evaluation Setup

Procedures

Step 1: Install Skillgrade

Verify Node.js 20+ and Docker are available.
Run npm i -g skillgrade to install the CLI globally.

Step 2: Initialize an Eval Configuration

Navigate to the skill directory (must contain a SKILL.md).
Set the appropriate API key environment variable (GEMINI_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY).
Run skillgrade init to generate an eval.yaml with AI-powered tasks and graders.
If an eval.yaml already exists, pass --force to overwrite: skillgrade init --force.
Without an API key, a well-commented template is generated instead.

Step 3: Configure eval.yaml

Read references/eval-yaml-spec.md for the full configuration schema.
Define one or more tasks under the tasks: key. Each task requires:
- name: unique task identifier
- instruction: what the agent should accomplish
- workspace: files to copy into the evaluation container
- graders: one or more scoring mechanisms (see the skillgrade-graders skill)
Optionally configure defaults: for agent, provider, trials, timeout, and threshold.

Step 4: Run Evaluations

Select an appropriate preset based on the evaluation goal:
- --smoke (5 trials): Quick capability check.
- --reliable (15 trials): Reliable pass rate estimate.
- --regression (30 trials): High-confidence regression detection.
Run the evaluation: skillgrade --smoke.
Run a specific eval by name: skillgrade --eval=fix-linting.
Run multiple evals: skillgrade --eval=fix-linting,write-tests.
Run only deterministic graders (skip LLM calls): skillgrade --grader=deterministic.
Run only LLM rubric graders: skillgrade --grader=llm_rubric.
The agent is auto-detected from the API key. Override with --agent=gemini|claude|codex.
Override the provider with --provider=docker|local.

Step 5: Review Results

Run skillgrade preview for a CLI report.
Run skillgrade preview browser to open the web UI at http://localhost:3847.
Reports are saved to $TMPDIR/skillgrade/<skill-name>/results/. Override with --output=DIR.

Step 6: Integrate with CI

Add a GitHub Actions step that installs skillgrade, navigates to the skill directory, and runs with --regression --ci --provider=local.
Use --provider=local in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.
The --ci flag causes a non-zero exit code if the pass rate falls below --threshold (default: 0.8).
Read references/ci-example.md for a complete workflow template.

Error Handling

If skillgrade init fails with "No SKILL.md found," verify the current directory contains a valid SKILL.md file.
If evaluation hangs, check Docker is running and the container has network access for API calls.
If all trials fail with "No API key," ensure the environment variable is exported, not just set inline for a different command.

Install command

One-click installPick a runner, click Copy, paste into your terminal, press enter

npx skills add mgechev/skillgrade/skills/skillgrade-setup -y

Open general install guide

View skill source

Works with

Claude Code
Cursor
Cline
Codex
Windsurf
Gemini
OpenCode
Roo Code
AAide
+Augment
GitHub Copilot
OpenClaw
KKiro
Junie
WWarp
Goose
Trae
Kilo Code
OpenHands
Amp
Zencoder
CContinue
AIAider
CodeBuddy
CodeGeeX
Replit Agent

Security audit

88 / 100

Community

未声明 license/许可。
未声明 version。
未限定 allowed-tools，默认拥有全部工具权限。

Metadata

Source: github
mgechev/skillgrade
中文

More by @mgechev

A skillgrade-graders 88

Category & tags

Category Writing