skill-tester-ci

AI Community

Repo stars 180
Author repo pskoett-ai-skills

Interpretation is structured for decision-making; original keeps the upstream SKILL.md unchanged.

Decide Fit First

Core job: Validates all CI skills in this repo. Checks Agent Skills spec compliance, gh-aw workflow compilation, permission correctness, a…
Best fit: Use it when the task has reusable inputs, steps, and validation criteria rather than a one-off answer.
Avoid forcing it: If the source lacks commands, platform support, or external-service evidence, keep those fields unknown instead of guessing.

Design Intent

Structure: The skill is organized around “When to Use”, “Prerequisites”, “Checks”, “1. Anthropic Spec Validation”, showing how the author expects the agent to judge fit, collect context, and produce verifiable output.
Trigger evidence: Prioritize the author’s wording around when to use it, what context to collect, and what output shape to produce.
Evidence boundary: Author text states facts, repository files prove commands and paths, and Fluxly only adds fit, limits, and usage judgment.

How To Use It

Inputs: Provide target material, scope, expected result, forbidden changes, and validation method.
Invocation: Name skill-tester-ci directly; if the source includes slash commands, start with the command and then add task context.
Validation: Start small and check whether the result follows “When to Use / Prerequisites / Checks” before expanding.

Boundaries And Review

Dependencies: It usually needs no extra API key, so start with a small validation task.
Permissions: Declared permissions include read / write / shell-exec; ask the agent to state file, command, and rollback boundaries before acting.
Quality bar: A useful result names the deliverable, evidence, and next action. Generic prose means the task needs tighter context.

Skill Tester CI

Validates all CI skills (*-ci) in this repo. Runs spec validation, compiles gh-aw workflows, and checks CI-specific conventions.

When to Use

After adding or modifying a CI skill
After upgrading gh-aw to a new version (API changes may break workflows)
Before committing CI skill changes
Before submitting the plugin for Anthropic review

Prerequisites

gh-aw CLI installed (gh extension install github/gh-aw)
python3 with pyyaml installed
quick_validate.py available at .claude/skills/skill-creator/scripts/

Checks

1. Anthropic Spec Validation

Run quick_validate.py on every CI skill:

for d in skills/*-ci/; do
  python3 .claude/skills/skill-creator/scripts/quick_validate.py "$d"
done

2. Workflow Example Compilation

Extract workflow blocks from each CI skill's references/workflow-example.md and compile with gh aw compile:

# Extract markdown code blocks with frontmatter
# Copy to temp .github/workflows/
# Run: gh aw compile

Pass criteria: Zero compilation errors. Warnings are reported but don't fail.

3. Permission Checks

CI workflows in strict mode must NOT use write permissions directly. Verify:

No issues: write (use safe-outputs: add-comment instead)
No pull-requests: write (use safe-outputs: create-pull-request-review-comment instead)
No contents: write unless the workflow creates files (eval-creator create mode)
Required read permissions present for declared toolsets

4. Structural Checks

Check	Rule	Severity
Has `references/workflow-example.md`	Required for all CI skills	Error
Workflow example has frontmatter block	At least one ```markdown block with `---`	Error
Name matches folder	Frontmatter `name` == directory name	Error
Description mentions gh-aw	CI skills should reference gh-aw	Warning
Has corresponding interactive skill	`foo-ci` should have a `foo` counterpart	Warning

5. Cross-Workflow Validation

For workflows that use call-workflow:

Verify the target workflow exists and has workflow_call in its on: section
Verify the target's inputs match what the caller provides

Output Format

## CI Skill Test Results

**Date:** YYYY-MM-DD
**gh-aw version:** vX.Y.Z
**Skills tested:** N
**Passed:** N
**Compile errors:** N
**Spec failures:** N

### Compilation Results
- [skill-name]: ✓ compiled (N KB) | ✗ error: [message]

### Spec Results
- [skill-name]: ✓ valid | ✗ [error]

### Permission Issues
- [skill-name]: [issue]

Running

Invoke manually:

/skill-tester-ci

Or run the script directly:

bash skills/skill-tester-ci/scripts/run-tests.sh

What This Skill Does NOT Do

Does not test interactive skills (use skill-tester for those)
Does not execute workflows against real repos — only compiles them
Does not modify skills — reports findings only
Does not test workflow runtime behavior — compilation validates structure and permissions only

Fluxly profile Author and license come from source; runtime, permissions, and network are Fluxly detections or estimates

Fluxly category: AI
Author-declared agents: No explicit declaration found; this is not inferred or tested compatibility
Static check: 88 / 100 · heuristic scan, not runtime safety proof
Author / version / license: @pskoett · no license declared
Fluxly token estimate: Lean
Fluxly setup estimate: Guided setup
External API key: No requirement detected
Detected OS requirements: Unspecified
Runtime requirements: Unspecified
Detected file/system behavior: Read-only
Write / modify
Shell exec
Detected network behavior: Local-only
Install commands: None (reference only)

Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.

Heads up: 未限定 allowed-tools，默认拥有全部工具权限。

Output preview skill-tester-ci.preview

# 2. Workflow Example Compilation

# Extract markdown code blocks with frontmatter
# Copy to temp .github/workflows/
# Run: gh aw compile

SKILL.md SKILL.md

# Skill Tester CI

Validates all CI skills (`*-ci`) in this repo. Runs spec validation, compiles gh-aw workflows, and checks CI-specific conventions.

## When to Use

- After adding or modifying a CI skill
- After upgrading `gh-aw` to a new version (API changes may break workflows)
- Before committing CI skill changes
- Before submitting the plugin for Anthropic review

## Prerequisites

- `gh-aw` CLI installed (`gh extension install github/gh-aw`)
- `python3` with `pyyaml` installed
- `quick_validate.py` available at `.claude/skills/skill-creator/scripts/`

## Checks

### 1. Anthropic Spec Validation

Run `quick_validate.py` on every CI skill:

```bash
for d in skills/*-ci/; do
  python3 .claude/skills/skill-creator/scripts/quick_validate.py "$d"
done
```

### 2. Workflow Example Compilation

Extract workflow blocks from each CI skill's `references/workflow-example.md` and compile with `gh aw compile`:

```bash
# Extract markdown code blocks with frontmatter
# Copy to temp .github/workflows/
# Run: gh aw compile
```

**Pass criteria:** Zero compilation errors. Warnings are reported but don't fail.

### 3. Permission Checks

CI workflows in strict mode must NOT use write permissions directly. Verify:

- No `issues: write` (use `safe-outputs: add-comment` instead)
- No `pull-requests: write` (use `safe-outputs: create-pull-request-review-comment` instead)
- No `contents: write` unless the workflow creates files (eval-creator create mode)
- Required read permissions present for declared toolsets

### 4. Structural Checks

| Check | Rule | Severity |
|-------|------|----------|
| Has `references/workflow-example.md` | Required for all CI skills | Error |
| Workflow example has frontmatter block | At least one `` ```markdown `` block with `---` | Error |

…

Evidence boundary and execution chain

Author text anchors workflow facts; Fluxly only indexes current sections, terms, files, and commands.

sections -> When to Use → Prerequisites → Checks → 1. Anthropic Spec Validation → 2. Workflow Example Compilation → 3. Permission Checks
terms    -> Pass criteria · Date · gh-aw version · Skills tested · Passed · Compile errors · Spec failures
files/cmd -> -ci · gh-aw · gh extension install github/gh-aw · python3 · pyyaml · quickvalidate.py · .claude/skills/skill-creator/scripts/ · references/workflow-example.md
body sha256 -> 6bbde45e9d07