skills-eval

Security Community

Repo stars 289
Author repo claude-night-market

Interpretation is structured for decision-making; original keeps the upstream SKILL.md unchanged.

Decide Fit First

Core job: Evaluate Claude skill quality through auditing. Use when reviewing or auditing skills. This framework audits Claude skills again…
Best fit: Use it when the task has reusable inputs, steps, and validation criteria rather than a one-off answer.
Avoid forcing it: If the source lacks commands, platform support, or external-service evidence, keep those fields unknown instead of guessing.

Design Intent

Structure: The skill is organized around “Table of Contents”, “Overview”, “Quick Start”, “Basic Audit”, showing how the author expects the agent to judge fit, collect context, and produce verifiable output.
Trigger evidence: Prioritize the author’s wording around when to use it, what context to collect, and what output shape to produce.
Evidence boundary: Author text states facts, repository files prove commands and paths, and Fluxly only adds fit, limits, and usage judgment.

How To Use It

Inputs: Provide target material, scope, expected result, forbidden changes, and validation method.
Invocation: Name skills-eval directly; if the source includes slash commands, start with the command and then add task context.
Validation: Start small and check whether the result follows “Table of Contents / Overview / Quick Start” before expanding.

Boundaries And Review

Dependencies: It usually needs no extra API key, so start with a small validation task.
Permissions: Declared permissions include read / write / shell-exec; ask the agent to state file, command, and rollback boundaries before acting.
Quality bar: A useful result names the deliverable, evidence, and next action. Generic prose means the task needs tighter context.

Skills Evaluation and Improvement

Overview
Quick Start
Evaluation Workflow
Evaluation and Optimization
Resources

Overview

This framework audits Claude skills against quality standards to improve performance and reduce token consumption. Automated tools analyze skill structure, measure context usage, and identify specific technical improvements. Run verification commands after each audit to confirm fixes work correctly.

The skills-auditor provides structural analysis, while the improvement-suggester ranks fixes by impact. Compliance is verified through the compliance-checker. Runtime efficiency is monitored by tool-performance-analyzer and token-usage-tracker.

Quick Start

Basic Audit

Run a full audit of all skills or target a specific file to identify structural issues.

# Audit all skills
make audit-all

# Audit specific skill
make audit-skill TARGET=path/to/skill/SKILL.md

Analysis and Optimization

Use skill_analyzer.py for complexity checks and token_estimator.py to verify the context budget.

make analyze-skill TARGET=path/to/skill/SKILL.md
make estimate-tokens TARGET=path/to/skill/SKILL.md

Improvements

Generate a prioritized plan and verify standards compliance using improvement_suggester.py and compliance_checker.py.

make improve-skill TARGET=path/to/skill/SKILL.md
make check-compliance TARGET=path/to/skill/SKILL.md

Evaluation Workflow

Start with make audit-all to inventory skills and identify high-priority targets. For each skill requiring attention, run analysis with analyze-skill to map complexity. Generate an improvement plan, apply fixes, and run check-compliance to verify the skill meets project standards. Finalize by checking the token budget for efficiency.

Evaluation and Optimization

Quality assessments use the skills-auditor and improvement-suggester to generate detailed reports. Performance analysis focuses on token efficiency through the token-usage-tracker and tool performance via tool-performance-analyzer. For standards compliance, the compliance-checker automates common fixes for structural issues.

Scoring and Prioritization

We evaluate skills across five dimensions: structure compliance, content quality, token efficiency, activation reliability, and tool integration. Scores above 90 represent production-ready skills, while scores below 50 indicate critical issues requiring immediate attention.

Improvements are prioritized by impact. Critical issues include security vulnerabilities or broken functionality. High-priority items cover structural flaws that hinder discoverability. Medium and low priorities focus on best practices and minor optimizations.

Structural Patterns

Deprecated: skills/shared/modules/ directories. Shared modules must be relocated into the consuming skill's own modules/ directory. The evaluator flags any remaining skills/shared/ as a structural warning.

Current: Each skill owns its modules at skills/<skill-name>/modules/. Cross-skill references use relative paths (e.g., ../skill-authoring/modules/anti-rationalization.md).

Resources

Shared Modules: Cross-Skill Patterns

Anti-Rationalization Patterns: See anti-rationalization.md
Enforcement Language: See enforcement-language.md
Trigger Patterns: See trigger-patterns.md

Skill-Specific Modules

Trigger Isolation Analysis: See modules/trigger-isolation-analysis.md
Authoring Checklist: See modules/authoring-checklist.md
Evaluation Workflows: See modules/evaluation-workflows.md
Advanced Tool Use Analysis: See modules/advanced-tool-use-analysis.md
Evaluation Framework: See modules/evaluation-framework.md
Integration Patterns: See modules/integration.md
Troubleshooting: See modules/troubleshooting.md
Pressure Testing: See modules/pressure-testing.md
Integration Testing: See modules/integration-testing.md
Performance Benchmarking: See modules/performance-benchmarking.md

Tools and Automation

Tools: Executable analysis utilities in scripts/ directory.
Automation: Setup and validation scripts in scripts/automation/.

Fluxly profile Author and license come from source; runtime, permissions, and network are Fluxly detections or estimates

Fluxly category: Security
Author-declared agents: No explicit declaration found; this is not inferred or tested compatibility
Static check: 88 / 100 · heuristic scan, not runtime safety proof
Author / version / license: @athola · no license declared
Fluxly token estimate: Lean
Fluxly setup estimate: Guided setup
External API key: No requirement detected
Detected OS requirements: Unspecified
Runtime requirements: Unspecified
Detected file/system behavior: Read-only
Write / modify
Shell exec
Detected network behavior: Local-only
Install commands: None (reference only)

Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.

Heads up: 未限定 allowed-tools，默认拥有全部工具权限。

Output preview skills-eval.preview

The current SKILL.md does not define a fixed output example.

SKILL.md SKILL.md

# Skills Evaluation and Improvement

## Table of Contents

1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [Evaluation Workflow](#evaluation-workflow)
4. [Evaluation and Optimization](#evaluation-and-optimization)
5. [Resources](#resources)

## Overview

This framework audits Claude skills against quality standards to improve performance and reduce token consumption. Automated tools analyze skill structure, measure context usage, and identify specific technical improvements. Run verification commands after each audit to confirm fixes work correctly.

The `skills-auditor` provides structural analysis, while the `improvement-suggester` ranks fixes by impact. Compliance is verified through the `compliance-checker`. Runtime efficiency is monitored by `tool-performance-analyzer` and `token-usage-tracker`.

## Quick Start

### Basic Audit
Run a full audit of all skills or target a specific file to identify structural issues.
```bash
# Audit all skills
make audit-all

# Audit specific skill
make audit-skill TARGET=path/to/skill/SKILL.md
```

### Analysis and Optimization
Use `skill_analyzer.py` for complexity checks and `token_estimator.py` to verify the context budget.
```bash
make analyze-skill TARGET=path/to/skill/SKILL.md
make estimate-tokens TARGET=path/to/skill/SKILL.md
```

### Improvements
Generate a prioritized plan and verify standards compliance using `improvement_suggester.py` and `compliance_checker.py`.
```bash
make improve-skill TARGET=path/to/skill/SKILL.md
make check-compliance TARGET=path/to/skill/SKILL.md
```

## Evaluation Workflow

…

Evidence boundary and execution chain

Author text anchors workflow facts; Fluxly only indexes current sections, terms, files, and commands.

sections -> Table of Contents → Overview → Quick Start → Basic Audit → Analysis and Optimization → Improvements
terms    -> Deprecated · Current · Anti-Rationalization Patterns · Enforcement Language · Trigger Patterns · Trigger Isolation Analysis · Authoring Checklist · Evaluation Workflows
files/cmd -> skills-auditor · improvement-suggester · compliance-checker · tool-performance-analyzer · token-usage-tracker · skillanalyzer.py · tokenestimator.py · improvementsuggester.py
body sha256 -> 962852d54eae