技能审核员
- 作者仓库星标 792
- 许可证 MIT
- 作者仓库 medical-research-skills
- 领域
- 运维部署
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 94 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @aipoch · MIT
- 上下文用量评级
- 较高消耗
- 接入复杂程度
- 需手动接入
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- macOS
- 运行环境要求
- Node.js · Python
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 允许外网请求
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
第 1 步:技能否决——结构性红线❌
硬门。任何失败=立即拒绝。不要继续执行步骤 2。阅读目标技能的`SKILL.md` 和任何捆绑的脚本。检查所有四个维度: → 完整标准:`references/basic_veto.md` 维度·立即拒绝触发器 T1。运行稳定性·故障率>20%;随机崩溃或无限循环;无法解决的依赖冲突需要手动干预
第 2 步:基本评估 — 25 个标准(ISO 25010 + OpenSSF + Shneiderman + Agent)
阅读该技能的完整 SKILL.md 和所有捆绑文件。对 25 个标准中的每一个进行评分(从 0 到 4)。 → 包含每个级别描述的完整标题: `references/basic_evaluation.md` · 类别(框架) · 标准 · 最大 1 · 功能适用性 (ISO 25010) · 完整性、正确性、适当性 · 12 2 · 可靠性 (ISO 25010) · 容错性、错误报告、可恢复性 · 12
第 4 步:动态输入生成
目的:生成直接源自技能自身描述的测试输入,以确保它们反映真实世界的使用模式。
第 5 步:执行测试
使用检测到的执行模式在 N 个测试输入中的每一个上运行技能。
第6步:多层输出评估
评估三个并行层的所有 N 个输出。
第 7 步:人工审核 — Eval Viewer
生成结构化 Markdown 审查文档以供人工检查。输出位置。将评估查看器保存在已审核技能旁边,使其与它所描述的工件保持一致:其中 `<audited_skill_path>` 是包含技能的 `SKILL.md` 的目录(用户传入的路径)。如果该目录不可写,请返回到`/tmp/eval_viewer_<skill_name>.md`并在最终报告中记下后备。
# Skill Auditor
This skill provides a standardized, end-to-end process for auditing any agent skill — from structural integrity to live functional performance. It combines two independent veto gates, static analysis across 25 criteria, dynamic execution, five-category specialized scoring, and human review into a single coherent workflow.
## Audit Pipeline
```
Step 1 │ Skill Veto → ❌ HARD GATE: Structural/security redlines — any FAIL = reject
Step 2 │ Basic Evaluation → Static quality scoring (25 criteria, 100 pts, ISO 25010 + OpenSSF + Agent)
Step 3 │ Classification → Route to one of 5 categories + detect execution mode
Step 4 │ Dynamic Input Gen → Generate N test inputs scaled to complexity
Step 5 │ Execution Testing → Run skill via correct execution mode
Step 6 │ Multi-Layer Evaluation → Basic rubric + Specialized rubric (category-specific, /60) + Assertions
│ ❌ HARD GATE: Research Veto — any FAIL = reject (categories 1–4 only)
Step 7 │ Human Review → Generate eval viewer (.md) + collect per-input scores for JSON
Step 8 │ Optimization Report → Final score + P0/P1/P2 recommendations
│ + emit eval_report_<n>_result.json for frontend visualization
```
**Two hard rejection gates run at Steps 1 and 6. Both are mandatory and cannot be skipped.**
All other steps run sequentially. **Step 9 always runs** — even when a veto gate fires, the polished output corrects the rejected skill rather than abandoning it.
---
## Language Policy
**All audit output must be written in English, regardless of the language used in the user's request or in the submitted skill.**
This applies to every artifact produced by this skill:
- Veto reports (Step 1 and Step 6)
- Static evaluation scores and notes (Step 2)
- Generated test inputs (Step 4)
- Execution summaries and per-output evaluations (Steps 5–6)
- The eval viewer `.md` file (Step 7)
- The final optimization report and JSON (Step 8)
If the user communicates in another language, Claude may briefly acknowledge the request in that language, but must then conduct and present the full audit in English.
---
## Step 1: Skill Veto — Structural Redlines ❌
> **HARD GATE. Any FAIL = immediate rejection. Do not proceed to Step 2.**
Read the target skill's `SKILL.md` and any bundled scripts. Check all four dimensions:
→ Full criteria: [`references/basic_veto.md`](references/basic_veto.md)
| Dimension | Immediate Rejection Triggers |
|---|---|
| **T1. Operational Stability** | Failure rate > 20%; random crashes or infinite loops; unresolvable dependency conflicts requiring manual intervention |
| **T2. Structural Consistency** | Missing required frontmatter fields (`name`, `description`); non-compliant schema; inconsistent return types or field names |
| **T3. Result Determinism** | Significant output variance on identical inputs at low temperature; no seed management; critical numerical results fluctuate randomly |
| **T4. System Security** | Direct execution of raw user-provided strings (`eval`/`exec`); no input filtering; prompt injection vectors present in scripts or instructions |
**► If any T1–T4 dimension is FAIL: stop immediately. Output the rejection report below and do not continue.**
```
SKILL VETO — REJECTED
══════════════════════════════════
Skill: <n>
Reason: Failed structural redline check
T1. Stability : PASS / FAIL — <reason>
T2. Contract : PASS / FAIL — <reason>
T3. Determinism : PASS / FAIL — <reason>
T4. Security : PASS / FAIL — <reason>
This skill must not be deployed. Fix all FAIL dimensions before resubmitting.
══════════════════════════════════
```
---
## Step 2: Basic Evaluation — 25 Criteria (ISO 25010 + OpenSSF + Shneiderman + Agent)
Read the skill's full SKILL.md and all bundled files. Score each of the 25 criteria from 0–4.
→ Full rubric with per-level descriptions: [`references/basic_evaluation.md`](references/basic_evaluation.md)
| # | Category (Framework) | Criteria | Max |
|---|---|---|---|
| 1 | **Functional Suitability** (ISO 25010) | Completeness, Correctness, Appropriateness | 12 |
| 2 | **Reliability** (ISO 25010) | Fault Tolerance, Error Reporting, Recoverability | 12 |
| 3 | **Performance & Context** (ISO 25010 + Agent) | Token Cost, Execution Efficiency | 8 |
| 4 | **Agent Usability** (Shneiderman · Gerhardt-Powals) | Learnability, Consistency, Feedback Design, Error Prevention | 16 |
| 5 | **Human Usability** (Tognazzini · Norman) | Discoverability, Forgiveness | 8 |
| 6 | **Security** (ISO 25010 + OpenSSF) | Credential Safety, Input Validation, Data Safety | 12 |
| 7 | **Maintainability** (ISO 25010) | Modularity, Modifiability, Testability | 12 |
| 8 | **Agent-Specific** (Novel) | Trigger Precision, Progressive Disclosure, Composability, Idempotency, Escape Hatches | 20 |
**Basic subtotal: __ / 100**
---
## Step 3: Classification + Execution Mode Detection
### 3.1 Classify the Skill
Read the skill's `description` frontmatter and `## When to Use` section.
→ Full category definitions: [`references/classification.md`](references/classification.md)
| # | Category | Typical Skills |
|---|---|---|
| 1 | **Evidence Insight** | Search strategy builders, database scouts, critical appraisal tools, evidence synthesizers |
| 2 | **Protocol Design** | Experimental design generators, study-type advisors, statistical power planners, validation strategists |
| 3 | **Data Analysis** | R/Python code generators, bioinformatics pipelines, statistical modeling tools, ML workflows |
| 4 | **Academic Writing** | SCI manuscript writers, abstract generators, methods/discussion drafters, cover-letter tools |
| 5 | **Other (General / Non-Research)** | All skills that do not fall into categories 1–4 |
> **Research Veto scope**: Categories 1–4 are subject to the Research Veto hard gate in Step 6. Category 5 is exempt.
### 3.2 Detect Execution Mode
Inspect the skill to determine how it is meant to be invoked.
| Mode | Indicators | How to Run in Step 5 |
|---|---|---|
| **A: Direct** | Only SKILL.md instructions, no scripts | Follow SKILL.md instructions to complete the task as Claude |
| **B: CLI / Script** | `scripts/` directory with Python/bash, CLI examples in SKILL.md | Execute via bash: `python scripts/xxx.py <args>` |
| **C: API** | API endpoint patterns, fetch/curl usage in SKILL.md | Simulate or call the API as documented |
| **D: Hybrid** | Both instructions and scripts/API | Run script for deterministic parts; Claude for reasoning/generation parts |
Record the detected mode. It will be used in Step 5.
---
## Step 4: Dynamic Input Generation
**Purpose:** Generate test inputs derived directly from the skill's own description to ensure they reflect real-world usage patterns.
### 4.1 Assess Skill Complexity
| Complexity Level | Criteria | Test Input Count (N) |
|---|---|---|
| **Simple** | Single task type, narrow scope, < 3 reference files, no branching workflow | **3 inputs** |
| **Moderate** | 2–3 task types, some branching, 3–5 reference files, moderate scope | **5 inputs** |
| **Complex** | Multiple task types, branching logic, 5+ reference files, broad or specialized scope | **7 inputs** |
Declare: `Complexity: [Simple / Moderate / Complex] → Generating N inputs`
### 4.2 Generate N Test Inputs
Use this distribution based on N:
| Slot | Type | Always Include? |
|---|---|---|
| Input 1 | Canonical / happy path | ✅ Always |
| Input 2 | Variant A (different valid use case) | ✅ Always |
| Input 3 | Edge / boundary | ✅ Always |
| Input 4 | Variant B (third central use case) | If N ≥ 5 |
| Input 5 | Stress / complex / multi-part | If N ≥ 5 |
| Input 6 | Scope boundary (slightly outside) | If N = 7 |
| Input 7 | Adversarial / ambiguous | If N = 7 |
Format each as a realistic user message. Do **not** include expected answers.
**Output:**
```
GENERATED TEST INPUTS
═══════════════════════════════════════
Skill: <n> | Category: <1–5 + label> | Mode: <A/B/C/D>
Complexity: <level> → Generating <N> inputs
Input 1 (Canonical) : <prompt>
Input 2 (Variant A) : <prompt>
Input 3 (Edge) : <prompt>
[Input 4–7 if applicable]
═══════════════════════════════════════
```
---
## Step 5: Execution Testing
Run the skill on each of the N test inputs using the detected execution mode.
### Mode A: Direct Execution
Load the skill's SKILL.md. Follow its instructions as if you are Claude-with-this-skill responding to the user message. Complete the task in full.
### Mode B: CLI / Script Execution
```bash
python scripts/<script_name>.py "<input_text_or_path>"
```
Capture stdout/stderr. If execution fails, record the error and continue.
### Mode C: API Execution
Follow the API usage pattern documented in the skill. Construct the request, execute, capture response. If credentials are unavailable, note this and simulate the expected output based on documentation.
### Mode D: Hybrid Execution
Run the script/API component first. Pass its output to Claude for the reasoning/generation component.
### Execution Log
For each input:
```
─── Input [N] ─────────────────────────
Mode : <A/B/C/D>
Input : <prompt>
Output :
<full output>
Status : COMPLETED / ERROR / PARTIAL
Notes : <anomalies, scope violations, unexpected behaviors>
────────────────────────────────────────
```
---
## Step 6: Multi-Layer Output Evaluation
Evaluate all N outputs across three parallel layers.
### Layer 1: Basic Rubric Scoring
→ Full criteria: [`references/basic_evaluation.md`](references/basic_evaluation.md)
For each output, score four aggregate dimensions (0–10 each):
- **Functional Correctness** (did output complete the task correctly and fully?): /10
- **Reliability & Clarity** (well-structured, consistent, clear feedback?): /10
- **Efficiency** (concise, no padding, no unnecessary context bloat?): /10
- **Scope & Safety** (stayed in scope, no harmful content, proper escape hatches?): /10
Per-output basic score: /40
### Layer 2: Specialized Rubric Scoring
Apply the rubric corresponding to the category from Step 3:
| Category | Reference File | Max |
|---|---|---|
| 1 — Evidence Insight | [`references/specialized_evaluation_literature.md`](references/specialized_evaluation_literature.md) | 60 |
| 2 — Protocol Design | [`references/specialized_evaluation_research_design.md`](references/specialized_evaluation_research_design.md) | 60 |
| 3 — Data Analysis | [`references/specialized_evaluation_data_analysis.md`](references/specialized_evaluation_data_analysis.md) | 60 |
| 4 — Academic Writing | [`references/specialized_evaluation_academic_writing.md`](references/specialized_evaluation_academic_writing.md) | 60 |
| 5 — Other | [`references/specialized_evaluation_other.md`](references/specialized_evaluation_other.md) | 60 |
Per-output specialized score: /60
### Layer 3: Assertion Checks
For each output, write and evaluate 3–5 boolean assertions derived from the skill's stated purpose.
**Assertion types:**
- **Format assertion**: "Output contains required section X"
- **Content assertion**: "Output does not fabricate specific claims"
- **Scope assertion**: "Output does not go beyond the skill's stated purpose"
- **Safety assertion**: "Output includes required disclaimers" (required for categories 1–4)
- **Completeness assertion**: "Output addresses all parts of a multi-part prompt"
Format:
```
Assertions for Input [N]:
[PASS/FAIL] <assertion statement> — <brief justification>
[PASS/FAIL] <assertion statement> — <brief justification>
[PASS/FAIL] <assertion statement> — <brief justification>
Assertion pass rate: X/Y
```
**GATE: Any safety or scope assertion FAIL on 2+ outputs → Flag for rejection review.**
### Scoring Aggregation
```
Per-output total = Basic Score (Layer 1, /40) + Specialized Score (Layer 2, /60) = /100
Execution avg score = Mean of all N output totals
Assertion summary = Total passed / Total assertions
```
---
### Research Veto — Scientific Integrity Redlines ❌ *(Categories 1–4 only)*
> **HARD GATE. Run after all N outputs are evaluated. Any FAIL = immediate rejection.**
> **Skip entirely for Category 5 (Other).**
→ Full criteria and rejection report template: [`references/scientific_veto.md`](references/scientific_veto.md)
| Dimension | Immediate Rejection Triggers |
|---|---|
| **M1. Scientific Integrity** | Any output fabricates DOI/PMID numbers, invents clinical trial results, sample sizes, p-values, or efficacy data that cannot be verified |
| **M2. Practice Boundaries** | Any output makes direct diagnostic or prescriptive medical conclusions; any output lacks required medical disclaimer; any output recommends unapproved treatments without explicit caveats |
| **M3. Methodological Baseline** | Any output commits a principled methodological fallacy; any output ignores or fails to warn about ethical compliance requirements |
| **M4. Code Usability** | Any generated bioinformatics/statistical code is unrunnable (syntax errors, infinite loops, missing core dependencies). Mark N/A for categories 1 & 4 if no code is generated. |
**► If any M1–M4 dimension is FAIL: stop. Output the Research Veto rejection report. Do not proceed to Steps 7–8.**
---
## Step 7: Human Review — Eval Viewer
Generate a structured Markdown review document for human inspection.
```markdown
# Eval Viewer — <Skill Name>
Generated: <date>
## Summary Table
| Input | Type | Basic /40 | Specialized /60 | Total /100 | Assertions | Status |
|---|---|---|---|---|---|---|
| 1 | Canonical | __ | __ | __ | X/Y PASS | ✅/⚠️/❌ |
...
**Execution Average: __ / 100**
**Assertion Pass Rate: __/__**
## Detailed Outputs
### Input 1 — [Type]
**Prompt:** <input text>
**Output:** <full output>
**Scores:** Basic: __/40 | Specialized: __/60 | Total: __/100
**Assertions:**
- [PASS/FAIL] <assertion statement> — <brief justification>
- ...
```
**Output location.** Save the eval viewer next to the audited skill so it stays with the artifact it describes:
```
<audited_skill_path>/eval_viewer_<skill_name>.md
```
Where `<audited_skill_path>` is the directory containing the skill's `SKILL.md` (the path the user passed in). If that directory is not writable, fall back to `/tmp/eval_viewer_<skill_name>.md` and note the fallback in the final report.
If no filesystem is available at all, render the viewer inline in the conversation.
> **Note for reviewer:** Check ⚠️ and ❌ rows first. Patterns across 2+ outputs indicate structural skill issues.
---
## Step 8: Optimization Report
### Final Score Calculation
```
Final Score = (Static Score × 0.4) + (Execution Avg × 0.6)
```
→ Full scoring thresholds: [`references/scoring_rubric.md`](references/scoring_rubric.md)
| Score | Grade | Recommendation |
|---|---|---|
| 85–100 | ⭐ Production Ready | Deploy publicly |
| 75–84 | ✅ Limited Release | Throttled / monitored rollout |
| 60–74 | ⚠️ Beta Only | Internal / greylist only |
| < 60 | ❌ Reject | Do not deploy |
### Optimization Recommendations
| Priority | Criteria | Action |
|---|---|---|
| **P0 — Blocker** | Any veto FAIL, safety assertion FAIL, score < 60 | Must fix before any deployment |
| **P1 — Major** | Score 60–74, repeated assertion failures, Layer 1 or 2 avg < 7/10 | Fix before production release |
| **P2 — Minor** | Score 75–84, isolated output weaknesses, style/format issues | Address before full scale |
For each issue found, output:
```
[P0/P1/P2] <Issue Title>
Observed in: Input(s) [N, M, ...]
Problem: <what went wrong>
Root cause: <likely cause in skill design>
Fix: <specific, actionable change to SKILL.md or scripts>
```
### Final Report
```
══════════════════════════════════════════════════
SKILL AUDIT REPORT
══════════════════════════════════════════════════
Skill Name : <n>
Category : <label>
Execution Mode : <A/B/C/D>
Complexity : <Simple/Moderate/Complex> (N=<n> inputs)
Audited On : <date>
── STEP 1: Structural Veto ───────────────────────
Stability : PASS / FAIL
Contract : PASS / FAIL
Determinism : PASS / FAIL
Security : PASS / FAIL
── STEP 2: Static Evaluation (25 criteria) ───────
Functional Suitability : __/12
Reliability : __/12
Performance/Context : __/8
Agent Usability : __/16
Human Usability : __/8
Security : __/12
Maintainability : __/12
Agent-Specific : __/20
Static Subtotal : __/100
── STEP 3: Classification ────────────────────────
Category : <label>
Execution Mode : <A / B / C / D>
── STEP 4: Test Inputs ───────────────────────────
[N inputs listed with type labels]
── STEP 5: Execution Summary ─────────────────────
Input 1: [COMPLETED/PARTIAL/ERROR] — <note>
...
── STEP 6: Output Evaluation ─────────────────────
Basic Specialized Total Assertions
Input 1: __/40 __/60 __/100 X/Y PASS
...
Execution Avg : __/100
Total Assertion Pass Rate : __/__
[Research Veto — Evidence Insight / Protocol Design / Data Analysis / Academic Writing only; N/A for Other]
Scientific Integrity : PASS / FAIL / N/A
Practice Boundaries : PASS / FAIL / N/A
Methodological Ground : PASS / FAIL / N/A
Code Usability : PASS / FAIL / N/A
── STEP 7: Outputs ───────────────────────────────
<audited_skill_path>/eval_viewer_<n>.md : SAVED ✅
<audited_skill_path>/eval_report_<n>_result.json : SAVED ✅
── STEP 8: Final Score ───────────────────────────
Static Score : __/100 × 40% = __
Dynamic Score : __/100 × 60% = __
FINAL SCORE : __ / 100
GRADE : ⭐/✅/⚠️/❌ [Production Ready / Limited Release / Beta Only / Reject]
Key Strengths:
- ...
Optimization Recommendations:
[P0] ...
[P1] ...
[P2] ...
══════════════════════════════════════════════════
```
### Step 7–8 JSON Output
> ⚠️ **STRICT SCHEMA COMPLIANCE — NOT OPTIONAL.**
> The JSON report is consumed by downstream tooling (skill-evaluator, the frontend viewer, automated regression dashboards). Any deviation from the schema below — a renamed key, a missing field, a string where an object is expected — breaks that tooling silently. **Before emitting the JSON, you MUST:**
>
> 1. Re-read [`references/report_json_schema.md`](references/report_json_schema.md) in full (not just the summary table below).
> 2. Build the JSON by copying the structure from the schema's complete example, then filling in values — do not invent your own key names or nesting.
> 3. Run every item in the schema's Pre-Emit Checklist (§ "Pre-Emit Checklist" in `report_json_schema.md`) before writing the file. Treat each unchecked item as a blocker, not a warning.
>
> Common drift to watch for — these have been observed in past audits and each one breaks the contract:
> - Using `static_score.total` instead of `static_score.subtotal`.
> - Using `dynamic_score.execution_avg_score` instead of `dynamic_score.execution_avg`.
> - Writing `veto_gates.research_veto.scientific_integrity: "PASS"` (a bare string) instead of `{ "result": "PASS", "detail": "..." }`.
> - Omitting the top-level `gate` field inside `skill_veto` and `research_veto`.
> - Per-input keys: `n` instead of `index`, missing `label` / `status` / `status_flag` / `note`.
> - Each static category value being a bare integer instead of `{ "score": N, "max": M, "note": "..." }`.
> - `final.final_score` / `final.weighted_static` / `final.weighted_dynamic` instead of `final.score` / `final.static_weighted` / `final.dynamic_weighted`.
> - Missing `final.grade_symbol`.
>
> If a section of the schema's example does not match what you are about to emit, the example is canonical — change your output, not the schema.
→ Full schema + complete example: [`references/report_json_schema.md`](references/report_json_schema.md)
**Output location.** Save the JSON next to the audited skill, using the same directory rule as the eval viewer:
```
<audited_skill_path>/eval_report_<skill_name>_result.json
```
If the audited skill directory already contains a prior `eval_report_<skill_name>_result.json`, **overwrite** it — the latest audit supersedes earlier ones. If that directory is not writable, fall back to `/tmp/eval_report_<skill_name>_result.json` and note the fallback in the final report.
**JSON top-level nodes (all 7 required) — abbreviated reminder; the schema file is canonical:**
| Node | Key rules |
|---|---|
| `meta` | `evaluator_version: "skill-auditor@1.0"`; includes `skill_name`, `description`, `evaluated_on`, `category`, `execution_mode`, `complexity`, `n_inputs` |
| `veto_gates` | `skill_veto`: top-level `gate` + `stability`, `contract`, `determinism`, `security` — **no T-prefixes**. `research_veto`: top-level `applicable`, `gate` + four dimensions, each as `{ result, detail }` — **no M-prefixes** |
| `static_score` | `subtotal`, `max`, `categories`. The 8 category keys must be exact and un-prefixed: `functional_suitability`, `reliability`, `performance_context`, `agent_usability`, `human_usability`, `security`, `maintainability`, `agent_specific`. **Each value is an object `{ score, max, note }`, never a bare integer.** |
| `dynamic_score` | `execution_avg` (not `execution_avg_score`), `max`, `assertion_pass_rate: { passed, total }`, `inputs[]`. Each input uses `index` (not `n`) and includes `label`, `status`, `status_flag`, `note`, `basic`, `specialized`, `total`, `assertions_passed`, `assertions_total`, and a full `assertions[]` array of `{ text, result, note }` objects. |
| `final` | `static_weighted`, `dynamic_weighted`, `score` (not `final_score`), `max`, `grade`, `grade_symbol`, `deployable`, `veto_override` |
| `key_strengths` | plain-string array, 2–5 entries |
| `recommendations` | P0 → P1 → P2 sorted, each with `priority`, `title`, `observed_in`, `problem`, `root_cause`, `fix` |
**Pre-emit checklist (MUST verify all of these before writing the file — see full list in `report_json_schema.md` § Pre-Emit Checklist):**
- All free-text fields written in **English**
- No T-, M-, or cat-numbered prefixes in any JSON key
- `static_score.categories` has exactly **8** keys; each value is `{ score, max, note }`; all scores within 0–max
- `static_score.subtotal` equals the sum of all 8 category `score` values
- `dynamic_score.inputs` has exactly **N** objects (matching `meta.n_inputs`); each has an `assertions` array
- **Each input's `assertions` array has 3–5 entries** *(cardinality constraint — count it)*
- Each input's `assertions_passed` equals count of `"PASS"` in its `assertions` array
- Each input's `basic + specialized = total`
- **`key_strengths` has 2–5 entries** *(cardinality constraint — count it)*
- `research_veto.applicable = false` and all research veto fields `= "N/A"` for category Other
- `final.veto_override = true` if any gate is FAIL; `final.deployable = false` in that case
- `final.grade` and `final.grade_symbol` consistent with `final.score` per the threshold table
- `recommendations` sorted P0 → P1 → P2 (empty array allowed)
- All averages/weighted scores rounded to **1 decimal**; all raw scores are **integers**
---
## Input Validation
This skill accepts: a SKILL.md file or skill description submitted for quality audit and improvement.
If the user's request does not involve auditing, evaluating, scoring, or improving an agent skill — for example, asking to write a story, build a website, or answer a general question — do not proceed with the audit pipeline. Instead respond:
> "Skill Auditor is designed to evaluate and improve agent skills (SKILL.md files). Your request appears to be outside this scope. Please submit a skill for auditing, or use a more appropriate tool for your task."
**Language note:** The user may submit requests or skills in any language. Always produce the full audit output in English. See [Language Policy](#language-policy) above.
---
## Reference Files
| File | Used In | Gate? |
|---|---|---|
| `references/basic_veto.md` | Step 1 — Structural redlines | ❌ Hard gate |
| `references/basic_evaluation.md` | Step 2 (static scoring) + Step 6 Layer 1 | — |
| `references/classification.md` | Step 3 — 5-category classification | — |
| `references/specialized_evaluation_literature.md` | Step 6 Layer 2 — Category 1 | — |
| `references/specialized_evaluation_research_design.md` | Step 6 Layer 2 — Category 2 | — |
| `references/specialized_evaluation_data_analysis.md` | Step 6 Layer 2 — Category 3 | — |
| `references/specialized_evaluation_academic_writing.md` | Step 6 Layer 2 — Category 4 | — |
| `references/specialized_evaluation_other.md` | Step 6 Layer 2 — Category 5 | — |
| `references/scientific_veto.md` | Step 6 Research Veto — categories 1–4 only | ❌ Hard gate |
| `references/scoring_rubric.md` | Step 8 — final score & deployment recommendation | — |
| `references/report_json_schema.md` | Step 7 (data collection) + Step 8 (JSON output) | — |
## Dependencies
- Python 3.x + standard libraries — for `scripts/evaluate_skill.py` (structural pre-checks)
- Claude — for Steps 4–8 (input generation, execution, multi-layer scoring, recommendations)
---
## Changelog
### v1.1.0 — 2026-04-02
**Scene Override additions** — Based on findings from the audit of `differential-expression-analysis`, three systematic biases were identified in `basic_evaluation.md` when applied to scientific computing + agent-first skills. Rather than modifying the shared basic evaluation criteria (which would affect all five categories), per-category scene override sections were added to the relevant specialized evaluation files.
**Files modified:**
- `references/specialized_evaluation_data_analysis.md` — Added Scene Override section covering Fault Tolerance (2.1), Forgiveness (5.2), and Recoverability (2.3)
- `references/specialized_evaluation_research_design.md` — Added Scene Override section with the same three overrides, adapted for protocol design context
- `references/specialized_evaluation_other.md` — Added Execution Mode Awareness note directing auditors to apply Category 3 overrides when the skill operates in agent-first Mode B/C/D context
**Rationale:** The three affected basic evaluation criteria assume (1) human direct CLI operation and (2) general-purpose software tools. These assumptions do not hold for scientific computing pipelines or agent-first skills, where strict input validation and hard stops are correct design decisions, and structured error codes are the appropriate recovery interface.
章节与文件索引
按说明文件中的顺序排列。
审核管道
语言政策
第 1 步:技能否决——结构性红线❌
第 2 步:基本评估 — 25 个标准(ISO 25010 + OpenSSF + Shneiderman + Agent)
3.1 技能分类
文件/命令 -> .md · references/basic_veto.md · name · description · eval · exec · references/basic_evaluation.md · ## When to Use
内容校验值 -> 3a252f0873cd
当前原文的自动摘译,尚未人工审校。
审核管道
第 1 步和第 6 步运行两个硬拒绝门。两者都是强制性的,不能跳过。所有其他步骤都按顺序运行。第 9 步始终运行——即使否决门启动,经过精心设计的输出也会纠正被拒绝的技能,而不是放弃它。
语言政策
所有审核输出都必须以英语编写,无论用户请求或提交的技能中使用何种语言。这适用于该技能生成的每个工件: 否决报告(步骤 1 和步骤 6) 静态评估分数和注释(步骤 2) 生成的测试输入(步骤 4)
3.1 技能分类
阅读该技能的
description前言和## When to Use部分。 → 完整类别定义:references/classification.md· 类别 · 典型技能 1 · 证据洞察力 · 搜索策略构建器、数据库侦察、关键评估工具、证据合成器 2 · 方案设计 · 实验设计生成器、研究类型顾问、统计功率规划器、验证策略师