规则质量评估器
- 作者仓库星标 0
- 作者仓库 skills-registry
- 领域
- 安全
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @tomevault-io · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- 未检出明确要求
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
阅读说明文件。检测格式(Markdown 散文、结构化规则列表、YAML 标题、围栏部分)。解析为单独的规则——一条规则=一条离散的行为指令。根据项目符号、编号项目或开始祈使句的段落分隔符进行拆分。可以选择扫描代码库以查找 linter 配置 (`.eslintrc`、`pyproject.toml`、`ruff.toml`)、类型配置 (`tsconfig.json`)、CI 配置 (`.github/workflows/`)。启用冗余检测。
通过计算满足的属性(每个二进制)来得分 0-7。属性 · ID · 得分 0 · 得分 1 具体且可证伪 · P1 · “编写干净的代码” · “每个 API 处理程序都必须返回 `Result<T, AppError>`,绝不抛出” 编码原因 · P2 · “不要使用 console.log” · “不要使用 console.log — 使用 `src/lib/logger.ts`。绕过相关 ID,破坏 Datadog 跟踪” 源于真正的失败 · P3 · “小心处理错误” · “未经 DBA 批准,切勿向 `reservations` 添加索引 - 2024 年第 2 季度复合索引锁定表 47 分钟”
冗余:标记重复 linter(ESLint、Ruff、Pylint)、类型系统(TypeScript strict、mypy)或 CI(测试门、构建门)的规则。冗余规则浪费预算并淡化信号。范围:“此规则是否适用于代理在此目录中看到的所有代码?”标志:范围过大:根文件中的包特定规则范围不足:子目录之间重复的相同规则应向上移动覆盖范围 - 映射到九个类别:
输出,然后询问是否运行阶段 2。 规则记分卡 — `[file path]` 规则(截断) · 得分 · 最弱属性 · 标志“…” · N/7 · P? · 弱/噪音/冗余/范围过大的摘要统计
生成 3 个编码任务。每个:大约 50 行代码来生成 2+ 条规则(区分评分) 优先级:规则评分 3-5(不确定)、C7 安全或 C2 域(高风险)、事件产生的规则 (P3 = 1) 每个任务:执行了哪些规则,正确的行为是什么样的。
每条规则一个可测试断言:可在代理输出中观察到(无需执行) 通过/失败检查 写入`evals/evals.json`:
被测线束 = 说明文件。基线 = 没有文件的相同任务。 context-eval 在有/没有安全带的情况下运行、评分、计算效益增量。
综合报告 第 1 阶段静态:[平均] / 7 → [强/足够/弱] 第 2 阶段行为增量:超过基线的+δ通过率 每规则行为确认规则 · P1 判决 · P2 增量 · 已确认?
# Rule Quality Evaluator
A rule that cannot be violated cannot be followed. Most instruction files fail because of unfalsifiable guidance, redundant noise, and implicit knowledge that never made it to paper. Score rules against seven properties, identify structural weaknesses, optionally verify with live tasks.
## Two-Phase Overview
- **Phase 1 — Static Critic** (always): Read rules, score against Seven Properties, detect structural issues, produce scorecard. No agent execution.
- **Phase 2 — Behavioral Test** (opt-in): Generate coding tasks, derive assertions, hand off to context-eval, synthesize report.
**Interview-only fallback**: If no instruction file accessible, ask user to paste rules. Flag at end: "I can't verify codebase alignment — code pattern checks need manual verification."
---
## Phase 1 — Static Critic
### Step 1: Ingest
Read instruction file(s). Detect format (Markdown prose, structured rules list, YAML front matter, fenced sections). Parse into individual rules — one rule = one discrete behavioral instruction. Split on bullets, numbered items, or paragraph breaks starting an imperative.
Optionally scan codebase for linter configs (`.eslintrc`, `pyproject.toml`, `ruff.toml`), type configs (`tsconfig.json`), CI configs (`.github/workflows/`). Enables redundancy detection.
### Step 2: Score Each Rule — The Seven Properties
Score 0–7 by counting properties satisfied (each binary).
| Property | ID | Score 0 | Score 1 |
|---|---|---|---|
| Specific and falsifiable | P1 | "Write clean code" | "Every API handler must return `Result<T, AppError>`, never throw" |
| Encodes WHY | P2 | "Don't use console.log" | "Don't use console.log — use `src/lib/logger.ts`. Bypasses correlation IDs, breaks Datadog traces" |
| Born from real failure | P3 | "Handle errors carefully" | "Never add indexes to `reservations` without DBA approval — compound index locked table 47 min in Q2 2024" |
| Scoped to right level | P4 | Rule about billing API in root file | Same rule in `src/billing/CLAUDE.md` |
| Points to canonical example | P5 | "Follow our API patterns" | "New endpoints follow `src/api/reservations/create.ts` — handler → validation → service → response" |
| Includes anti-pattern | P6 | "Use the service layer" | "We do NOT use the repository pattern. Each service calls Prisma directly" |
| Token-efficient | P7 | "When writing tests, please make sure to always use Vitest and not Jest" | "Tests: Vitest, never Jest. Config: `vitest.config.ts`" |
**Score ≤ 2 = weak** — rewrite/remove candidate.
**P1 = 0 = noise** — agent can't verify, can't steer.
### Step 3: Structural Analysis
**Redundancy**: Flag rules duplicating linter (ESLint, Ruff, Pylint), type system (TypeScript strict, mypy), or CI (test gates, build gates). Redundant rules waste budget and dilute signal.
**Scope**: "Does this rule apply to ALL code agent will see in this directory?" Flag:
- **Over-scoped**: package-specific rule in root file
- **Under-scoped**: identical rules duplicated across subdirs that should move up
**Coverage** — map to nine categories:
- C1 Architecture & Boundaries
- C2 Domain Model & Business Rules
- C3 Conventions & Patterns
- C4 Integrations & External Dependencies
- C5 Operations & Deployment
- C6 Testing Philosophy & Strategy
- C7 Security Model
- C8 Performance Constraints
- C9 Historical Decisions & Tech Debt
Mark `●` (covered), `◐` (partial), `○` (missing).
**Token budget**:
- Claude Code: root <4,000 tokens, subdir <1,000
- Copilot: root <1,000 lines, first 4,000 chars read per code review
- Cursor / Windsurf / AGENTS.md: similar to Claude Code
### Step 4: Produce Scorecard
Output, then ask whether to run Phase 2.
---
#### Rule Scorecard — `[file path]`
| Rule (truncated) | Score | Weakest Property | Flags |
|---|---|---|---|
| "…" | N/7 | P? | weak / noise / redundant / over-scoped |
#### Summary Stats
- Total rules: N
- Mean score: X.X / 7
- Strong (5–7): N | Adequate (3–4): N | Weak (0–2): N
- Noise rules (P1 = 0): N
- Redundant with linter/CI/types: N
- Over-scoped: N | Under-scoped: N
#### Coverage Map
```
C1[●] C2[○] C3[●] C4[○] C5[◐] C6[●] C7[○] C8[○] C9[○]
● covered ◐ partial ○ missing
```
#### Token Budget
- Current: ~N tokens / N chars
- Limit: [system-appropriate]
- Headroom: [remaining / over by N]
#### Top 3 Improvements
1. [Most impactful — which rule, which property, what to add]
2. [Second]
3. [Third]
#### Phase 1 Verdict
| Threshold | Verdict |
|---|---|
| Mean ≥ 5 | **STRONG** — specific, grounded, efficient |
| Mean 3–4 | **ADEQUATE** — usable but with gaps |
| Mean < 3 | **WEAK** — won't reliably steer behavior |
---
**Run Phase 2 — Behavioral Testing?** Generates real coding tasks, measures whether rules change agent behavior. Requires context-eval.
---
## Phase 2 — Behavioral Test
### Step 5: Generate Coding Tasks
Generate 3 coding tasks. Each:
- ~50 lines of code to produce
- Touches 2+ rules (discriminating grading)
- Prioritize: rules scoring 3-5 (uncertain), C7 Security or C2 Domain (high-stakes), incident-born rules (P3 = 1)
Per task: which rules exercised, what correct behavior looks like.
### Step 6: Generate Assertions
One testable assertion per rule:
- Observable in agent output (no execution required)
- Pass/fail check
Write to `evals/evals.json`:
```json
{
"harness_name": "[instruction file name]",
"harness_type": "coding agent instruction rules",
"harness_path": "[path to instruction file]",
"evals": [
{
"id": 1,
"prompt": "[the coding task prompt]",
"expected_output": "[description of correct output under the rules]",
"files": [],
"assertions": [
"[rule → assertion, e.g., 'Uses src/lib/logger.ts, not console.log']",
"[second rule → assertion]"
]
}
]
}
```
### Step 7: Hand Off to context-eval
Harness under test = instruction file. Baseline = same tasks without the file.
```
→ context-eval: evaluate harness at [instruction file path] using evals/evals.json
```
context-eval runs with/without harness, grades, computes benefit delta.
### Step 8: Synthesize Report
---
#### Combined Report
**Phase 1 Static**: [mean] / 7 → [STRONG / ADEQUATE / WEAK]
**Phase 2 Behavioral Delta**: +[delta] pass rate over baseline
#### Per-Rule Behavioral Confirmation
| Rule | P1 Verdict | P2 Delta | Confirmed? |
|---|---|---|---|
| "…" | score/7 | +X% | yes / partial / no |
#### Final Verdict
| Condition | Verdict |
|---|---|
| Static ≥ 5 AND delta ≥ 0.25 | **STRONG** — well-formed, measurable improvement |
| Static ≥ 3 AND delta ≥ 0.10 | **ADEQUATE** — works, room to improve |
| Static < 3 OR delta < 0.10 | **WEAK** — not reliably changing behavior |
| Delta < 0 | **HARMFUL** — degrading performance |
Delta = pass-rate diff between with-rules and without-rules (0.0–1.0; 0.25 = 25 pp).
#### Discrepancies
Rules high in Phase 1 with no behavioral delta (or vice versa) — most actionable findings. High-score + no delta = either not reachable in eval tasks or not being read. Low-score + high delta = encodes something valuable scoring missed.
---
## Calibration Rules
1. **Phase 1 always first.** Scorecard catches most failures without tooling.
2. **P1 is the gatekeeper.** Unfalsifiable rules can't be improved by other properties. Fix specificity first.
3. **Redundancy is the easiest win.** Lint/type/CI duplicates can be deleted immediately.
4. **Behavioral testing is gold standard, not default.** Phase 2 needs infrastructure and time. Most rule sets benefit more from targeted Phase 1 rewrites.
5. **Score what you see.** Literal text, not charitable interpretation. Rules requiring generous reading fail in practice.
---
## Composes With
`context-gap-analyzer` → `agent-instruction-forge` → `rule-quality-evaluator` → `context-eval`
---
> Source: [AndurilCode/craftwork](https://github.com/AndurilCode/craftwork) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-05-23 -->
按说明文件中的顺序排列。
两阶段概述
第 1 步:摄取
第 2 步:对每条规则进行评分——七个属性
第三步:结构分析
第四步:制作记分卡
文件/命令 -> .eslintrc · pyproject.toml · ruff.toml · tsconfig.json · .github/workflows/ · Result<T, AppError> · src/lib/logger.ts · reservations
内容校验值 -> a53bc85f76c6
当前原文的自动摘译,尚未人工审校。
两阶段概述
第 1 阶段 — 静态批评(始终):阅读规则,根据七个属性进行评分,检测结构问题,生成记分卡。无代理执行。第 2 阶段 — 行为测试(选择加入):生成编码任务、导出断言、移交给上下文评估、综合报告。仅面试后备:如果无法访问说明文件,请要求用户粘贴规则。末尾标记:“我无法验证代码库对齐 - 代码模式检查需要手动验证。”
第 2 步:对每条规则进行评分——七个属性
通过计算满足的属性(每个二进制)来得分 0-7。属性 · ID · 得分 0 · 得分 1 具体且可证伪 · P1 · “编写干净的代码” · “每个 API 处理程序都必须返回
Result<T, AppError>,绝不抛出” 编码原因 · P2 · “不要使用 console.log” · “不要使用 console.log — 使用src/lib/logger.ts。绕过相关 ID,破坏 Datadog 跟踪” 源于真正的失败 · P3 · “小心处理错误” · “从不在未经 DBA 批准的情况下向reservations添加索引 — 复合索引在 2024 年第 2 季度锁定表 47 分钟”