rule quality evaluator

Security Community

Skill notes

Two-Phase Overview

Phase 1 — Static Critic (always): Read rules, score against Seven Properties, detect structural issues, produce scorecard. No agent execution. Phase 2 — Behavioral Test (opt-in): Generate coding tasks, derive assertions, hand off to context-eval, synthesize report. Interview-only fallback: If no instruction file accessible, ask user to paste rules. Flag at end: "I can't verify codebase alignment — code pattern checks need manual verification."

Step 1: Ingest

Read instruction file(s). Detect format (Markdown prose, structured rules list, YAML front matter, fenced sections). Parse into individual rules — one rule = one discrete behavioral instruction. Split on bullets, numbered items, or paragraph breaks starting an imperative. Optionally scan codebase for linter configs (.eslintrc, pyproject.toml, ruff.toml), type configs (tsconfig.json), CI configs (.github/workflows/). Enables redundancy detection.

Step 2: Score Each Rule — The Seven Properties

Score 0–7 by counting properties satisfied (each binary). Property · ID · Score 0 · Score 1 Specific and falsifiable · P1 · "Write clean code" · "Every API handler must return Result<T, AppError>, never throw" Encodes WHY · P2 · "Don't use console.log" · "Don't use console.log — use src/lib/logger.ts. Bypasses correlation IDs, breaks Datadog traces" Born from real failure · P3 · "Handle errors carefully" · "Never add indexes to reservations without DBA approval — compound index locked table 47 min in Q2 2024"

Step 3: Structural Analysis

Redundancy: Flag rules duplicating linter (ESLint, Ruff, Pylint), type system (TypeScript strict, mypy), or CI (test gates, build gates). Redundant rules waste budget and dilute signal. Scope: "Does this rule apply to ALL code agent will see in this directory?" Flag: Over-scoped: package-specific rule in root file Under-scoped: identical rules duplicated across subdirs that should move up Coverage — map to nine categories:

Step 4: Produce Scorecard

Output, then ask whether to run Phase 2. Rule Scorecard — [file path] Rule (truncated) · Score · Weakest Property · Flags "…" · N/7 · P? · weak / noise / redundant / over-scoped Summary Stats

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Security
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@tomevault-io · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

01
Phase 1 — Static Critic

Phase 1 — Static Critic

02
Step 1: Ingest

Read instruction file(s). Detect format (Markdown prose, structured rules list, YAML front matter, fenced sections). Parse into individual rules — one rule = one discrete behavioral instruction. Split on bullets, numbered items, or paragraph breaks starting an imperative. Optionally scan codebase for linter configs (`.eslintrc`, `pyproject.toml`, `ruff.toml`), type configs (`tsconfig.json`), CI configs (`.github/workflows/`). Enables redundancy detection.

03
Step 2: Score Each Rule — The Seven Properties

Score 0–7 by counting properties satisfied (each binary). Property · ID · Score 0 · Score 1 Specific and falsifiable · P1 · "Write clean code" · "Every API handler must return `Result<T, AppError>`, never throw" Encodes WHY · P2 · "Don't use console.log" · "Don't use console.log — use `src/lib/logger.ts`. Bypasses correlation IDs, breaks Datadog traces" Born from real failure · P3 · "Handle errors carefully" · "Never add indexes to `reservations` without DBA approval — compound index locked table 47 min in Q2 2024"

04
Step 3: Structural Analysis

Redundancy: Flag rules duplicating linter (ESLint, Ruff, Pylint), type system (TypeScript strict, mypy), or CI (test gates, build gates). Redundant rules waste budget and dilute signal. Scope: "Does this rule apply to ALL code agent will see in this directory?" Flag: Over-scoped: package-specific rule in root file Under-scoped: identical rules duplicated across subdirs that should move up Coverage — map to nine categories:

05
Step 4: Produce Scorecard

Output, then ask whether to run Phase 2. Rule Scorecard — `[file path]` Rule (truncated) · Score · Weakest Property · Flags "…" · N/7 · P? · weak / noise / redundant / over-scoped Summary Stats

06
Phase 2 — Behavioral Test

Phase 2 — Behavioral Test

07
Step 5: Generate Coding Tasks

Generate 3 coding tasks. Each: ~50 lines of code to produce Touches 2+ rules (discriminating grading) Prioritize: rules scoring 3-5 (uncertain), C7 Security or C2 Domain (high-stakes), incident-born rules (P3 = 1) Per task: which rules exercised, what correct behavior looks like.

08
Step 6: Generate Assertions

One testable assertion per rule: Observable in agent output (no execution required) Pass/fail check Write to `evals/evals.json`:

09
Step 7: Hand Off to context-eval

Harness under test = instruction file. Baseline = same tasks without the file. context-eval runs with/without harness, grades, computes benefit delta.

10
Step 8: Synthesize Report

Combined Report Phase 1 Static: [mean] / 7 → [STRONG / ADEQUATE / WEAK] Phase 2 Behavioral Delta: +[delta] pass rate over baseline Per-Rule Behavioral Confirmation Rule · P1 Verdict · P2 Delta · Confirmed?

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.