agent-evaluation-framework
- Repo stars 25,042
- Author updated Live
- Author repo v8
- Domain
- Design
- Compatible agents
-
- Claude Code
- Cursor
- Cline
- Codex
- Windsurf
- Gemini CLI
- +20
- Trust score
- 88 / 100 · community maintained
- Author / version / license
- @v8 · no license declared
- Token usage
- Lean
- Setup complexity
- Guided setup
- External API key
- Not required
- Operating systems
- Unspecified (assume cross-platform)
- Runtime requirements
- No special requirements
- Permissions
-
- Read-only
- Write / modify
- Shell exec
- Network behavior
- Local-only
- Install commands
- 26 variants
Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.
Heads up: 未限定 allowed-tools，默认拥有全部工具权限。
---
name: agent-evaluation-framework
description: Workflow for evaluating and refining agent debugging capabilities using designated test cases an…
category: design
runtime: no special runtime
---
# agent-evaluation-framework output preview
## PART A: Task fit
- Use case: Workflow for evaluating and refining agent debugging capabilities using designated test cases and Swarm principles. Use when evaluating subagent performance or creating benchmarks. Do not use for regular bug fixing..
- Inputs: target material, constraints, expected output, and acceptance criteria.
- Evidence boundary: follow “0. Preparation / 1. Core Directives / 2. Agent Orchestration & Lifecycle Management” and do not present inference as author intent.
## PART B: Execution result
- **01** The card summarizes the use case; runtime output centers on “Workflow for evaluating and refining agent debugging capabilities using designated test cases and Swarm principles. Use when evaluating subagent performance or creating benchmarks. Do not use for regular bug fixing.”.
- **02** When the source has headings, the agent prioritizes “0. Preparation / 1. Core Directives / 2. Agent Orchestration & Lifecycle Management” so the result follows the author’s structure.
- **03** Typical output includes task judgment, concrete steps, required commands or file edits, validation, and follow-up options.
- **04** Risk context follows the fingerprint: read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
## Running Rules
- read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding scope.
- Return the result, validation criteria, and next iteration options.The source does not require a stable slash command. After installation, invoke the skill by name and describe the task.
Name target files or source material, expected output, forbidden changes, and whether network or shell access is allowed. Permission fingerprint: read files, write/modify files, run shell commands.
Start with a small task and check whether the result follows “0. Preparation / 1. Core Directives / 2. Agent Orchestration & Lifecycle Management”. Inspect diffs, logs, previews, or tests before expanding scope.
Confirm the final output includes a concrete result, evidence, and next action. If it stays generic, tighten inputs, boundaries, and acceptance criteria.
---
name: agent-evaluation-framework
description: Workflow for evaluating and refining agent debugging capabilities using designated test cases an…
category: design
source: v8/v8
---
# agent-evaluation-framework
## When to use
- Workflow for evaluating and refining agent debugging capabilities using designated test cases and Swarm principles. Us…
- Use it when the task has clear inputs, repeatable steps, and validation criteria.
## What to provide
- Target material, scope, expected result, and forbidden changes.
- Whether network, commands, file writes, or external services are allowed.
## Execution rules
- Organize steps around “0. Preparation / 1. Core Directives / 2. Agent Orchestration & Lifecycle Management” and keep inference separate from source facts.
- read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding the task.
## Output requirements
- Return the deliverable, key evidence, validation method, and next action.
- Mark missing information as unknown; do not invent commands, platforms, or dependencies.The author source anchors workflow facts; repository files anchor sources and commands; Fluxly only adds fit, limitations, and quality judgment.
skill "agent-evaluation-framework" {
input -> user goal + target files + boundaries + acceptance criteria
context -> 0. Preparation / 1. Core Directives / 2. Agent Orchestration & Lifecycle Management
rules -> SKILL.md triggers / order / output contract
runtime -> no special runtime | read files, write/modify files, run shell commands | mostly runs locally
guardrails -> usually needs no extra API key + small-sample validation + diff/log review
output -> copyable result + checklist + next iteration
}Agent Evaluation Framework Workflow
Use this skill to orchestrate evaluation sessions for subagents, identify procedural bottlenecks, and iteratively refine system prompts and capabilities utilizing Swarm intelligence principles.
0. Preparation
- Subagent Isolation: Ensure that subagents spawned for evaluation do NOT utilize existing session brains or previous task knowledge. This is critical to maintain the integrity of meta-testing.
- Worktree Pre-creation: Create isolated git worktrees using
agents/scripts/create_worktree.sh <task_id>for each test case beforehand. Report where the worktrees were created to the user. Inside worktrees, builds MUST use thetools/dev/gm.pytool INSIDE the worktree.gm.pywill automatically runsetup_worktree_build.pyto prepare the symlinks; manual execution ofsetup_worktree_build.pyis not required. - Test Injection: Copy the target test case into the worktree (e.g.,
test/mjsunit/repro.js). - Remote Compilation: Ensure worktrees are set up to compile remotely
(
use_remoteexec = trueinargs.gn) before proceeding.
1. Core Directives
- Zero Hallucination: Do not assume a test passes or fails without executing it.
- Worktree Enforcement: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists.
- Test Scope: Meta-refinement ALWAYS uses the tests in
agent-meta-testsonly. - Test Immutability: The
agent-meta-testsdirectory cannot be changed. - Crash Verification: Only work on test-cases that still crash.
- Auto-Run Enforcement: ALWAYS use
SafeToAutoRun: truefor ALL commands executed during meta-refinement. Approval must NEVER be asked of the user. - Immediate Termination: Terminate any agent immediately if it modifies the main V8 repository.
2. Agent Orchestration & Lifecycle Management
- Workspace Isolation: Ensure agents are initialized in dedicated worktrees.
- Communication Routing: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work.
- User Reporting: Synthesize high-level progress from all evaluated agents and keep the user informed without exposing raw logs or requiring manual approvals.
3. Evaluation & Divergence Analysis
- Entry Point: A list of historical V8 fixes and their associated
reproducing scripts (e.g., from
test/mjsunit/or Buganizer). - Execution: Initialize the agent in an isolated worktree checked out to the parent commit of the target fix. Copy the repro script and command the agent to resolve the bug.
- Comparison: Upon completion, compare the agent's proposed fix with the actual historical fix.
- Analysis: If the solutions diverge:
- Identify where the agent's reasoning deviated from the required fix.
- Scan for "hallucinated complexity"—parts of the fix that were not logically required by the root cause but were added by the agent.
- Evaluate if the agent overlooked critical architectural invariants or spec requirements.
- Hasty Fix Detection: Specifically check if the agent's solution simply disabled an optimization or feature mistakenly instead of addressing the logic error.
- Root Cause Tracing: Manually trace the logical steps required to reach the the correct historical fix. Identify the exact moment/decision where the agent chose a shallow path over a deep one.
4. Iterative Process Refinement & Skepticism
The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth:
Architectural Skepticism: Require subagents to explicitly argue against a proposed fix before accepting it. Look at the problem from multiple orthogonal angles.
Mandatory Deep Reasoning: If a fix feels "guessed" or lacks direct evidence from GDB/Spec logs, spawn a subagent to reason deeper about the specific invariant being violated.
Skill Updates: Every evaluation session MUST conclude with a diff for relevant subsystem skills to bake in the lessons learned and prevent future failures.
analyze_brain.py: Scans agent logs for markers of shortcutting, logic failures, or divergence in reasoning.
Decide Fit First
Design Intent
How To Use It
Boundaries And Review