agent evaluation framework
- Repo stars 25,042
- Author repo v8
- Domain
- Design
- Author-declared agents
- No explicit declaration found; this is not inferred or tested compatibility
- Static check
- 88 / 100 · heuristic scan, not runtime safety proof
- Runtime evidence
- Not run; no isolated runtime test evidence is currently recorded
- Author / version / license
- @v8 · no license declared
- Token usage
- Lean
- Setup complexity
- Guided setup
- External API key
- Not required
- Operating systems
- Unspecified (assume cross-platform)
- Runtime requirements
- No explicit requirements detected
- Detected file/system behavior
-
- Read-only
- Write / modify
- Shell exec
- Network behavior
- Local-only
- Install commands
- None (reference only)
Heads up: Tool permissions are not declared; actual access depends on the runtime.
# Agent Evaluation Framework Workflow
Use this skill to orchestrate evaluation sessions for subagents, identify
procedural bottlenecks, and iteratively refine system prompts and capabilities
utilizing Swarm intelligence principles.
## 0. Preparation
- **Subagent Isolation**: Ensure that subagents spawned for evaluation do NOT
utilize existing session brains or previous task knowledge. This is critical
to maintain the integrity of meta-testing.
- **Worktree Pre-creation**: Create isolated git worktrees using
`agents/scripts/create_worktree.sh <task_id>` for each test case beforehand.
Report where the worktrees were created to the user. Inside worktrees, builds
MUST use the `tools/dev/gm.py` tool INSIDE the worktree. `gm.py` will
automatically run `setup_worktree_build.py` to prepare the symlinks; manual
execution of `setup_worktree_build.py` is not required.
- **Test Injection**: Copy the target test case into the worktree (e.g.,
`test/mjsunit/repro.js`).
- **Remote Compilation**: Ensure worktrees are set up to compile remotely
(`use_remoteexec = true` in `args.gn`) before proceeding.
## 1. Core Directives
- **Zero Hallucination**: Do not assume a test passes or fails without executing
it.
- **Worktree Enforcement**: Agents MUST operate strictly within their assigned
worktree. They should NOT know the main V8 root exists.
- **Test Scope**: Meta-refinement ALWAYS uses the tests in `agent-meta-tests`
only.
- **Test Immutability**: The `agent-meta-tests` directory cannot be changed.
- **Crash Verification**: Only work on test-cases that still crash.
- **Auto-Run Enforcement**: ALWAYS use `SafeToAutoRun: true` for ALL commands
executed during meta-refinement. Approval must NEVER be asked of the user.
- **Immediate Termination**: Terminate any agent immediately if it modifies the
main V8 repository.
## 2. Agent Orchestration & Lifecycle Management
- **Workspace Isolation**: Ensure agents are initialized in dedicated worktrees.
- **Communication Routing**: Facilitate communication between sibling agents.
Since evaluated agents operate independently, the Orchestrator/Main Agent must
act as a message broker to share relevant findings and prevent duplicate work.
- **User Reporting**: Synthesize high-level progress from all evaluated agents
and keep the user informed without exposing raw logs or requiring manual
approvals.
## 3. Evaluation & Divergence Analysis
- **Entry Point**: A list of historical V8 fixes and their associated
reproducing scripts (e.g., from `test/mjsunit/` or Buganizer).
- **Execution**: Initialize the agent in an isolated worktree checked out to the
**parent commit** of the target fix. Copy the repro script and command the
agent to resolve the bug.
- **Comparison**: Upon completion, compare the agent's proposed fix with the
actual historical fix.
- **Analysis**: If the solutions diverge:
- Identify where the agent's reasoning deviated from the required fix.
- Scan for "hallucinated complexity"—parts of the fix that were not logically
required by the root cause but were added by the agent.
- Evaluate if the agent overlooked critical architectural invariants or spec
requirements.
- **Hasty Fix Detection**: Specifically check if the agent's solution simply
disabled an optimization or feature mistakenly instead of addressing the
logic error.
- **Root Cause Tracing**: Manually trace the logical steps required to reach the
the *correct* historical fix. Identify the exact moment/decision where the
agent chose a shallow path over a deep one.
## 4. Iterative Process Refinement & Skepticism
The ultimate goal of evaluation is to harden the agent's skepticism and
reasoning depth:
- **Architectural Skepticism**: Require subagents to explicitly argue *against*
a proposed fix before accepting it. Look at the problem from multiple
orthogonal angles.
- **Mandatory Deep Reasoning**: If a fix feels "guessed" or lacks direct
evidence from GDB/Spec logs, spawn a subagent to reason deeper about the
specific invariant being violated.
- **Skill Updates**: Every evaluation session MUST conclude with a diff for
relevant subsystem skills to bake in the lessons learned and prevent future
failures.
- **analyze_brain.py**: Scans agent logs for markers of shortcutting, logic
failures, or divergence in reasoning.
Sections and files
Listed in source order.
sections -> 0. Preparation → 1. Core Directives → 2. Agent Orchestration & Lifecycle Management → 3. Evaluation & Divergence Analysis → 4. Iterative Process Refinement & Skepticism
terms -> Subagent Isolation · Worktree Pre-creation · Test Injection · Remote Compilation · Zero Hallucination · Worktree Enforcement · Test Scope · Test Immutability
files/cmd -> agents/scripts/create_worktree.sh <task_id> · tools/dev/gm.py · gm.py · setup_worktree_build.py · test/mjsunit/repro.js · use_remoteexec = true · args.gn · agent-meta-tests
body sha256 -> 11b539fc1996
0. Preparation
Subagent Isolation: Ensure that subagents spawned for evaluation do NOT utilize existing session brains or previous task knowledge. This is critical to maintain the integrity of meta-testing. Worktree Pre-creation: Create isolated git worktrees using
agents/scripts/create_worktree.sh <task_id>for each test case beforehand.1. Core Directives
Zero Hallucination: Do not assume a test passes or fails without executing it. Worktree Enforcement: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists. Test Scope: Meta-refinement ALWAYS uses the tests in
agent-meta-tests2. Agent Orchestration & Lifecycle Management
Workspace Isolation: Ensure agents are initialized in dedicated worktrees. Communication Routing: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work. User Reporting: Synthesize high-level progress from all evaluated agents
3. Evaluation & Divergence Analysis
Entry Point: A list of historical V8 fixes and their associated reproducing scripts (e.g., from
test/mjsunit/or Buganizer). Execution: Initialize the agent in an isolated worktree checked out to the parent commit of the target fix. Copy the repro script and command the agent to resolve the bug.4. Iterative Process Refinement & Skepticism
The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth: Architectural Skepticism: Require subagents to explicitly argue against a proposed fix before accepting it. Look at the problem from multiple orthogonal angles.