agent evaluation framework

Design Community

Skill notes

0. Preparation

Subagent Isolation: Ensure that subagents spawned for evaluation do NOT utilize existing session brains or previous task knowledge. This is critical to maintain the integrity of meta-testing. Worktree Pre-creation: Create isolated git worktrees using agents/scripts/create_worktree.sh <task_id> for each test case beforehand.

1. Core Directives

Zero Hallucination: Do not assume a test passes or fails without executing it. Worktree Enforcement: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists. Test Scope: Meta-refinement ALWAYS uses the tests in agent-meta-tests

2. Agent Orchestration & Lifecycle Management

Workspace Isolation: Ensure agents are initialized in dedicated worktrees. Communication Routing: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work. User Reporting: Synthesize high-level progress from all evaluated agents

3. Evaluation & Divergence Analysis

Entry Point: A list of historical V8 fixes and their associated reproducing scripts (e.g., from test/mjsunit/ or Buganizer). Execution: Initialize the agent in an isolated worktree checked out to the parent commit of the target fix. Copy the repro script and command the agent to resolve the bug.

4. Iterative Process Refinement & Skepticism

The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth: Architectural Skepticism: Require subagents to explicitly argue against a proposed fix before accepting it. Look at the problem from multiple orthogonal angles.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Design
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@v8 · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Author source file SKILL.md

# Agent Evaluation Framework Workflow

Use this skill to orchestrate evaluation sessions for subagents, identify
procedural bottlenecks, and iteratively refine system prompts and capabilities
utilizing Swarm intelligence principles.

## 0. Preparation

- **Subagent Isolation**: Ensure that subagents spawned for evaluation do NOT
  utilize existing session brains or previous task knowledge. This is critical
  to maintain the integrity of meta-testing.
- **Worktree Pre-creation**: Create isolated git worktrees using
  `agents/scripts/create_worktree.sh <task_id>` for each test case beforehand.
  Report where the worktrees were created to the user. Inside worktrees, builds
  MUST use the `tools/dev/gm.py` tool INSIDE the worktree. `gm.py` will
  automatically run `setup_worktree_build.py` to prepare the symlinks; manual
  execution of `setup_worktree_build.py` is not required.
- **Test Injection**: Copy the target test case into the worktree (e.g.,
  `test/mjsunit/repro.js`).
- **Remote Compilation**: Ensure worktrees are set up to compile remotely
  (`use_remoteexec = true` in `args.gn`) before proceeding.

## 1. Core Directives

- **Zero Hallucination**: Do not assume a test passes or fails without executing
  it.
- **Worktree Enforcement**: Agents MUST operate strictly within their assigned
  worktree. They should NOT know the main V8 root exists.
- **Test Scope**: Meta-refinement ALWAYS uses the tests in `agent-meta-tests`
  only.
- **Test Immutability**: The `agent-meta-tests` directory cannot be changed.
- **Crash Verification**: Only work on test-cases that still crash.
- **Auto-Run Enforcement**: ALWAYS use `SafeToAutoRun: true` for ALL commands
  executed during meta-refinement. Approval must NEVER be asked of the user.
- **Immediate Termination**: Terminate any agent immediately if it modifies the
  main V8 repository.

## 2. Agent Orchestration & Lifecycle Management

- **Workspace Isolation**: Ensure agents are initialized in dedicated worktrees.
- **Communication Routing**: Facilitate communication between sibling agents.
  Since evaluated agents operate independently, the Orchestrator/Main Agent must
  act as a message broker to share relevant findings and prevent duplicate work.
- **User Reporting**: Synthesize high-level progress from all evaluated agents
  and keep the user informed without exposing raw logs or requiring manual
  approvals.

## 3. Evaluation & Divergence Analysis

- **Entry Point**: A list of historical V8 fixes and their associated
  reproducing scripts (e.g., from `test/mjsunit/` or Buganizer).
- **Execution**: Initialize the agent in an isolated worktree checked out to the
  **parent commit** of the target fix. Copy the repro script and command the
  agent to resolve the bug.
- **Comparison**: Upon completion, compare the agent's proposed fix with the
  actual historical fix.
- **Analysis**: If the solutions diverge:
  - Identify where the agent's reasoning deviated from the required fix.
  - Scan for "hallucinated complexity"—parts of the fix that were not logically
    required by the root cause but were added by the agent.
  - Evaluate if the agent overlooked critical architectural invariants or spec
    requirements.
  - **Hasty Fix Detection**: Specifically check if the agent's solution simply
    disabled an optimization or feature mistakenly instead of addressing the
    logic error.
- **Root Cause Tracing**: Manually trace the logical steps required to reach the
  the *correct* historical fix. Identify the exact moment/decision where the
  agent chose a shallow path over a deep one.

## 4. Iterative Process Refinement & Skepticism

The ultimate goal of evaluation is to harden the agent's skepticism and
reasoning depth:

- **Architectural Skepticism**: Require subagents to explicitly argue *against*
  a proposed fix before accepting it. Look at the problem from multiple
  orthogonal angles.

- **Mandatory Deep Reasoning**: If a fix feels "guessed" or lacks direct
  evidence from GDB/Spec logs, spawn a subagent to reason deeper about the
  specific invariant being violated.

- **Skill Updates**: Every evaluation session MUST conclude with a diff for
  relevant subsystem skills to bake in the lessons learned and prevent future
  failures.

- **analyze_brain.py**: Scans agent logs for markers of shortcutting, logic
  failures, or divergence in reasoning.

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.