代理人评估框架
- 作者仓库星标 25,042
- 作者仓库 v8
- 领域
- 设计与多媒体
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @v8 · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- 未检出明确要求
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
# Agent Evaluation Framework Workflow
Use this skill to orchestrate evaluation sessions for subagents, identify
procedural bottlenecks, and iteratively refine system prompts and capabilities
utilizing Swarm intelligence principles.
## 0. Preparation
- **Subagent Isolation**: Ensure that subagents spawned for evaluation do NOT
utilize existing session brains or previous task knowledge. This is critical
to maintain the integrity of meta-testing.
- **Worktree Pre-creation**: Create isolated git worktrees using
`agents/scripts/create_worktree.sh <task_id>` for each test case beforehand.
Report where the worktrees were created to the user. Inside worktrees, builds
MUST use the `tools/dev/gm.py` tool INSIDE the worktree. `gm.py` will
automatically run `setup_worktree_build.py` to prepare the symlinks; manual
execution of `setup_worktree_build.py` is not required.
- **Test Injection**: Copy the target test case into the worktree (e.g.,
`test/mjsunit/repro.js`).
- **Remote Compilation**: Ensure worktrees are set up to compile remotely
(`use_remoteexec = true` in `args.gn`) before proceeding.
## 1. Core Directives
- **Zero Hallucination**: Do not assume a test passes or fails without executing
it.
- **Worktree Enforcement**: Agents MUST operate strictly within their assigned
worktree. They should NOT know the main V8 root exists.
- **Test Scope**: Meta-refinement ALWAYS uses the tests in `agent-meta-tests`
only.
- **Test Immutability**: The `agent-meta-tests` directory cannot be changed.
- **Crash Verification**: Only work on test-cases that still crash.
- **Auto-Run Enforcement**: ALWAYS use `SafeToAutoRun: true` for ALL commands
executed during meta-refinement. Approval must NEVER be asked of the user.
- **Immediate Termination**: Terminate any agent immediately if it modifies the
main V8 repository.
## 2. Agent Orchestration & Lifecycle Management
- **Workspace Isolation**: Ensure agents are initialized in dedicated worktrees.
- **Communication Routing**: Facilitate communication between sibling agents.
Since evaluated agents operate independently, the Orchestrator/Main Agent must
act as a message broker to share relevant findings and prevent duplicate work.
- **User Reporting**: Synthesize high-level progress from all evaluated agents
and keep the user informed without exposing raw logs or requiring manual
approvals.
## 3. Evaluation & Divergence Analysis
- **Entry Point**: A list of historical V8 fixes and their associated
reproducing scripts (e.g., from `test/mjsunit/` or Buganizer).
- **Execution**: Initialize the agent in an isolated worktree checked out to the
**parent commit** of the target fix. Copy the repro script and command the
agent to resolve the bug.
- **Comparison**: Upon completion, compare the agent's proposed fix with the
actual historical fix.
- **Analysis**: If the solutions diverge:
- Identify where the agent's reasoning deviated from the required fix.
- Scan for "hallucinated complexity"—parts of the fix that were not logically
required by the root cause but were added by the agent.
- Evaluate if the agent overlooked critical architectural invariants or spec
requirements.
- **Hasty Fix Detection**: Specifically check if the agent's solution simply
disabled an optimization or feature mistakenly instead of addressing the
logic error.
- **Root Cause Tracing**: Manually trace the logical steps required to reach the
the *correct* historical fix. Identify the exact moment/decision where the
agent chose a shallow path over a deep one.
## 4. Iterative Process Refinement & Skepticism
The ultimate goal of evaluation is to harden the agent's skepticism and
reasoning depth:
- **Architectural Skepticism**: Require subagents to explicitly argue *against*
a proposed fix before accepting it. Look at the problem from multiple
orthogonal angles.
- **Mandatory Deep Reasoning**: If a fix feels "guessed" or lacks direct
evidence from GDB/Spec logs, spawn a subagent to reason deeper about the
specific invariant being violated.
- **Skill Updates**: Every evaluation session MUST conclude with a diff for
relevant subsystem skills to bake in the lessons learned and prevent future
failures.
- **analyze_brain.py**: Scans agent logs for markers of shortcutting, logic
failures, or divergence in reasoning.
章节与文件索引
按说明文件中的顺序排列。
0. 准备工作
1. 核心指令
2. 代理编排和生命周期管理
3. 评估与差异分析
4.迭代过程的细化和怀疑
文件/命令 -> agents/scripts/create_worktree.sh <task_id> · tools/dev/gm.py · gm.py · setup_worktree_build.py · test/mjsunit/repro.js · use_remoteexec = true · args.gn · agent-meta-tests
内容校验值 -> 11b539fc1996
当前原文的自动摘译,尚未人工审校。
0. 准备工作
子代理隔离:确保为评估而生成的子代理不会利用现有的会话大脑或先前的任务知识。这对于保持元测试的完整性至关重要。工作树预创建:预先使用
agents/scripts/create_worktree.sh <task_id>为每个测试用例创建独立的 git 工作树。1. 核心指令
零幻觉:不要假设测试通过或失败而不执行它。工作树执行:代理必须严格在其分配的工作树内操作。他们不应该知道主 V8 根的存在。测试范围:元细化始终使用
agent-meta-tests中的测试2. 代理编排和生命周期管理
工作区隔离:确保代理在专用工作树中初始化。通信路由:促进兄弟代理之间的通信。由于评估的代理独立运行,因此协调器/主代理必须充当消息代理来共享相关发现并防止重复工作。用户报告:综合所有评估代理的高级进展
3. 评估与差异分析
入口点:历史 V8 修复及其相关复制脚本的列表(例如,来自
test/mjsunit/或 Buganizer)。执行:在检出到目标修复的父提交的隔离工作树中初始化代理。复制重现脚本并命令代理解决错误。4.迭代过程的细化和怀疑
评估的最终目标是强化代理的怀疑论和推理深度: 架构怀疑论:要求子代理在接受建议的修复之前明确反对它。从多个正交角度看问题。