线束评估

人工智能 社区 v0.2.0

技能解读

当前原文的自动摘译,尚未人工审校。

核心原则

在不熟悉的项目上进行测试——切勿在 OpenHarness 本身上进行测试(代理修改了自己的代码)。克隆一个真实的项目作为工作区。使用真实的 API 调用——没有模拟。配置真实的 LLM 端点。多回合对话——在模型需要先验上下文的情况下,始终测试 2 个以上回合。组合功能——测试钩子+技能+代理循环在一起,而不是孤立的。验证工具执行 - 检查工具调用列表和输出文件,而不仅仅是模型文本。

1. 准备工作空间

克隆一个不熟悉的项目(不要使用 OpenHarness):

2. 配置环境

对于长时间运行的真实评估,不要人为地降低max_turns。使用产品默认值 (200),除非用户明确想要更严格的限制。

3. 在相关时准备真正的沙箱运行时

如果任务是验证沙箱行为,请在运行代理循环之前安装并验证实际运行时:然后通过 OpenHarness 运行最小烟雾检查,而不仅仅是原始srt,以便验证真正的适配器路径:如果缺少沙箱依赖项,请将其视为环境/设置失败,而不是功能回归。

4. 设计测试

每个测试都遵循以下模式:有关详细的代码模板和make_engine/collect帮助程序,请参阅references/test-patterns.md。

流狐档案 运行与权限由流狐静态检查推断,不代表作者声明或实际测试结果。
领域
人工智能
作者声明的智能体
未找到明确声明;不据此推断已兼容或已测试
静态检查
92 / 100 · 启发式扫描,不代表运行安全
运行环境证据
未执行;当前没有隔离运行测试证据
作者 / 版本 / 许可证
@HKUDS · v0.2.0 · 未声明许可证
上下文用量评级
低消耗
接入复杂程度
需简单配置
是否需要外部接口密钥
需要 · Anthropic
兼容的系统
macOS · Linux · Windows
运行环境要求
Python
检测到的文件与系统行为
  • 只读
  • 允许写入 / 修改
  • 命令执行
网络行为
允许外网请求
安装命令数
无(仅作为资料)

需要注意: 未声明工具权限范围;实际权限由运行环境决定。

作者说明文件(原文) SKILL.md

# Harness Eval — End-to-End Feature Validation

Validate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.

## Core Principles

1. **Test on an unfamiliar project** — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace.
2. **Use real API calls** — no mocks. Configure a real LLM endpoint.
3. **Multi-turn conversations** — always test 2+ turns where the model needs prior context.
4. **Combine features** — test hooks+skills+agent loop together, not in isolation.
5. **Verify tool execution** — inspect tool call lists and output files, not just model text.

## Workflow

### 1. Prepare Workspace

Clone an unfamiliar project (do not use OpenHarness):

```bash
git clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace
```

### 2. Configure Environment

```bash
export ANTHROPIC_API_KEY=sk-xxx
export ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic  # or any provider
export ANTHROPIC_MODEL=kimi-k2.5
```

For long-running real evals, do not artificially lower `max_turns`. Use the product default (`200`) unless the user explicitly wants a tighter bound.

### 3. Prepare Real Sandbox Runtime When Relevant

If the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:

```bash
npm install -g @anthropic-ai/sandbox-runtime
sudo apt-get update
sudo apt-get install -y bubblewrap ripgrep
which srt
which bwrap
which rg
srt --version
```

Then run a minimal smoke check through OpenHarness, not just raw `srt`, so you verify the real adapter path:

```python
from pathlib import Path
from openharness.config.settings import Settings, SandboxSettings, save_settings
from openharness.tools.bash_tool import BashTool

cfg = Path("/tmp/openharness-sandbox-settings.json")
save_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)
# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.
```

If sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.

### 4. Design Tests

Each test follows this pattern:

```python
engine = make_engine(system_prompt="...", cwd=UNFAMILIAR_PROJECT)
evs1 = [ev async for ev in engine.submit_message("Read X, analyze Y")]
r1 = collect(evs1)  # text, tools, turns, tokens
evs2 = [ev async for ev in engine.submit_message("Based on what you found...")]
r2 = collect(evs2)
assert "grep" in r1["tools"]  # verify tools ran
```

For detailed code templates and the `make_engine`/`collect` helpers, consult `references/test-patterns.md`.

### 5. Prefer Long-Horizon, Real Agent Loops

For meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.

Recommended pattern:

- Use a real external workspace such as `AutoAgent`
- Use real provider credentials and the actual target model
- Keep `max_turns=200`
- Use per-prompt timeouts large enough for real exploration, such as `240-600s`
- Require at least 2 turns per scenario
- Verify both text quality and tool traces
- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause

Recommended long-horizon scenarios:

- `architecture_multiturn`
  - Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints
  - Turn 2: identify top risks and propose refactors
  - Turn 3: condense into onboarding or remediation actions
  - Success: `bash`, `glob`, `grep`, `read_file` all appear; no timeout; no `MaxTurnsExceeded`

- `hook_block_and_recover`
  - Force the model to try `bash`
  - Block it with a real pre-tool hook
  - Verify the model adapts with `glob`/`grep`/`read_file`

- `sandbox_multiturn`
  - Enable real sandbox settings with `fail_if_unavailable=true`
  - First prompt must start with exactly one shell command such as `pwd && ls -la`
  - Second prompt must explicitly reuse the prior shell findings
  - Success: `bash` executes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors

When a scenario fails, classify it before changing code:

- `MaxTurnsExceeded`: likely eval harness misconfiguration if `max_turns` was manually lowered
- `timeout`: task is too broad or per-prompt timeout is too small
- sandbox unavailable: environment missing `srt`, `bwrap`, or `rg`
- tool error with task still completed: feature may still be healthy; inspect recovery behavior

### 6. Run Tests

```bash
python tests/test_merged_prs_on_autoagent.py   # PR feature tests
python tests/test_real_large_tasks.py           # large multi-step tasks
python tests/test_hooks_skills_plugins_real.py  # hooks/skills/plugins
python -m pytest tests/ -q -k "not autoagent"  # unit tests (no API)
```

For ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:

- uses real OpenHarness engine/tool objects
- targets an unfamiliar repository
- prints per-scenario JSON summaries
- records tools, errors, turns, and token usage
- stays attached until completion

### 7. Interpret Results

| Result | Meaning | Action |
|--------|---------|--------|
| PASS with tool calls | Feature works end-to-end | Done |
| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |
| FAIL with exception | Code bug | Read traceback |
| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |
| Timeout | Task too complex | Increase `max_turns` or simplify prompt |

For long-running real evals, refine the timeout guidance:

- First check whether `max_turns` was manually set too low
- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count
- Distinguish environment failures from product failures
  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression
  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue

## Feature Coverage Checklist

- [ ] Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction
- [ ] Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications
- [ ] Hooks: pre_tool_use blocking → model adapts, post_tool_use firing
- [ ] Skills: skill tool invocation → model follows instructions
- [ ] Plugins: plugin-provided skill loaded and used in agent loop
- [ ] Memory: YAML frontmatter parsing, body content search, context injection
- [ ] Session: save → load → resume with context preserved
- [ ] Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn
- [ ] Cost: token accumulation across turns

## Common Pitfalls

- Testing on OpenHarness itself — agent modifies its own running code
- Using mocks — misses serialization and API compatibility bugs
- Single-turn only — misses context accumulation and compaction bugs
- Artificially lowering `max_turns` during real evals — can create false failures that do not reflect product defaults
- Not checking tool call list — model may claim tool use without calling it
- Hardcoding paths — use `WORKSPACE` variable, skip in CI with `pytest.mark.skipif`
- Declaring sandbox “tested” after only checking raw `srt` — verify the OpenHarness adapter path too
- Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives

## Additional Resources

### Reference Files

- **`references/test-patterns.md`** — Complete code templates for `make_engine`, `collect`, and each feature category
- **`references/feature-matrix.md`** — Detailed test cases for every OpenHarness module

### Existing Test Files

Working test suites in the repo:
- `tests/test_merged_prs_on_autoagent.py` — PR feature validation
- `tests/test_real_large_tasks.py` — Large multi-step tasks
- `tests/test_hooks_skills_plugins_real.py` — Hooks/skills/plugins in agent loops
- `tests/test_untested_features.py` — Module-level integration tests

讨论

基于 GitHub Discussions。登录 GitHub 即可参与讨论、点赞、订阅更新。