harness eval

AI Community v0.2.0

Skill notes

Core Principles

Test on an unfamiliar project — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace. Use real API calls — no mocks. Configure a real LLM endpoint. Multi-turn conversations — always test 2+ turns where the model needs prior context. Combine features — test hooks+skills+agent loop together, not in isolation. Verify tool execution — inspect tool call lists and output files, not just model text.

1. Prepare Workspace

Clone an unfamiliar project (do not use OpenHarness):

2. Configure Environment

For long-running real evals, do not artificially lower max_turns. Use the product default (200) unless the user explicitly wants a tighter bound.

3. Prepare Real Sandbox Runtime When Relevant

If the task is validating sandbox behavior, install and verify the actual runtime before running agent loops: Then run a minimal smoke check through OpenHarness, not just raw srt, so you verify the real adapter path: If sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.

4. Design Tests

Each test follows this pattern: For detailed code templates and the make_engine/collect helpers, consult references/test-patterns.md.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
AI
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
92 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@HKUDS · v0.2.0 · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Required · Anthropic
Operating systems
macOS · Linux · Windows
Runtime requirements
Python
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview harness-eval.preview
# 7. Interpret Results

- First check whether `max_turns` was manually set too low
- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count
- Distinguish environment failures from product failures
  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression
  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.