代理评估

写作 社区

技能解读

当前原文的自动摘译,尚未人工审校。

数据集形状

evals/datasets/smoke.jsonl — 每行一个 JSON 对象:reference 是可选的,仅由与已知答案进行比较的评估者使用(LLM 作为判断的正确性、精确匹配等)。

评估者

evals/evaluators.py:

跑步者

evals/run.py:使用:python evals/run.py(全)或python evals/run.py --smoke(仅烟雾)运行。

建议的烟雾模式(用于部署门控)

保持evals/datasets/smoke.jsonl小(3-5行)和快(总共<60秒)。在任何部署之前运行python evals/run.py --smoke。如果烟雾失败,请修复代理或烟雾数据集——切勿绕过。建议使用此模式,但不强制执行;部署技能需要您手动完成。

比较两次运行

langsmith UI 是并排比较实验的最佳位置。要在两个 evals/results/*.json 文件之间进行快速 CLI 比较,请编写一个 30 行脚本来加载这两个文件并打印指标增量 - 没有内置的 CLI 用于此操作。测试策略 langsmith 包附带了一个 pytest 插件。生产代理的三层测试:

流狐档案 运行与权限由流狐静态检查推断,不代表作者声明或实际测试结果。
领域
写作
作者声明的智能体
未找到明确声明;不据此推断已兼容或已测试
静态检查
88 / 100 · 启发式扫描,不代表运行安全
运行环境证据
未执行;当前没有隔离运行测试证据
作者 / 版本 / 许可证
@tomevault-io · 未声明许可证
上下文用量评级
低消耗
接入复杂程度
需简单配置
是否需要外部接口密钥
需要 · Vendor-specific
兼容的系统
未声明(默认跨平台)
运行环境要求
Python
检测到的文件与系统行为
  • 只读
  • 允许写入 / 修改
  • 命令执行
  • 读取环境变量
网络行为
允许外网请求
安装命令数
无(仅作为资料)

需要注意: 未声明工具权限范围;实际权限由运行环境决定。

作者说明文件(原文) SKILL.md

# Evals + Testing

Use the `langsmith` Python SDK directly. There is no CLI wrapper — write a small `evals/run.py` and run it with `python evals/run.py`. For per-test-function CI checks, use `pytest` with the `langsmith` pytest plugin.

`LANGSMITH_API_KEY` must be set. `LANGSMITH_PROJECT` controls which project the trace lands in.

## Dataset shape

`evals/datasets/smoke.jsonl` — one JSON object per line:

```jsonl
{"input": {"messages": [{"role": "user", "content": "Say hello in one word."}]}, "reference": "Hi"}
{"input": {"messages": [{"role": "user", "content": "What is 2+2?"}]}, "reference": "4"}
```

`reference` is optional and only used by evaluators that compare against a known answer (correctness LLM-as-judge, exact-match, etc.).

## Evaluators

`evals/evaluators.py`:

```python
"""LangSmith evaluators. Each takes (run, example) and returns a result dict."""
from typing import Any

def correctness_llm_judge(run: Any, example: Any) -> dict[str, Any]:
    """LLM-as-judge against `example.outputs['reference']`."""
    # ... call an LLM with run.outputs and example.outputs['reference']
    return {"key": "correctness", "score": 0.9, "comment": "close enough"}

def trajectory(run: Any, example: Any) -> dict[str, Any]:
    """Did the expected tool calls fire?"""
    expected = example.outputs.get("expected_tools", [])
    actual = [c["name"] for c in run.outputs.get("tool_calls", [])]
    score = 1.0 if set(expected).issubset(actual) else 0.0
    return {"key": "trajectory", "score": score}

EVALUATORS = [correctness_llm_judge, trajectory]
```

## Runner

`evals/run.py`:

```python
"""Run LangSmith evals. Usage: python evals/run.py [--smoke]"""
import argparse, json, os
from datetime import UTC, datetime
from pathlib import Path

import langsmith
from agent.agent import agent
from evals.evaluators import EVALUATORS

ROOT = Path(__file__).resolve().parent
DATASETS = ROOT / "datasets"
RESULTS = ROOT / "results"


def _load(name: str) -> list[dict]:
    path = DATASETS / f"{name}.jsonl"
    return [json.loads(l) for l in path.read_text("utf-8").splitlines() if l.strip()]


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--smoke", action="store_true")
    args = ap.parse_args()

    if not os.getenv("LANGSMITH_API_KEY"):
        print("ERROR: LANGSMITH_API_KEY is not set.")
        return 2

    datasets = ["smoke"] if args.smoke else [p.stem for p in DATASETS.glob("*.jsonl")]
    RESULTS.mkdir(exist_ok=True)
    results: dict[str, list[dict]] = {}

    for ds in datasets:
        examples = _load(ds)
        out = langsmith.evaluate(
            lambda inp: agent.invoke(inp),
            data=examples,
            evaluators=EVALUATORS,
            experiment_prefix=f"{ds}-",
        )
        results[ds] = list(out)

    out_path = RESULTS / f"{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}.json"
    out_path.write_text(json.dumps(results, default=str, indent=2), encoding="utf-8")
    print(f"Wrote {out_path}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
```

Run with: `python evals/run.py` (full) or `python evals/run.py --smoke` (smoke only).

## Suggested smoke pattern (for deploy gating)

Keep `evals/datasets/smoke.jsonl` small (3–5 rows) and fast (<60 s total). Run `python evals/run.py --smoke` before any deploy. **If smoke fails, fix the agent or the smoke dataset — never bypass.** This pattern is recommended but not enforced; the deploy skill expects you to do it manually.

## Comparing two runs

The `langsmith` UI is the best place to compare experiments side-by-side. For a quick CLI diff between two `evals/results/*.json` files, write a 30-line script that loads both and prints metric deltas — there's no built-in CLI for this.

---

# Testing strategies

The `langsmith` package ships a pytest plugin. Three layers of testing for a production agent:

## 1. Unit tests (no API calls)

Use `LLMToolEmulator` middleware to short-circuit tool execution, and a fake / mocked model:

```python
# tests/test_agent_unit.py
from langchain.agents import create_agent
from langchain.agents.middleware import LLMToolEmulator

def test_agent_handles_empty_input():
    agent = create_agent(
        model="claude-haiku-4-5",      # fastest available; or use a stub
        tools=[my_tool],
        middleware=[LLMToolEmulator()],   # tools return LLM-emulated outputs
    )
    result = agent.invoke({"messages": [{"role": "user", "content": "hi"}]})
    assert "messages" in result
```

For full hermetic unit tests with no LLM calls at all, mock `init_chat_model` or pass a `FakeChatModel` (`from langchain_core.language_models.fake import FakeChatModel`).

## 2. Integration tests (real LLM, hermetic tools)

Real model, real middleware, but tools mocked / point at sandboxes:

```python
# tests/test_agent_integration.py
import pytest
from agent.agent import agent

@pytest.mark.integration
def test_smoke_basic():
    result = agent.invoke({"messages": [{"role": "user", "content": "Say hi."}]})
    msg = result["messages"][-1]
    assert "hi" in str(msg.content).lower()
```

Run with `pytest -m integration`. Skip in CI without API keys; run locally or in a separately-credentialed CI step.

## 3. Trajectory + dataset tests via langsmith pytest plugin

```python
# tests/test_agent_trajectories.py
import pytest
from langsmith import testing as t

@t.expect(score_min=0.8, evaluator="correctness")
@pytest.mark.parametrize("example", [
    {"messages": [{"role": "user", "content": "What is 2+2?"}], "reference": "4"},
    {"messages": [{"role": "user", "content": "Capital of France?"}], "reference": "Paris"},
])
def test_basic_qa(example):
    from agent.agent import agent
    return agent.invoke({"messages": example["messages"]})
```

The plugin uploads each test result to LangSmith, runs the named evaluator(s), and fails the test if the score falls below the threshold. This is the recommended way to gate PRs on agent quality.

## Eval-as-monitor (production)

In production, the same evaluators that grade your dev runs can run continuously against live traces:

1. Set up an evaluator function in LangSmith UI.
2. Configure it to run on incoming traces (sampled or all).
3. Wire an alert when score drops below threshold.

This catches model drift, prompt rot, and gradual quality degradation that smoke evals don't see.

## Common eval gotchas

- LLM-as-judge evaluators are themselves non-deterministic. Run each example 3+ times and average, OR use a low-temperature judge model.
- Don't mix metric directions. `correctness` higher = better; `latency` higher = worse. Make this explicit in evaluator names so dashboards read correctly.
- Smoke datasets that are too large (>10 rows) become a deploy bottleneck. Keep them tight; rely on the full eval suite running async post-deploy for breadth.
- `langsmith.evaluate(...)` runs invocations in parallel by default. If your agent has rate-limited tools, set `max_concurrency=1` or accept that retries will fire.

---
> Source: [cwijayasundara/agent_cli_langchain](https://github.com/cwijayasundara/agent_cli_langchain) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-05-23 -->

讨论

基于 GitHub Discussions。登录 GitHub 即可参与讨论、点赞、订阅更新。