技能测试

Name: 技能 测试
Author: databricks-solutions

数据社区

作者仓库星标 1,553
作者仓库 ai-dev-kit

解读按原文结构重写，命令、链接、术语均保留；右侧可核对作者原始 SKILL.md

原文结构

Quick References：Scorers - Available scorers and quality gates YAML Schemas - Manifest and ground truth formats Python API - Programmatic usage examples
/skill-test Command：The /skill-test command provides an interactive CLI for testing Databricks skills with real execution on Databricks.
Basic Usage：原文以此为独立章节。

适用与边界

当前原文没有单列适用场景。
当前原文没有单列不适用场景或限制。

原文中的明确线索

要点：「Important」、「repository root」、「Not」、「Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement.」、「The /skill-test command provides an interactive CLI for testing Databricks skills with real execution on Databricks.」、「See Workflows for detailed examples of each subcommand.」、「Use --help on any script for available options.」、「When /skill-test is invoked, parse arguments and execute the appropriate command.」
文件与命令：/skill-test、run、regression、init、add、add --trace、review、review --batch

流狐整理：以上内容来自当前 SKILL.md 的章节与原词；未补写作者没有声明的工具、兼容性或能力。

Databricks Skills Testing Framework

Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement.

Quick References

Scorers - Available scorers and quality gates
YAML Schemas - Manifest and ground truth formats
Python API - Programmatic usage examples
Workflows - Detailed example workflows
Trace Evaluation - Session trace analysis

/skill-test Command

The /skill-test command provides an interactive CLI for testing Databricks skills with real execution on Databricks.

Basic Usage

/skill-test <skill-name> [subcommand]

Subcommands

Subcommand	Description
`run`	Run evaluation against ground truth (default)
`regression`	Compare current results against baseline
`init`	Initialize test scaffolding for a new skill
`add`	Interactive: prompt -> invoke skill -> test -> save
`add --trace`	Add test case with trace evaluation
`review`	Review pending candidates interactively
`review --batch`	Batch approve all pending candidates
`baseline`	Save current results as regression baseline
`mlflow`	Run full MLflow evaluation with LLM judges
`trace-eval`	Evaluate traces against skill expectations
`list-traces`	List available traces (MLflow or local)
`scorers`	List configured scorers for a skill
`scorers update`	Add/remove scorers or update default guidelines
`sync`	Sync YAML to Unity Catalog (Phase 2)

Quick Examples

/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace
/skill-test databricks-spark-declarative-pipelines review --batch --filter-success
/skill-test my-new-skill init

See Workflows for detailed examples of each subcommand.

Execution Instructions

Environment Setup

uv pip install -e .test/

Environment variables for Databricks MLflow:

DATABRICKS_CONFIG_PROFILE - Databricks CLI profile (default: "DEFAULT")
MLFLOW_TRACKING_URI - Set to "databricks" for Databricks MLflow
MLFLOW_EXPERIMENT_NAME - Experiment path (e.g., "/Users/{user}/skill-test")

Running Scripts

All subcommands have corresponding scripts in .test/scripts/:

uv run python .test/scripts/{subcommand}.py {skill_name} [options]

Subcommand	Script
`run`	`run_eval.py`
`regression`	`regression.py`
`init`	`init_skill.py`
`add`	`add.py`
`review`	`review.py`
`baseline`	`baseline.py`
`mlflow`	`mlflow_eval.py`
`scorers`	`scorers.py`
`scorers update`	`scorers_update.py`
`sync`	`sync.py`
`trace-eval`	`trace_eval.py`
`list-traces`	`list_traces.py`
`_routing mlflow`	`routing_eval.py`

Use --help on any script for available options.

Command Handler

When /skill-test is invoked, parse arguments and execute the appropriate command.

Argument Parsing

args[0] = skill_name (required)
args[1] = subcommand (optional, default: "run")

Subcommand Routing

Subcommand	Action
`run`	Execute `run(skill_name, ctx)` and display results
`regression`	Execute `regression(skill_name, ctx)` and display comparison
`init`	Execute `init(skill_name, ctx)` to create scaffolding
`add`	Prompt for test input, invoke skill, run `interactive()`
`review`	Execute `review(skill_name, ctx)` to review pending candidates
`baseline`	Execute `baseline(skill_name, ctx)` to save as regression baseline
`mlflow`	Execute `mlflow_eval(skill_name, ctx)` with MLflow logging
`scorers`	Execute `scorers(skill_name, ctx)` to list configured scorers
`scorers update`	Execute `scorers_update(skill_name, ctx, ...)` to modify scorers

init Behavior

When running /skill-test <skill-name> init:

Read the skill's SKILL.md to understand its purpose
Create manifest.yaml with appropriate scorers and trace_expectations
Create empty ground_truth.yaml and candidates.yaml templates
Recommend test prompts based on documentation examples

Follow with /skill-test <skill-name> add using recommended prompts.

Context Setup

Create CLIContext with MCP tools before calling any command. See Python API for details.

File Locations

Important: All test files are stored at the repository root level, not relative to this skill's directory.

File Type	Path
Ground truth	`{repo_root}/.test/skills/{skill-name}/ground_truth.yaml`
Candidates	`{repo_root}/.test/skills/{skill-name}/candidates.yaml`
Manifest	`{repo_root}/.test/skills/{skill-name}/manifest.yaml`
Routing tests	`{repo_root}/.test/skills/_routing/ground_truth.yaml`
Baselines	`{repo_root}/.test/baselines/{skill-name}/baseline.yaml`

For example, to test databricks-spark-declarative-pipelines in this repository:

/Users/.../ai-dev-kit/.test/skills/databricks-spark-declarative-pipelines/ground_truth.yaml

Not relative to the skill definition:

/Users/.../ai-dev-kit/.claude/skills/skill-test/skills/...  # WRONG

Directory Structure

.test/                          # At REPOSITORY ROOT (not skill directory)
├── pyproject.toml              # Package config (pip install -e ".test/")
├── README.md                   # Contributor documentation
├── SKILL.md                    # Source of truth (synced to .claude/skills/)
├── install_skill_test.sh       # Sync script
├── scripts/                    # Wrapper scripts
│   ├── _common.py              # Shared utilities
│   ├── run_eval.py
│   ├── regression.py
│   ├── init_skill.py
│   ├── add.py
│   ├── baseline.py
│   ├── mlflow_eval.py
│   ├── routing_eval.py
│   ├── trace_eval.py           # Trace evaluation
│   ├── list_traces.py          # List available traces
│   ├── scorers.py
│   ├── scorers_update.py
│   └── sync.py
├── src/
│   └── skill_test/             # Python package
│       ├── cli/                # CLI commands module
│       ├── fixtures/           # Test fixture setup
│       ├── scorers/            # Evaluation scorers
│       ├── grp/                # Generate-Review-Promote pipeline
│       └── runners/            # Evaluation runners
├── skills/                     # Per-skill test definitions
│   ├── _routing/               # Routing test cases
│   └── {skill-name}/           # Skill-specific tests
│       ├── ground_truth.yaml
│       ├── candidates.yaml
│       └── manifest.yaml
├── tests/                      # Unit tests
├── references/                 # Documentation references
└── baselines/                  # Regression baselines

References

Scorers - Available scorers and quality gates
YAML Schemas - Manifest and ground truth formats
Python API - Programmatic usage examples
Workflows - Detailed example workflows
Trace Evaluation - Session trace analysis

流狐档案作者与许可取自来源；运行、权限和网络为流狐检测或估算

流狐分类: 数据
作者声明 Agent: 未找到明确声明；不据此推断已兼容或已测试
静态检查: 88 / 100 · 启发式扫描，不代表运行安全
作者 / 版本 / 许可: @databricks-solutions · 未声明 license
流狐 Token 估算: 低消耗
流狐接入估算: 需简单配置
是否需要外部 API Key: 未发现要求
检测到的系统要求: 未声明
底层运行要求: Python
检测到的文件与系统行为: 只读
允许写入 / 修改
Shell 执行
读取环境变量
检测到的网络行为: 仅限本地
安装命令数: 无（仅作为资料）

档案由构建时根据 SKILL.md 与安装命令自动衍生，可能与作者实际意图存在差异。

需要注意： 未限定 allowed-tools，默认拥有全部工具权限。

输出预览 skill-test-databricks-solutions.preview

# Quick Examples

/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace
/skill-test databricks-spark-declarative-pipelines review --batch --filter-success
/skill-test my-new-skill init

SKILL.md SKILL.md

# Databricks Skills Testing Framework

Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement.

## Quick References

- [Scorers](references/scorers.md) - Available scorers and quality gates
- [YAML Schemas](references/yaml-schemas.md) - Manifest and ground truth formats
- [Python API](references/python-api.md) - Programmatic usage examples
- [Workflows](references/workflows.md) - Detailed example workflows
- [Trace Evaluation](references/trace-eval.md) - Session trace analysis

## /skill-test Command

The `/skill-test` command provides an interactive CLI for testing Databricks skills with real execution on Databricks.

### Basic Usage

```
/skill-test <skill-name> [subcommand]
```

### Subcommands

| Subcommand | Description |
|------------|-------------|
| `run` | Run evaluation against ground truth (default) |
| `regression` | Compare current results against baseline |
| `init` | Initialize test scaffolding for a new skill |
| `add` | Interactive: prompt -> invoke skill -> test -> save |
| `add --trace` | Add test case with trace evaluation |
| `review` | Review pending candidates interactively |
| `review --batch` | Batch approve all pending candidates |
| `baseline` | Save current results as regression baseline |
| `mlflow` | Run full MLflow evaluation with LLM judges |
| `trace-eval` | Evaluate traces against skill expectations |
| `list-traces` | List available traces (MLflow or local) |
| `scorers` | List configured scorers for a skill |
| `scorers update` | Add/remove scorers or update default guidelines |
| `sync` | Sync YAML to Unity Catalog (Phase 2) |

### Quick Examples

```
/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace

…

证据边界与执行链路

作者原文负责流程事实；流狐只索引当前章节、要点、文件与命令。

章节    -> Quick References → /skill-test Command → Basic Usage → Subcommands → Quick Examples → Execution Instructions
要点    -> Important · repository root · Not · Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement. · The /skill-test command provides an interactive CLI for testing Databricks skills with real execution on Databricks. · See [Workflows](references/workflows.md) for detailed examples of each subcommand. · Use --help on any script for available options. · When /skill-test is invoked, parse arguments and execute the appropriate command.
文件/命令 -> /skill-test · run · regression · init · add · add --trace · review · review --batch
内容 SHA-256 -> 7ff12a59851e