技能测试
- 作者仓库星标 1,553
- 作者仓库 ai-dev-kit
- 领域
- 数据
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @databricks-solutions · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需简单配置
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- 未声明(默认跨平台)
- 运行环境要求
- Python
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 读取环境变量
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace
/skill-test databricks-spark-declarative-pipelines review --batch --filter-success
/skill-test my-new-skill init
# Databricks Skills Testing Framework
Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement.
## Quick References
- [Scorers](references/scorers.md) - Available scorers and quality gates
- [YAML Schemas](references/yaml-schemas.md) - Manifest and ground truth formats
- [Python API](references/python-api.md) - Programmatic usage examples
- [Workflows](references/workflows.md) - Detailed example workflows
- [Trace Evaluation](references/trace-eval.md) - Session trace analysis
## /skill-test Command
The `/skill-test` command provides an interactive CLI for testing Databricks skills with real execution on Databricks.
### Basic Usage
```
/skill-test <skill-name> [subcommand]
```
### Subcommands
| Subcommand | Description |
|------------|-------------|
| `run` | Run evaluation against ground truth (default) |
| `regression` | Compare current results against baseline |
| `init` | Initialize test scaffolding for a new skill |
| `add` | Interactive: prompt -> invoke skill -> test -> save |
| `add --trace` | Add test case with trace evaluation |
| `review` | Review pending candidates interactively |
| `review --batch` | Batch approve all pending candidates |
| `baseline` | Save current results as regression baseline |
| `mlflow` | Run full MLflow evaluation with LLM judges |
| `trace-eval` | Evaluate traces against skill expectations |
| `list-traces` | List available traces (MLflow or local) |
| `scorers` | List configured scorers for a skill |
| `scorers update` | Add/remove scorers or update default guidelines |
| `sync` | Sync YAML to Unity Catalog (Phase 2) |
### Quick Examples
```
/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace
/skill-test databricks-spark-declarative-pipelines review --batch --filter-success
/skill-test my-new-skill init
```
See [Workflows](references/workflows.md) for detailed examples of each subcommand.
## Execution Instructions
### Environment Setup
```bash
uv pip install -e .test/
```
Environment variables for Databricks MLflow:
- `DATABRICKS_CONFIG_PROFILE` - Databricks CLI profile (default: "DEFAULT")
- `MLFLOW_TRACKING_URI` - Set to "databricks" for Databricks MLflow
- `MLFLOW_EXPERIMENT_NAME` - Experiment path (e.g., "/Users/{user}/skill-test")
### Running Scripts
All subcommands have corresponding scripts in `.test/scripts/`:
```bash
uv run python .test/scripts/{subcommand}.py {skill_name} [options]
```
| Subcommand | Script |
|------------|--------|
| `run` | `run_eval.py` |
| `regression` | `regression.py` |
| `init` | `init_skill.py` |
| `add` | `add.py` |
| `review` | `review.py` |
| `baseline` | `baseline.py` |
| `mlflow` | `mlflow_eval.py` |
| `scorers` | `scorers.py` |
| `scorers update` | `scorers_update.py` |
| `sync` | `sync.py` |
| `trace-eval` | `trace_eval.py` |
| `list-traces` | `list_traces.py` |
| `_routing mlflow` | `routing_eval.py` |
Use `--help` on any script for available options.
## Command Handler
When `/skill-test` is invoked, parse arguments and execute the appropriate command.
### Argument Parsing
- `args[0]` = skill_name (required)
- `args[1]` = subcommand (optional, default: "run")
### Subcommand Routing
| Subcommand | Action |
|------------|--------|
| `run` | Execute `run(skill_name, ctx)` and display results |
| `regression` | Execute `regression(skill_name, ctx)` and display comparison |
| `init` | Execute `init(skill_name, ctx)` to create scaffolding |
| `add` | Prompt for test input, invoke skill, run `interactive()` |
| `review` | Execute `review(skill_name, ctx)` to review pending candidates |
| `baseline` | Execute `baseline(skill_name, ctx)` to save as regression baseline |
| `mlflow` | Execute `mlflow_eval(skill_name, ctx)` with MLflow logging |
| `scorers` | Execute `scorers(skill_name, ctx)` to list configured scorers |
| `scorers update` | Execute `scorers_update(skill_name, ctx, ...)` to modify scorers |
### init Behavior
When running `/skill-test <skill-name> init`:
1. Read the skill's SKILL.md to understand its purpose
2. Create `manifest.yaml` with appropriate scorers and trace_expectations
3. Create empty `ground_truth.yaml` and `candidates.yaml` templates
4. Recommend test prompts based on documentation examples
Follow with `/skill-test <skill-name> add` using recommended prompts.
### Context Setup
Create CLIContext with MCP tools before calling any command. See [Python API](references/python-api.md#clicontext-setup) for details.
## File Locations
**Important:** All test files are stored at the **repository root** level, not relative to this skill's directory.
| File Type | Path |
|-----------|------|
| Ground truth | `{repo_root}/.test/skills/{skill-name}/ground_truth.yaml` |
| Candidates | `{repo_root}/.test/skills/{skill-name}/candidates.yaml` |
| Manifest | `{repo_root}/.test/skills/{skill-name}/manifest.yaml` |
| Routing tests | `{repo_root}/.test/skills/_routing/ground_truth.yaml` |
| Baselines | `{repo_root}/.test/baselines/{skill-name}/baseline.yaml` |
For example, to test `databricks-spark-declarative-pipelines` in this repository:
```
/Users/.../ai-dev-kit/.test/skills/databricks-spark-declarative-pipelines/ground_truth.yaml
```
**Not** relative to the skill definition:
```
/Users/.../ai-dev-kit/.claude/skills/skill-test/skills/... # WRONG
```
## Directory Structure
```
.test/ # At REPOSITORY ROOT (not skill directory)
├── pyproject.toml # Package config (pip install -e ".test/")
├── README.md # Contributor documentation
├── SKILL.md # Source of truth (synced to .claude/skills/)
├── install_skill_test.sh # Sync script
├── scripts/ # Wrapper scripts
│ ├── _common.py # Shared utilities
│ ├── run_eval.py
│ ├── regression.py
│ ├── init_skill.py
│ ├── add.py
│ ├── baseline.py
│ ├── mlflow_eval.py
│ ├── routing_eval.py
│ ├── trace_eval.py # Trace evaluation
│ ├── list_traces.py # List available traces
│ ├── scorers.py
│ ├── scorers_update.py
│ └── sync.py
├── src/
│ └── skill_test/ # Python package
│ ├── cli/ # CLI commands module
│ ├── fixtures/ # Test fixture setup
│ ├── scorers/ # Evaluation scorers
│ ├── grp/ # Generate-Review-Promote pipeline
│ └── runners/ # Evaluation runners
├── skills/ # Per-skill test definitions
│ ├── _routing/ # Routing test cases
│ └── {skill-name}/ # Skill-specific tests
│ ├── ground_truth.yaml
│ ├── candidates.yaml
│ └── manifest.yaml
├── tests/ # Unit tests
├── references/ # Documentation references
└── baselines/ # Regression baselines
```
## References
- [Scorers](references/scorers.md) - Available scorers and quality gates
- [YAML Schemas](references/yaml-schemas.md) - Manifest and ground truth formats
- [Python API](references/python-api.md) - Programmatic usage examples
- [Workflows](references/workflows.md) - Detailed example workflows
- [Trace Evaluation](references/trace-eval.md) - Session trace analysis
章节与文件索引
按说明文件中的顺序排列。
快速参考
/技能测试命令
子命令
快速示例
环境设置
文件/命令 -> /skill-test · run · regression · init · add · add --trace · review · review --batch
内容校验值 -> 7ff12a59851e
当前原文的自动摘译,尚未人工审校。
快速参考
记分器 - 可用的记分器和质量门 YAML 模式 - 清单和基本事实格式 Python API - 编程使用示例 工作流程 - 详细的示例工作流程 跟踪评估 - 会话跟踪分析
/技能测试命令
/skill-test命令提供了一个交互式 CLI,用于在 Databricks 上实际执行来测试 Databricks 技能。子命令
子命令 · 描述
run· 根据真实情况运行评估(默认)regression· 将当前结果与基线进行比较init· 初始化新技能的测试脚手架add· 交互:提示 -> 调用技能 -> 测试 -> 保存快速示例
有关每个子命令的详细示例,请参阅工作流程。
环境设置
Databricks MLflow 的环境变量:
DATABRICKS_CONFIG_PROFILE- Databricks CLI 配置文件(默认值:“DEFAULT”)MLFLOW_TRACKING_URI- 设置为 Databricks MLflow 的“databricks”MLFLOW_EXPERIMENT_NAME- 实验路径(例如,“/Users/{user}/skill-test”)