skill-test
- Repo stars 1,553
- Author updated Live
- Author repo ai-dev-kit
- Domain
- Data
- Compatible agents
-
- Claude Code
- Cursor
- Cline
- Codex
- Windsurf
- Gemini CLI
- +20
- Trust score
- 88 / 100 · community maintained
- Author / version / license
- @databricks-solutions · no license declared
- Token usage
- Lean
- Setup complexity
- Guided setup
- External API key
- Not required
- Operating systems
- Unspecified (assume cross-platform)
- Runtime requirements
- Python
- Permissions
-
- Read-only
- Write / modify
- Shell exec
- Env read
- Network behavior
- Local-only
- Install commands
- 26 variants
Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.
Heads up: 未限定 allowed-tools,默认拥有全部工具权限。
---
name: skill-test
description: Testing framework for evaluating Databricks skills. Use when building test cases for skills, run…
category: data
runtime: Python
---
# skill-test output preview
## PART A: Task fit
- Use case: Testing framework for evaluating Databricks skills. Use when building test cases for skills, running skill evaluations, comparing skill versions, or creating ground truth datasets with the Generate-Review-Promote (GRP) pipeline. Triggers include "test skill", "evaluate skill", "skill regression", "ground truth", "GRP pipeline", "skill quality", and "skill metrics"..
- Inputs: target material, constraints, expected output, and acceptance criteria.
- Evidence boundary: follow “Quick References / /skill-test Command / Basic Usage” and do not present inference as author intent.
## PART B: Execution result
- **01** The card summarizes the use case; runtime output centers on “Testing framework for evaluating Databricks skills. Use when building test cases for skills, running skill evaluations, comparing skill versions, or creating ground truth datasets with the Generate-Review-Promote (GRP) pipeline. Triggers include "test skill", "evaluate skill", "skill regression", "ground truth", "GRP pipeline", "skill quality", and "skill metrics".”.
- **02** When the source has headings, the agent prioritizes “Quick References / /skill-test Command / Basic Usage” so the result follows the author’s structure.
- **03** Typical output includes task judgment, concrete steps, required commands or file edits, validation, and follow-up options.
- **04** Risk context follows the fingerprint: read files, write/modify files, run shell commands, read environment variables; mostly runs locally; usually needs no extra API key.
## Running Rules
- read files, write/modify files, run shell commands, read environment variables; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding scope.
- Return the result, validation criteria, and next iteration options. The source mentions slash commands such as `/skill-test`, `/users`; use them first when your agent supports command triggers.
Name target files or source material, expected output, forbidden changes, and whether network or shell access is allowed. Permission fingerprint: read files, write/modify files, run shell commands, read environment variables.
Start with a small task and check whether the result follows “Quick References / /skill-test Command / Basic Usage”. Inspect diffs, logs, previews, or tests before expanding scope.
Confirm the final output includes a concrete result, evidence, and next action. If it stays generic, tighten inputs, boundaries, and acceptance criteria.
---
name: skill-test
description: Testing framework for evaluating Databricks skills. Use when building test cases for skills, run…
category: data
source: databricks-solutions/ai-dev-kit
---
# skill-test
## When to use
- Testing framework for evaluating Databricks skills. Use when building test cases for skills, running skill evaluations…
- Use it when the task has clear inputs, repeatable steps, and validation criteria.
## What to provide
- Target material, scope, expected result, and forbidden changes.
- Whether network, commands, file writes, or external services are allowed.
## Execution rules
- Organize steps around “Quick References / /skill-test Command / Basic Usage” and keep inference separate from source facts.
- read files, write/modify files, run shell commands, read environment variables; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding the task.
## Output requirements
- Return the deliverable, key evidence, validation method, and next action.
- Mark missing information as unknown; do not invent commands, platforms, or dependencies. The author source anchors workflow facts; repository files anchor sources and commands; Fluxly only adds fit, limitations, and quality judgment.
skill "skill-test" {
input -> user goal + target files + boundaries + acceptance criteria
context -> Quick References / /skill-test Command / Basic Usage
rules -> SKILL.md triggers / order / output contract
runtime -> Python | read files, write/modify files, run shell commands, read environment variables | mostly runs locally
guardrails -> usually needs no extra API key + small-sample validation + diff/log review
output -> copyable result + checklist + next iteration
} Databricks Skills Testing Framework
Offline YAML-first evaluation with human-in-the-loop review and interactive skill improvement.
Quick References
- Scorers - Available scorers and quality gates
- YAML Schemas - Manifest and ground truth formats
- Python API - Programmatic usage examples
- Workflows - Detailed example workflows
- Trace Evaluation - Session trace analysis
/skill-test Command
The /skill-test command provides an interactive CLI for testing Databricks skills with real execution on Databricks.
Basic Usage
/skill-test <skill-name> [subcommand]
Subcommands
| Subcommand | Description |
|---|---|
run |
Run evaluation against ground truth (default) |
regression |
Compare current results against baseline |
init |
Initialize test scaffolding for a new skill |
add |
Interactive: prompt -> invoke skill -> test -> save |
add --trace |
Add test case with trace evaluation |
review |
Review pending candidates interactively |
review --batch |
Batch approve all pending candidates |
baseline |
Save current results as regression baseline |
mlflow |
Run full MLflow evaluation with LLM judges |
trace-eval |
Evaluate traces against skill expectations |
list-traces |
List available traces (MLflow or local) |
scorers |
List configured scorers for a skill |
scorers update |
Add/remove scorers or update default guidelines |
sync |
Sync YAML to Unity Catalog (Phase 2) |
Quick Examples
/skill-test databricks-spark-declarative-pipelines run
/skill-test databricks-spark-declarative-pipelines add --trace
/skill-test databricks-spark-declarative-pipelines review --batch --filter-success
/skill-test my-new-skill init
See Workflows for detailed examples of each subcommand.
Execution Instructions
Environment Setup
uv pip install -e .test/
Environment variables for Databricks MLflow:
DATABRICKS_CONFIG_PROFILE- Databricks CLI profile (default: "DEFAULT")MLFLOW_TRACKING_URI- Set to "databricks" for Databricks MLflowMLFLOW_EXPERIMENT_NAME- Experiment path (e.g., "/Users/{user}/skill-test")
Running Scripts
All subcommands have corresponding scripts in .test/scripts/:
uv run python .test/scripts/{subcommand}.py {skill_name} [options]
| Subcommand | Script |
|---|---|
run |
run_eval.py |
regression |
regression.py |
init |
init_skill.py |
add |
add.py |
review |
review.py |
baseline |
baseline.py |
mlflow |
mlflow_eval.py |
scorers |
scorers.py |
scorers update |
scorers_update.py |
sync |
sync.py |
trace-eval |
trace_eval.py |
list-traces |
list_traces.py |
_routing mlflow |
routing_eval.py |
Use --help on any script for available options.
Command Handler
When /skill-test is invoked, parse arguments and execute the appropriate command.
Argument Parsing
args[0]= skill_name (required)args[1]= subcommand (optional, default: "run")
Subcommand Routing
| Subcommand | Action |
|---|---|
run |
Execute run(skill_name, ctx) and display results |
regression |
Execute regression(skill_name, ctx) and display comparison |
init |
Execute init(skill_name, ctx) to create scaffolding |
add |
Prompt for test input, invoke skill, run interactive() |
review |
Execute review(skill_name, ctx) to review pending candidates |
baseline |
Execute baseline(skill_name, ctx) to save as regression baseline |
mlflow |
Execute mlflow_eval(skill_name, ctx) with MLflow logging |
scorers |
Execute scorers(skill_name, ctx) to list configured scorers |
scorers update |
Execute scorers_update(skill_name, ctx, ...) to modify scorers |
init Behavior
When running /skill-test <skill-name> init:
- Read the skill's SKILL.md to understand its purpose
- Create
manifest.yamlwith appropriate scorers and trace_expectations - Create empty
ground_truth.yamlandcandidates.yamltemplates - Recommend test prompts based on documentation examples
Follow with /skill-test <skill-name> add using recommended prompts.
Context Setup
Create CLIContext with MCP tools before calling any command. See Python API for details.
File Locations
Important: All test files are stored at the repository root level, not relative to this skill's directory.
| File Type | Path |
|---|---|
| Ground truth | {repo_root}/.test/skills/{skill-name}/ground_truth.yaml |
| Candidates | {repo_root}/.test/skills/{skill-name}/candidates.yaml |
| Manifest | {repo_root}/.test/skills/{skill-name}/manifest.yaml |
| Routing tests | {repo_root}/.test/skills/_routing/ground_truth.yaml |
| Baselines | {repo_root}/.test/baselines/{skill-name}/baseline.yaml |
For example, to test databricks-spark-declarative-pipelines in this repository:
/Users/.../ai-dev-kit/.test/skills/databricks-spark-declarative-pipelines/ground_truth.yaml
Not relative to the skill definition:
/Users/.../ai-dev-kit/.claude/skills/skill-test/skills/... # WRONG
Directory Structure
.test/ # At REPOSITORY ROOT (not skill directory)
├── pyproject.toml # Package config (pip install -e ".test/")
├── README.md # Contributor documentation
├── SKILL.md # Source of truth (synced to .claude/skills/)
├── install_skill_test.sh # Sync script
├── scripts/ # Wrapper scripts
│ ├── _common.py # Shared utilities
│ ├── run_eval.py
│ ├── regression.py
│ ├── init_skill.py
│ ├── add.py
│ ├── baseline.py
│ ├── mlflow_eval.py
│ ├── routing_eval.py
│ ├── trace_eval.py # Trace evaluation
│ ├── list_traces.py # List available traces
│ ├── scorers.py
│ ├── scorers_update.py
│ └── sync.py
├── src/
│ └── skill_test/ # Python package
│ ├── cli/ # CLI commands module
│ ├── fixtures/ # Test fixture setup
│ ├── scorers/ # Evaluation scorers
│ ├── grp/ # Generate-Review-Promote pipeline
│ └── runners/ # Evaluation runners
├── skills/ # Per-skill test definitions
│ ├── _routing/ # Routing test cases
│ └── {skill-name}/ # Skill-specific tests
│ ├── ground_truth.yaml
│ ├── candidates.yaml
│ └── manifest.yaml
├── tests/ # Unit tests
├── references/ # Documentation references
└── baselines/ # Regression baselines
References
- Scorers - Available scorers and quality gates
- YAML Schemas - Manifest and ground truth formats
- Python API - Programmatic usage examples
- Workflows - Detailed example workflows
- Trace Evaluation - Session trace analysis
Decide Fit First
Design Intent
How To Use It
Boundaries And Review