实验室操作
- 作者仓库星标 0
- 作者仓库 skills-registry
- 领域
- 运维部署
- 作者声明的智能体
- 未找到明确声明;不据此推断已兼容或已测试
- 静态检查
- 88 / 100 · 启发式扫描,不代表运行安全
- 运行环境证据
- 未执行;当前没有隔离运行测试证据
- 作者 / 版本 / 许可证
- @tomevault-io · 未声明许可证
- 上下文用量评级
- 低消耗
- 接入复杂程度
- 需手动接入
- 是否需要外部接口密钥
- 不需要
- 兼容的系统
- Docker
- 运行环境要求
- Docker
- 检测到的文件与系统行为
-
- 只读
- 允许写入 / 修改
- 命令执行
- 网络行为
- 仅限本地
- 安装命令数
- 无(仅作为资料)
需要注意: 未声明工具权限范围;实际权限由运行环境决定。
# Docker Databricks Lab Ops
## Overview
Use this skill for the operational loop of this repository: bring up Docker services, generate source-table mutations, reset Databricks tables, execute a Databricks notebook or job, and verify whether the notebook run finished successfully.
Prefer the bundled scripts over rewriting shell commands. They encode the repository-specific paths and the expected sequence.
## Workflow
### 1. Inspect the repo inputs first
- Confirm `docker-compose.yml`, `postgres-connector.json`, and the target notebook paths exist.
- Read [references/repo-workflow.md](./references/repo-workflow.md) if you need the repo-specific sequence or parameters.
### 2. Bring up the local CDC stack
- Use `scripts/start_stack.sh`.
- This runs `docker compose up -d` from the repository root.
- If the user asked for verification, follow with `docker compose ps` or service-specific health checks.
- If Kafka must be reachable from Databricks through an internal network boundary, first use `scripts/prepare_ngrok_kafka.py` and then start Compose with the discovered `KAFKA_EXTERNAL_HOST` and `KAFKA_EXTERNAL_PORT`.
### 3. Register Debezium connector if CDC ingestion needs to be exercised
- Use `scripts/register_connector.sh`.
- Only do this after Kafka Connect is accepting requests.
- If the connector already exists, report that clearly instead of treating it as a fatal failure.
### 4. Run load generators
- Use `scripts/run_generators.sh`.
- The first argument is film update iterations, the second is rental/payment iterations.
- Prefer bounded runs for verification by passing `ITERATIONS`; avoid indefinite generators unless the user asked for sustained load.
Example:
```bash
skills/docker-databricks-lab-ops/scripts/run_generators.sh 20 40
```
This runs 20 film mutations and 40 rental/payment mutations.
### 5. Reset Databricks tables (when starting fresh)
- Use `scripts/reset_databricks_tables.py` to drop all Bronze/Silver/Gold tables and clear streaming checkpoints before a fresh load.
- Requires `--cluster-id` or will submit via git source.
- Use `--dry-run` to preview what would be dropped without dropping.
```bash
python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
--cluster-id <cluster-id> \
--catalog workspace
# preview only:
python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
--cluster-id <cluster-id> \
--dry-run
```
### 6. Trigger a Databricks notebook job
- Use `scripts/run_databricks_notebook.py`.
- Provide either:
- `--job-id` to run an existing Databricks job, or
- `--notebook-path` and `--cluster-id` to submit a one-off notebook run.
- For dynamic Kafka exposure, pass `--notebook-param KAFKA_BOOTSTRAP=<ngrok-host:port>` so the current tunnel endpoint is used at run time instead of a stale static value.
- This script uses `DATABRICKS_HOST` and `DATABRICKS_TOKEN` from the environment.
Examples:
```bash
python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
--job-id 123 \
--notebook-param KAFKA_BOOTSTRAP=0.tcp.eu.ngrok.io:12345
```
```bash
python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
--notebook-path /Workspace/agent/notebook \
--cluster-id 0123-456abc-cluster
```
### 7. Verify notebook behavior
- Treat the job as successful only when lifecycle is terminal and result is `SUCCESS`.
- On failure, capture:
- `run_id`
- lifecycle state
- result state
- state message
- notebook path or job id
- If the run succeeded, summarize which notebook or job was exercised and what evidence was collected.
### 8. Report the outcome
- State what was started locally.
- State whether load generation ran and with what iteration counts.
- State which Databricks job or notebook was executed.
- State whether notebook verification passed or failed.
- If it failed, include the exact failure message and the next corrective step.
### 9. Use the smoke test when the user wants one-command verification
- Use `scripts/smoke_test_notebooks.py`.
- It discovers or starts an ngrok tunnel, restarts Docker with the correct advertised Kafka listener, registers the connector, runs bounded load generators, triggers the Databricks job, and waits for terminal results.
- Pass `--reset --cluster-id <id>` to drop all tables and checkpoints before the smoke run.
```bash
# Full smoke test with table reset:
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
--job-id 123 \
--reset \
--cluster-id <cluster-id>
# Smoke test without reset (append to existing data):
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
--job-id 123
```
## Scripts
- `scripts/start_stack.sh`: start Docker Compose services for the lab
- `scripts/prepare_ngrok_kafka.py`: discover or start an ngrok TCP tunnel for Kafka and print the current public bootstrap
- `scripts/register_connector.sh`: register the Debezium connector from `postgres-connector.json`
- `scripts/run_generators.sh`: run film and rental/payment generators
- `scripts/reset_databricks_tables.py`: drop all Bronze/Silver/Gold Delta tables and clear streaming checkpoints
- `scripts/run_databricks_notebook.py`: launch or submit a Databricks run and poll to completion
- `scripts/smoke_test_notebooks.py`: run the end-to-end smoke test with dynamic ngrok bootstrap handling
- `scripts/migrate_and_run.py`: full migration script — drops legacy tables, updates the Databricks job via API, resets dvdrental tables, starts Docker+connector+generators, and triggers the job end-to-end
## References
- `references/repo-workflow.md`: repo-specific execution order, assumptions, and parameters
---
> Source: [alexeyban/databricks-lab](https://github.com/alexeyban/databricks-lab) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-05-22 -->
章节与文件索引
按说明文件中的顺序排列。
概述
1.首先检查repo输入
2. 调出本地CDC堆栈
3. 如果需要执行 CDC 摄取,请注册 Debezium 连接器
4. 运行负载生成器
文件/命令 -> docker-compose.yml · postgres-connector.json · scripts/start_stack.sh · docker compose up -d · docker compose ps · scripts/prepare_ngrok_kafka.py · KAFKA_EXTERNAL_HOST · KAFKA_EXTERNAL_PORT
内容校验值 -> a44f2c351990
当前原文的自动摘译,尚未人工审校。
概述
将此技能用于此存储库的操作循环:启动 Docker 服务、生成源表突变、重置 Databricks 表、执行 Databricks 笔记本或作业,以及验证笔记本运行是否成功完成。与重写 shell 命令相比,更喜欢捆绑脚本。它们对特定于存储库的路径和预期序列进行编码。
1.首先检查repo输入
确认
docker-compose.yml、postgres-connector.json,目标笔记本路径存在。如果您需要特定于存储库的序列或参数,请阅读references/repo-workflow.md。2. 调出本地CDC堆栈
使用
scripts/start_stack.sh。这将从存储库根运行docker compose up -d。如果用户要求验证,请执行docker compose ps或特定于服务的运行状况检查。如果必须通过内部网络边界从 Databricks 访问 Kafka,请首先使用scripts/prepare_ngrok_kafka.py,然后使用发现的KAFKA_EXTERNAL_HOST和KAFKA_EXTERNAL_PORT启动 Compose。3. 如果需要执行 CDC 摄取,请注册 Debezium 连接器
使用
scripts/register_connector.sh。仅在 Kafka Connect 接受请求后执行此操作。如果连接器已经存在,请清楚地报告,而不是将其视为致命故障。4. 运行负载生成器
使用
scripts/run_generators.sh。第一个参数是电影更新迭代,第二个参数是租赁/付款迭代。通过ITERATIONS优先选择有界运行进行验证;避免无限期的发电机,除非用户要求持续负载。示例:这运行 20 个电影突变和 40 个租赁/付款突变。