docker-databricks-lab-ops
- Repo stars 0
- Author updated Live
- Author repo skills-registry
- Domain
- DevOps
- Compatible agents
-
- Claude Code
- Cursor
- Cline
- Codex
- Windsurf
- Gemini CLI
- +20
- Trust score
- 88 / 100 · community maintained
- Author / version / license
- @tomevault-io · no license declared
- Token usage
- Lean
- Setup complexity
- Manual integration
- External API key
- Not required
- Operating systems
- Docker
- Runtime requirements
- Docker
- Permissions
-
- Read-only
- Write / modify
- Shell exec
- Network behavior
- Local-only
- Install commands
- 26 variants
Profile is derived at build time from SKILL.md and install vectors. Subject to drift from author intent.
Heads up: 未限定 allowed-tools,默认拥有全部工具权限。
---
name: docker-databricks-lab-ops
description: Start and verify the local Docker CDC lab (dvdrental), run the PostgreSQL load generators, reset…
category: devops
runtime: Docker
---
# docker-databricks-lab-ops output preview
## PART A: Task fit
- Use case: Start and verify the local Docker CDC lab (dvdrental), run the PostgreSQL load generators, reset Databricks tables, trigger Databricks notebook jobs through databricks.sdk, and check whether Bronze/Silver notebooks completed successfully. Use when Codex needs to bring up the repo's local infrastructure, generate CDC traffic, reset or clean Databricks tables, execute Databricks jobs, poll run status, inspect failures, or validate notebook outputs for this lab. Use when this capability is needed..
- Inputs: target material, constraints, expected output, and acceptance criteria.
- Evidence boundary: follow “Overview / Workflow / 1. Inspect the repo inputs first” and do not present inference as author intent.
## PART B: Execution result
- **01** The card summarizes the use case; runtime output centers on “Start and verify the local Docker CDC lab (dvdrental), run the PostgreSQL load generators, reset Databricks tables, trigger Databricks notebook jobs through databricks.sdk, and check whether Bronze/Silver notebooks completed successfully. Use when Codex needs to bring up the repo's local infrastructure, generate CDC traffic, reset or clean Databricks tables, execute Databricks jobs, poll run status, inspect failures, or validate notebook outputs for this lab. Use when this capability is needed.”.
- **02** When the source has headings, the agent prioritizes “Overview / Workflow / 1. Inspect the repo inputs first” so the result follows the author’s structure.
- **03** Typical output includes task judgment, concrete steps, required commands or file edits, validation, and follow-up options.
- **04** Risk context follows the fingerprint: read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
## Running Rules
- read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding scope.
- Return the result, validation criteria, and next iteration options. The source mentions slash commands such as `/workspace`; use them first when your agent supports command triggers.
Name target files or source material, expected output, forbidden changes, and whether network or shell access is allowed. Permission fingerprint: read files, write/modify files, run shell commands.
Start with a small task and check whether the result follows “Overview / Workflow / 1. Inspect the repo inputs first”. Inspect diffs, logs, previews, or tests before expanding scope.
Confirm the final output includes a concrete result, evidence, and next action. If it stays generic, tighten inputs, boundaries, and acceptance criteria.
---
name: docker-databricks-lab-ops
description: Start and verify the local Docker CDC lab (dvdrental), run the PostgreSQL load generators, reset…
category: devops
source: tomevault-io/skills-registry
---
# docker-databricks-lab-ops
## When to use
- Start and verify the local Docker CDC lab (dvdrental), run the PostgreSQL load generators, reset Databricks tables, tr…
- Use it when the task has clear inputs, repeatable steps, and validation criteria.
## What to provide
- Target material, scope, expected result, and forbidden changes.
- Whether network, commands, file writes, or external services are allowed.
## Execution rules
- Organize steps around “Overview / Workflow / 1. Inspect the repo inputs first” and keep inference separate from source facts.
- read files, write/modify files, run shell commands; mostly runs locally; usually needs no extra API key.
- Validate with a small sample before expanding the task.
## Output requirements
- Return the deliverable, key evidence, validation method, and next action.
- Mark missing information as unknown; do not invent commands, platforms, or dependencies. The author source anchors workflow facts; repository files anchor sources and commands; Fluxly only adds fit, limitations, and quality judgment.
skill "docker-databricks-lab-ops" {
input -> user goal + target files + boundaries + acceptance criteria
context -> Overview / Workflow / 1. Inspect the repo inputs first
rules -> SKILL.md triggers / order / output contract
runtime -> Docker | read files, write/modify files, run shell commands | mostly runs locally
guardrails -> usually needs no extra API key + small-sample validation + diff/log review
output -> copyable result + checklist + next iteration
} Docker Databricks Lab Ops
Overview
Use this skill for the operational loop of this repository: bring up Docker services, generate source-table mutations, reset Databricks tables, execute a Databricks notebook or job, and verify whether the notebook run finished successfully.
Prefer the bundled scripts over rewriting shell commands. They encode the repository-specific paths and the expected sequence.
Workflow
1. Inspect the repo inputs first
- Confirm
docker-compose.yml,postgres-connector.json, and the target notebook paths exist. - Read references/repo-workflow.md if you need the repo-specific sequence or parameters.
2. Bring up the local CDC stack
- Use
scripts/start_stack.sh. - This runs
docker compose up -dfrom the repository root. - If the user asked for verification, follow with
docker compose psor service-specific health checks. - If Kafka must be reachable from Databricks through an internal network boundary, first use
scripts/prepare_ngrok_kafka.pyand then start Compose with the discoveredKAFKA_EXTERNAL_HOSTandKAFKA_EXTERNAL_PORT.
3. Register Debezium connector if CDC ingestion needs to be exercised
- Use
scripts/register_connector.sh. - Only do this after Kafka Connect is accepting requests.
- If the connector already exists, report that clearly instead of treating it as a fatal failure.
4. Run load generators
- Use
scripts/run_generators.sh. - The first argument is film update iterations, the second is rental/payment iterations.
- Prefer bounded runs for verification by passing
ITERATIONS; avoid indefinite generators unless the user asked for sustained load.
Example:
skills/docker-databricks-lab-ops/scripts/run_generators.sh 20 40
This runs 20 film mutations and 40 rental/payment mutations.
5. Reset Databricks tables (when starting fresh)
- Use
scripts/reset_databricks_tables.pyto drop all Bronze/Silver/Gold tables and clear streaming checkpoints before a fresh load. - Requires
--cluster-idor will submit via git source. - Use
--dry-runto preview what would be dropped without dropping.
python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
--cluster-id <cluster-id> \
--catalog workspace
# preview only:
python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
--cluster-id <cluster-id> \
--dry-run
6. Trigger a Databricks notebook job
- Use
scripts/run_databricks_notebook.py. - Provide either:
--job-idto run an existing Databricks job, or--notebook-pathand--cluster-idto submit a one-off notebook run.
- For dynamic Kafka exposure, pass
--notebook-param KAFKA_BOOTSTRAP=<ngrok-host:port>so the current tunnel endpoint is used at run time instead of a stale static value. - This script uses
DATABRICKS_HOSTandDATABRICKS_TOKENfrom the environment.
Examples:
python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
--job-id 123 \
--notebook-param KAFKA_BOOTSTRAP=0.tcp.eu.ngrok.io:12345
python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
--notebook-path /Workspace/agent/notebook \
--cluster-id 0123-456abc-cluster
7. Verify notebook behavior
- Treat the job as successful only when lifecycle is terminal and result is
SUCCESS. - On failure, capture:
run_id- lifecycle state
- result state
- state message
- notebook path or job id
- If the run succeeded, summarize which notebook or job was exercised and what evidence was collected.
8. Report the outcome
- State what was started locally.
- State whether load generation ran and with what iteration counts.
- State which Databricks job or notebook was executed.
- State whether notebook verification passed or failed.
- If it failed, include the exact failure message and the next corrective step.
9. Use the smoke test when the user wants one-command verification
- Use
scripts/smoke_test_notebooks.py. - It discovers or starts an ngrok tunnel, restarts Docker with the correct advertised Kafka listener, registers the connector, runs bounded load generators, triggers the Databricks job, and waits for terminal results.
- Pass
--reset --cluster-id <id>to drop all tables and checkpoints before the smoke run.
# Full smoke test with table reset:
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
--job-id 123 \
--reset \
--cluster-id <cluster-id>
# Smoke test without reset (append to existing data):
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
--job-id 123
Scripts
scripts/start_stack.sh: start Docker Compose services for the labscripts/prepare_ngrok_kafka.py: discover or start an ngrok TCP tunnel for Kafka and print the current public bootstrapscripts/register_connector.sh: register the Debezium connector frompostgres-connector.jsonscripts/run_generators.sh: run film and rental/payment generatorsscripts/reset_databricks_tables.py: drop all Bronze/Silver/Gold Delta tables and clear streaming checkpointsscripts/run_databricks_notebook.py: launch or submit a Databricks run and poll to completionscripts/smoke_test_notebooks.py: run the end-to-end smoke test with dynamic ngrok bootstrap handlingscripts/migrate_and_run.py: full migration script — drops legacy tables, updates the Databricks job via API, resets dvdrental tables, starts Docker+connector+generators, and triggers the job end-to-end
References
references/repo-workflow.md: repo-specific execution order, assumptions, and parameters
Source: alexeyban/databricks-lab — distributed by TomeVault.
Decide Fit First
Design Intent
How To Use It
Boundaries And Review