docetl

Engineering Community

Skill notes

Workflow Overview: Iterative Data Analysis

Work like a data analyst: write → run → inspect → iterate. Never write all scripts at once and run them all at once. Each phase should be completed and validated before moving to the next.

Phase 1: Data Collection

Write data collection script Run it immediately (with user permission) Inspect the dataset - show the user: Total document count Keys/fields in each document

Phase 2: Pipeline Development

Read sample documents to understand format Write pipeline YAML with sample: 10-20 for testing Run the test pipeline Inspect intermediate results - show the user: Extraction quality on samples

Phase 3: Visualization & Presentation

Write visualization script based on actual output structure Run and show the report to the user Iterate on charts/tables if needed Visualization Aesthetics: Clean and minimalist - no clutter, generous whitespace

Step 1: Data Preparation

DocETL datasets must be JSON arrays or CSV files.

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
Engineering
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
94 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@ucbepic · MIT
Token usage
Heavy
Setup complexity
Manual integration
External API key
Required · OpenAI / Anthropic / Gemini
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
Python
Detected file/system behavior
  • Read-only
  • Write / modify
  • Shell exec
  • Env read
Network behavior
External requests
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Output preview docetl.preview
# Bad outputs

- Read more input data examples to improve prompt specificity
- Add `validate` rules with retries
- Simplify output schema
- Add concrete examples to prompt

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.