Skip to content
Back to

Workflow orchestration

On the Flow canvas, you design visual pipelines that move data from source to outcome. The workflow engine runs them reactively on the server: when inputs or configuration change, affected nodes re-run while independent branches execute in parallel. You reuse work through templates and subgraphs, and every run can leave audit and lineage metadata for compliance review. See Compliance evidence for data lineage and Data for connectors, storage, and file formats.

Visual pipeline model

Section titled “Visual pipeline model”

A workflow is a directed graph of nodes connected by edges. Each node performs one step and passes typed dataset ports downstream.

Execution engine

Section titled “Execution engine”

When you trigger a run, the engine:

  • Reacts to change - hash-driven scheduling re-runs nodes when inputs or configuration change
  • Runs in parallel - independent branches execute concurrently where the graph allows
  • Executes server-side - large files, PDF parsing, connector fetch, and PyAirbyte jobs run on the workflow engine
  • Persists run-scoped data - uploads, Arrow datasets, exports, and audit sidecars survive pod restarts within your workspace boundary
Validations

Apply schema, business-rule, and tolerance checks as deterministic steps; failures route to owners with full context attached.

Transformations

Reshape, normalise, and map payloads between source formats and target schemas with versioned transformation logic.

Compute

Run grouping, filtering, aggregations, enrichment, and other compute steps on governed datasets inside the workflow boundary.

Node families

Section titled “Node families”

The sidebar under Workflows → Nodes documents each node type. Representative families:

Operations and validations

Section titled “Operations and validations”

The Operations node exposes column-level transforms (strings, dates, decimals, conditionals, lookups). You combine visual operation stacks in the UI or use natural-language compile (Governed AI) to generate operations from plain English.

The sidebar catalogues Transformations, Validations, and Conditions you attach to Operations nodes. Use them to enforce data quality before data leaves a pipeline branch.

Document processing

Section titled “Document processing”

PDF and office documents uploaded to File Input are parsed server-side via Docling on the platform cluster (shared document-parse service for all workspaces). Optional AI structuring enriches results for review in the UI.

Subgraphs and templates

Section titled “Subgraphs and templates”
  • Save workflows as templates and share them within teams or the community catalog
  • Encapsulate reusable fragments as subgraphs and graft them onto larger pipelines
Section titled “Related documentation”
  • Security - platform security controls and SOC 2 summary
  • Compliance evidence - atomic data lineage and workflow file store
  • Data - File Input, PyAirbyte connectors, Arrow storage, REST triggers, exports
  • Governed AI - AI transforms and natural-language operation compile
  • Observability - run telemetry; canvas port audit items vs security audit ledger