Task structure and execution model

A task is a YAML-defined streaming pipeline. At runtime, Envoy resolves connector slots, validates arguments, builds the step chain, and passes each source result through the remaining steps with backpressure.

Top-level fields

name: tickets/migrate
description: Migrate tickets
output: none
 
connectors:
  source:
    type: truto
  destination:
    type: postgres
 
arguments:
  account_id:
    type: string
    required: true
 
variables:
  destination_table:
    type: string
    value: tickets
    description: Destination table name
 
steps:
  - name: list
    type: call_method
    config: {}

Connector slots do not contain credentials. Server runs resolve them from a stored connector mapping; CLI runs resolve them from connector configuration.

Step fields

Every step has:

• name — unique within the task;
• type — registered step type;
• config — YAML object or JSONata expression string.

Common optional fields:

• run_if and skip_if;
• error_handling;
• concurrency;
• batch;
• spool;
• debug.

See Step types.

Execution context

JSONata expressions can read:

• input — the current item;
• arguments — validated task arguments;
• variables — top-level constants;
• job_config — the job-run configuration snapshot;
• steps.<name>.output — output captured from an earlier step for the current item;
• steps.<name>.error — earlier step error information where available.

Use JSONata reference for expression rules.

Pipeline behavior

The first streaming step produces records. Each later step receives one current item and emits:

• one value, which becomes the next input;
• an array, which is unwrapped into individual downstream items;
• no value, which follows the step contract for that type.

Use a filter step to remove items intentionally. A skipped step passes its current input through unchanged.

call_method can stream an async connector result. Backpressure limits how quickly upstream pages are consumed.

Concurrency, batch, and spool

concurrency permits parallel handling at that step while preserving bounded flow. Consider provider limits and write idempotency.

batch groups items into fixed-size arrays. Downstream configuration must expect an array.

spool collects the complete incoming stream before continuing. It is unbounded and should be reserved for an operation that truly needs the entire dataset.

Loop blocks

A top-level loop repeats the complete main pipeline after a required sleep. It can run continuously, stop after continuation steps return false, or use until_empty with a lightweight named-step check. Use iteration or duration caps for bounded operations.

See Masking, loops, and error handling.

Output formats

• none suppresses final data output; use it for side-effect pipelines.
• ndjson emits one JSON object per line and is suitable for large streams.
• json collects final results into one JSON array and should be limited to bounded output.

Debug messages go to stderr and structured application logs, not the task data stream.

Snapshots

Standalone task runs snapshot the task version and connector mapping. A job snapshots its template and current task definitions when the job is created. Later edits do not mutate an existing run.

Validation boundaries

Static validation checks YAML shape, known connector and step types, references, argument definitions, and expression compilation. It cannot prove:

• credentials are valid;
• fields exist in live data;
• external permissions are sufficient;
• expressions return the intended runtime type;
• writes are idempotent.

Follow validation with a bounded runtime test. See Author a production task.