Task anatomy and authoring workflow

A Task is a self-contained YAML file that defines one streaming pipeline: connector slots, typed inputs, and an ordered list of steps that each record flows through. This page walks through a complete task file and the write → validate → debug loop for authoring one.

Prerequisites

• The envoy CLI, or access to the Task editor in the web UI
• A Connector configuration for each connector type your task needs (stored in the server, or in a connectors.yaml file for CLI runs)

A complete annotated task

name: stage-contacts
description: Read contacts from CSV and upsert them into SQLite
output: none            # ndjson (default), json, or none
 
connectors:             # slots: name + type only, never credentials
  source:
    type: csv
  staging:
    type: sqlite
 
arguments:              # caller-supplied at run time
  dry_run:
    type: boolean
    default: true
    description: Skip the write step when true
 
variables:              # typed values defined inside the task
  target_status:
    type: string
    value: active
    description: Status to keep
 
steps:
  - name: read_contacts
    type: call_method
    config:             # static object — no input reference needed
      connector: source
      method: read
      args:
        resource: contacts
 
  - name: only_target_status
    type: filter
    config: "input.status = variables.target_status"
 
  - name: normalize
    type: transform
    config: |
      {
        "id": $string(input.id),
        "name": $trim(input.name),
        "email": $lowercase(input.email),
        "status": input.status
      }
 
  - name: write_contact
    type: call_method
    skip_if: "arguments.dry_run"
    config: |
      {
        "connector": "staging",
        "method": "upsert",
        "args": {
          "table": "contacts",
          "data": input,
          "conflict_columns": ["id"]
        }
      }

The top-level keys, all parsed by the task parser:

Each step takes name, type, and config, plus the optional keys run_if, skip_if, error_handling, concurrency, batch, spool, debug, and cursor. A step config is either a static YAML object (passed to the step as-is) or a JSONata string (evaluated per item — required whenever the config references input, arguments, variables, steps, or job_config). See the Task YAML schema for the exhaustive key-by-key reference.

output is a top-level key only. There is no per-step output control — the validator warns if you put output (or quiet, silent, and similar keys) on a step.

The authoring workflow

1. Write the YAML. Start from the source step, confirm its output shape, then add downstream steps one at a time.

2. Validate it. From the CLI:

   envoy validate stage-contacts

   Validation checks YAML syntax, task structure, step types, connector references, and that every JSONata expression compiles. It does not run anything: expressions that compile can still fail at evaluation time against a real record.

3. Run it against real data with debug output:

   envoy stage-contacts --config connectors.yaml --debug

   --debug prints each step's input, config, and output to stderr. Combine it with a dry_run-style argument (as in the example above) so the write step is skipped while you inspect the transformed records.

4. Iterate: adjust expressions, re-validate, re-run. Enable writes only once the debug output looks right for a small, representative set of records.

In the web UI, the Task editor gives you the same loop: edit the YAML, validate, and trigger a run from the task's detail page. See Tasks and task runs.

Preserve streaming

Every item flows through the pipeline independently — the source step's records are not collected into memory before downstream steps run. Design tasks to keep that property:

• Return-value contract. A step that returns an array fans out: each element is emitted downstream as a separate item. Returning undefined or [] drops the item. Connector methods that return collections (find, read, list) stream each record as it arrives.
• concurrency sets how many items a step processes in parallel. It uses backpressure — upstream is paused when the limit is reached, so memory stays bounded.
• batch collects a step's output into arrays of N before pushing downstream (the final batch may be smaller). Use it for bulk writes.
• spool collects all output into a single array, emitted once when the stream ends. It is intentionally unbounded — use it only when a downstream step genuinely needs the whole dataset (for example, counting or aggregation).
• One task, one source. A pipeline is a single linear stream, so pull independent resources in separate tasks and combine them with a Job template.

Conditional step execution

run_if and skip_if are JSONata expressions evaluated per item. When run_if is false or skip_if is true, the step's processing is skipped and the item passes through to the next step unchanged — it is not removed from the stream. To drop items, use a filter step instead (see Transforming and filtering).

- name: write_contact
  type: call_method
  skip_if: "arguments.dry_run"

Setting both run_if and skip_if on one step produces a validation warning; it is almost always unintentional.

Common validation errors

Messages come verbatim from the validator:

• Task must have a steps array — the file has no steps key, or steps is not a YAML list. Every task needs at least one step.
• Duplicate step name: 'write_contact' — two steps share a name. Step names must be unique because they key steps.{name}.output references.
• Connector 'staging' not defined in task — a step config references a connector slot that is missing from the top-level connectors block. Add the slot or fix the name in the config.

Verify it worked

A valid task prints a pass summary and exits with code 0:

envoy validate stage-contacts

Validating task: stage-contacts

  PASS  YAML syntax valid
  PASS  Task structure valid (4 steps)
  PASS  Step types valid
  PASS  Connector references valid
  PASS  JSONata expressions compile

Task is valid!

Related

• Task YAML schema — every top-level and step key in detail
• Reading and writing data — call_method, streaming, and batching
• Working with JSONata — the expression language used in step configs
• Tasks and task runs — the Task editor and run surfaces in the UI