Step types

Step availability is determined by the deployed Envoy build. The following built-in types cover connector calls, transformation, filtering, mapping state, database writes, validation, and file transfer.

Common properties

- name: step_name
  type: transform
  run_if: "input.enabled = true"
  error_handling: fail
  concurrency: 1
  debug: false
  config: |
    input

config can be a static YAML object or a JSONata expression string. run_if and skip_if are JSONata expressions.

General pipeline steps

call_method

Calls a method on a mapped connector. Use it for list, read, create, update, delete, and connector-specific operations.

type: call_method
config: |
  {
    "connector": "source",
    "method": "find",
    "args": {
      "table": "tickets",
      "limit": 100
    }
  }

Collection methods can stream results. Prefer this step to $callMethod() for large results.

transform

Evaluates JSONata and emits the transformed value:

type: transform
config: |
  {
    "id": $string(input.id),
    "email": $lowercase(input.email)
  }

An emitted array becomes multiple downstream items.

filter

Keeps the item when its expression is truthy and drops it otherwise:

type: filter
config: "input.active = true"

console

Writes an inspection representation for debugging and passes the item through. Keep it disabled or remove it from high-volume production tasks.

Mapping and change detection

compute_hash

Computes a deterministic hash from selected or transformed fields. Normalize values before hashing and use identical logic across initial and delta tasks.

get_destination_id

Looks up an existing source-to-destination mapping and adds the mapped destination identity to the pipeline context.

set_destination_id

Persists a source-to-destination mapping after a confirmed destination write. Guard it against missing write output.

sync_record

Combines mapping lookup, payload hash comparison, destination create/update, and mapping persistence. Prefer it for standard idempotent synchronization.

See Mapping database and sync_record.

Database-oriented steps

ensure_table

Creates or verifies a table used by a task. Keep schemas explicit and run setup before concurrent writes.

upsert_record

Inserts or updates a record according to declared conflict columns. Back the conflict key with a destination uniqueness rule.

lookup

Performs a bounded lookup to enrich or resolve a record. Avoid using it for an unindexed scan per source item.

mark_processed

Records processing state for a source or staging record. Invoke it only after the relevant external operation is known to have succeeded.

Field and source handling

map_fields

Maps source fields to destination fields and can apply per-field transforms. It accepts static mapping or job configuration prepared by a field-mapping editor.

validate_source

Checks required source fields or conditions before later work. Choose whether invalid items should fail the run or move to an explicit exception path.

transfer_file

Streams or transfers a file between compatible connectors. Use a stable destination key, bound file sizes where supported, and ensure container-visible paths.

Batch and spool modifiers

batch and spool are common step wrappers rather than separate step types:

batch: 100

Batching sends arrays downstream. Spooling collects the whole incoming stream and can consume unbounded memory or disk depending on runtime behavior.

Error handling

• fail stops the task.
• ignore logs the error and continues other items.
• ignore_rest stops processing the affected item through later steps.

Use conditional guards after ignored writes. See Masking, loops, and error handling.