Documentation by ClonePartner

Start typing to search.

Task Reference

Steps overview

View Markdown

Browse every built-in step type, what each one does, and how to choose the right step for a pipeline stage.

Steps overview

A step is one stage of a task pipeline. Each entry in a task's steps array names a step, picks a registered type, and supplies a config — a static YAML object or a JSONata expression evaluated per item. Items flow through the steps as a single linear stream with backpressure. Where steps sit in the file, and the keys every step shares, are defined in the Task YAML schema.

Step types

Envoy registers 15 step types, listed here in registry order:

Type Purpose
call_method Call a Connector method — the primary step for all I/O.
transform Reshape each item with a JSONata expression.
filter Keep or drop items with a boolean JSONata predicate.
console Print each item to stderr for debugging; pass it through unchanged.
compute_hash Add a content hash field to each record for change detection.
map_fields Declarative source-to-destination field mapping for migrations.
get_destination_id Look up a source-to-destination ID mapping from the mapping store.
set_destination_id Upsert a source-to-destination ID mapping.
ensure_table Run SQL DDL to bootstrap tables.
upsert_record Declarative SQL or MongoDB upsert.
sync_record Composite dedup, hash check, create/update, and mapping persistence.
lookup Cross-store join or foreign-key resolution with a fallback chain.
mark_processed Write a flag, destination ID, or hash back to the source row.
validate_source Validate primary keys and required columns in source data.
transfer_file Download a file from one Connector and upload it to another.

Choosing a step

  • Pull a whole resource or stream many records — use a call_method step as the pipeline source. It streams array results with backpressure. Do not wrap list methods in the $callMethod JSONata function, which buffers the entire result into memory.
  • Migrate records idempotently — prefer sync_record over the manual get_destination_id + skip_if + set_destination_id triad. It combines mapping lookup, hash-based change detection, create/update, and mapping persistence in one step.
  • Remove items from the stream — use filter. A transform that evaluates to null passes the input through unchanged, and run_if/skip_if skip a step but keep the item flowing.
  • Call a connector per item and keep the item's fields — use $callMethod inside a transform, for bounded single-record lookups only (find_one, count, grouped aggregate).
  • Pull multiple independent resource types — one task per resource, combined in a Job. A task pipeline is a single linear stream with one source.
  • Stage or upsert rows to SQL or MongoDB — upsert_record; bootstrap the tables first with ensure_table.
  • Write a synced flag or destination ID back to the source — mark_processed.
  • Move attachments or other files between systems — transfer_file.

Shared step keys

Every step, regardless of type, accepts run_if, skip_if, error_handling, concurrency, batch, spool, and debug alongside name, type, and config. These keys are documented once in the Task YAML schema — step pages only cover their type-specific config.