# Steps overview

> Source: https://docs.clonepartner.com/reference/steps/

<!-- sources: src/steps/index.ts, src/steps/types.ts, references/envoy/step-types.md, references/envoy/prompt/decision-table.md -->

# Steps overview

A step is one stage of a task pipeline. Each entry in a task's `steps` array names a step, picks a registered type, and supplies a `config` — a static YAML object or a JSONata expression evaluated per item. Items flow through the steps as a single linear stream with backpressure. Where steps sit in the file, and the keys every step shares, are defined in the [Task YAML schema](/reference/task-yaml/).

## Step types

Envoy registers 15 step types, listed here in registry order:

| Type | Purpose |
|---|---|
| [call_method](/reference/steps/call-method/) | Call a Connector method — the primary step for all I/O. |
| [transform](/reference/steps/transform/) | Reshape each item with a JSONata expression. |
| [filter](/reference/steps/filter/) | Keep or drop items with a boolean JSONata predicate. |
| [console](/reference/steps/console/) | Print each item to stderr for debugging; pass it through unchanged. |
| [compute_hash](/reference/steps/compute-hash/) | Add a content hash field to each record for change detection. |
| [map_fields](/reference/steps/map-fields/) | Declarative source-to-destination field mapping for migrations. |
| [get_destination_id](/reference/steps/get-destination-id/) | Look up a source-to-destination ID mapping from the mapping store. |
| [set_destination_id](/reference/steps/set-destination-id/) | Upsert a source-to-destination ID mapping. |
| [ensure_table](/reference/steps/ensure-table/) | Run SQL DDL to bootstrap tables. |
| [upsert_record](/reference/steps/upsert-record/) | Declarative SQL or MongoDB upsert. |
| [sync_record](/reference/steps/sync-record/) | Composite dedup, hash check, create/update, and mapping persistence. |
| [lookup](/reference/steps/lookup/) | Cross-store join or foreign-key resolution with a fallback chain. |
| [mark_processed](/reference/steps/mark-processed/) | Write a flag, destination ID, or hash back to the source row. |
| [validate_source](/reference/steps/validate-source/) | Validate primary keys and required columns in source data. |
| [transfer_file](/reference/steps/transfer-file/) | Download a file from one Connector and upload it to another. |

## Choosing a step

- **Pull a whole resource or stream many records** — use a `call_method` step as the pipeline source. It streams array results with backpressure. Do not wrap list methods in the `$callMethod` JSONata function, which buffers the entire result into memory.
- **Migrate records idempotently** — prefer `sync_record` over the manual `get_destination_id` + `skip_if` + `set_destination_id` triad. It combines mapping lookup, hash-based change detection, create/update, and mapping persistence in one step.
- **Remove items from the stream** — use `filter`. A `transform` that evaluates to `null` passes the input through unchanged, and `run_if`/`skip_if` skip a step but keep the item flowing.
- **Call a connector per item and keep the item's fields** — use `$callMethod` inside a `transform`, for bounded single-record lookups only (`find_one`, `count`, grouped `aggregate`).
- **Pull multiple independent resource types** — one task per resource, combined in a Job. A task pipeline is a single linear stream with one source.
- **Stage or upsert rows to SQL or MongoDB** — `upsert_record`; bootstrap the tables first with `ensure_table`.
- **Write a synced flag or destination ID back to the source** — `mark_processed`.
- **Move attachments or other files between systems** — `transfer_file`.

## Shared step keys

Every step, regardless of type, accepts `run_if`, `skip_if`, `error_handling`, `concurrency`, `batch`, `spool`, and `debug` alongside `name`, `type`, and `config`. These keys are documented once in the [Task YAML schema](/reference/task-yaml/) — step pages only cover their type-specific `config`.

## Related

- [Task YAML schema](/reference/task-yaml/) — the full task file schema, including per-step keys
- [JSONata reference](/reference/jsonata/) — the expression language used in step configs
- [Migration playbook](/building/migration-playbook/) — patterns combining these steps end to end