# Error handling, retries, and loops > Source: https://docs.clonepartner.com/building/error-handling/ # Error handling, retries, and loops This guide covers the three task YAML blocks that control failure behavior and execution lifetime: per-step `error_handling`, the task-level `loop` block, and the task-level `masking` block. Configure explicit bounds and test failure behavior before running against production-scale data. ## Per-step `error_handling` Every step accepts an `error_handling` key with one of three values. There is no per-step retry count — retries happen by rerunning the task, which is why idempotent writes matter (see [Idempotent sync and the mapping DB](/building/idempotent-sync/)). ### `fail` (default) The error stops the entire task. The Run fails and the error is logged with the task's failure entry. ```yaml - name: create_ticket type: call_method error_handling: fail config: | { "connector": "destination", "method": "create", "args": { "resource": "tickets", "data": input.payload } } ``` ### `ignore` The error is logged as a warning (`Step failed (ignored)`, with `err`, `errorDetail`, `input`, and `config` fields) and the item continues downstream with the failed step's output as `undefined`. Other items are unaffected. ```yaml - name: create_ticket type: call_method error_handling: ignore config: | { "connector": "destination", "method": "create", "args": { "resource": "tickets", "data": input.payload } } ``` ### `ignore_rest` The error is logged as a warning (`Step failed (skipping rest)`) and the remaining pipeline is skipped for the affected item. Other items continue normally. ```yaml - name: create_ticket type: call_method error_handling: ignore_rest config: | { "connector": "destination", "method": "create", "args": { "resource": "tickets", "data": input.payload } } ``` ### Guard downstream steps after `ignore` With `error_handling: ignore`, a failed step emits `undefined`, so a downstream success marker would otherwise run with an empty result. Gate it with `skip_if` and record failures separately: ```yaml - name: mark_synced type: call_method skip_if: "$not(steps.create_ticket.output.id)" config: | { "connector": "mapping_db", "method": "update", "args": { "table": "tickets", "data": { "destination_id": $string(steps.create_ticket.output.id) }, "filter": { "id": steps.build_payload.output.source_id } } } ``` Never write a mapping or success marker from an empty create result. ## Task-level `loop` A top-level `loop` block reruns the task's main pipeline repeatedly within a single Run — for polling, delta drains, and daemon-style workloads. All keys come from the task parser: | Key | Type | Required | Default | Description | |---|---|---|---|---| | `sleep_ms` | integer | Yes | — | Milliseconds to sleep between iterations. Must be at least 10000. | | `check_num_times` | integer | No | 1 | Consecutive false continuation checks required before exiting. | | `max_iterations` | integer | No | unlimited | Safety cap on main pipeline runs. | | `max_duration_ms` | integer | No | unlimited | Wall-clock limit from task start. | | `steps` | array | No | — | Steps that decide continuation; the final step's output is coerced to a boolean. | | `until_empty` | object | No | — | `{ from_step: "" }` — continue while that step returns rows. | Without `steps` or `until_empty`, the loop reruns the pipeline unconditionally until aborted or a cap is hit. Always set `max_iterations` or `max_duration_ms` on an unconditional loop. ### Polling loop example A complete task that pushes pending rows and keeps looping while any remain: ```yaml name: push-pending-tickets description: Push staged tickets to the destination until none remain output: none connectors: destination: type: truto staging: type: mongodb steps: - name: find_pending type: call_method config: connector: staging method: find args: collection: tickets filter: destination_id: $exists: false - name: push_ticket type: call_method error_handling: ignore config: | { "connector": "destination", "method": "proxy_create", "args": { "resource": "tickets", "data": input } } loop: sleep_ms: 15000 check_num_times: 3 max_duration_ms: 3600000 steps: - name: check_remaining type: call_method config: connector: staging method: find_one args: collection: tickets filter: destination_id: $exists: false - name: should_continue type: transform config: "steps.check_remaining.output != null" ``` After each pipeline drain, Envoy sleeps `sleep_ms`, runs the loop steps, and reruns the pipeline when the final output is truthy. A falsy result increments a false counter; the loop exits once it reaches `check_num_times`. Inside `loop.steps`, JSONata can read `variables.envoy_iteration`, `variables.envoy_last_iteration_at`, and `variables.envoy_false_count`. ### `until_empty` Instead of writing check steps, reference an existing main step that returns remaining work: ```yaml loop: sleep_ms: 15000 until_empty: from_step: find_pending ``` The named step is re-executed in isolation each iteration; truthy output means more work remains. Because it reruns every iteration, make `from_step` a lightweight existence check — a `find` with `limit: 1` or a `count` — not a full extraction. Use `output: none` or `ndjson` for looped tasks. `output: json` accumulates all items from all iterations in memory before returning. ## Task-level `masking` The `masking` block controls the `$mask` family of JSONata functions (`$mask`, `$maskEmail`, `$maskPhone`, `$maskName`, `$maskText`, `$maskingEnabled`): ```yaml masking: enabled: ${MASK_PII} seed: ${MASK_SEED} ``` - `enabled` — boolean toggle. Environment variables here resolve leniently: an unset variable means `false`, so the same task YAML is safe in production where the toggle is not defined. - `seed` — optional string that makes masking deterministic: the same value and seed always produce the same token, so masked test data stays joinable across tasks. When masking is disabled, every `$mask*` call returns its input unchanged. Use masking when running real migration tasks against test environments without leaking production PII. Deterministic masking preserves equality and is guessable for low-entropy values — it protects test data but is not cryptographic anonymization. See [Working with JSONata](/building/working-with-jsonata/) for the function signatures. ## How failures surface at run level A step failure with `error_handling: fail` fails the Run; ignored failures appear as warnings in the Run's logs and the Run itself completes. See [Run lifecycle](/concepts/run-lifecycle/) for run states. When the task runs inside a Job, the Job Template's `on_failure` policy (`pause`, `abort`, or `continue`) decides what happens to the rest of the DAG when a task Run fails. See [Job templates and DAGs](/building/job-templates-and-dags/). ## Verify it worked Force a failure on a bounded input — for example, point the destination at an invalid resource for one record — and confirm: - With `fail`, the Run fails immediately and the error appears in the logs. - With `ignore`, the log shows `Step failed (ignored)` at warn level, the remaining items complete, and any guarded marker steps were skipped for the failed item. - A loop exits when its continuation check stays false `check_num_times` times, or when `max_iterations` or `max_duration_ms` is reached. :::callout{type="warning"} Stopping a loop does not reverse completed connector calls. Design every write inside a loop to be idempotent so a stop-and-restart cannot duplicate records. ::: ## Related - [Idempotent sync and the mapping DB](/building/idempotent-sync/) — make ignored failures safe to rerun - [Testing and debugging tasks](/building/testing-and-debugging/) — where ignored errors are logged and how to inspect them - [Task YAML schema](/reference/task-yaml/) — every top-level and step-level key - [Job templates and DAGs](/building/job-templates-and-dags/) — `on_failure` behavior across a DAG