Error handling, retries, and loops

This guide covers the three task YAML blocks that control failure behavior and execution lifetime: per-step error_handling, the task-level loop block, and the task-level masking block. Configure explicit bounds and test failure behavior before running against production-scale data.

Per-step error_handling

Every step accepts an error_handling key with one of three values. There is no per-step retry count — retries happen by rerunning the task, which is why idempotent writes matter (see Idempotent sync and the mapping DB).

fail (default)

The error stops the entire task. The Run fails and the error is logged with the task's failure entry.

- name: create_ticket
  type: call_method
  error_handling: fail
  config: |
    {
      "connector": "destination",
      "method": "create",
      "args": { "resource": "tickets", "data": input.payload }
    }

ignore

The error is logged as a warning (Step failed (ignored), with err, errorDetail, input, and config fields) and the item continues downstream with the failed step's output as undefined. Other items are unaffected.

- name: create_ticket
  type: call_method
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "create",
      "args": { "resource": "tickets", "data": input.payload }
    }

ignore_rest

The error is logged as a warning (Step failed (skipping rest)) and the remaining pipeline is skipped for the affected item. Other items continue normally.

- name: create_ticket
  type: call_method
  error_handling: ignore_rest
  config: |
    {
      "connector": "destination",
      "method": "create",
      "args": { "resource": "tickets", "data": input.payload }
    }

Guard downstream steps after ignore

With error_handling: ignore, a failed step emits undefined, so a downstream success marker would otherwise run with an empty result. Gate it with skip_if and record failures separately:

- name: mark_synced
  type: call_method
  skip_if: "$not(steps.create_ticket.output.id)"
  config: |
    {
      "connector": "mapping_db",
      "method": "update",
      "args": {
        "table": "tickets",
        "data": { "destination_id": $string(steps.create_ticket.output.id) },
        "filter": { "id": steps.build_payload.output.source_id }
      }
    }

Never write a mapping or success marker from an empty create result.

Task-level loop

A top-level loop block reruns the task's main pipeline repeatedly within a single Run — for polling, delta drains, and daemon-style workloads. All keys come from the task parser:

Without steps or until_empty, the loop reruns the pipeline unconditionally until aborted or a cap is hit. Always set max_iterations or max_duration_ms on an unconditional loop.

Polling loop example

A complete task that pushes pending rows and keeps looping while any remain:

name: push-pending-tickets
description: Push staged tickets to the destination until none remain
output: none
 
connectors:
  destination:
    type: truto
  staging:
    type: mongodb
 
steps:
  - name: find_pending
    type: call_method
    config:
      connector: staging
      method: find
      args:
        collection: tickets
        filter:
          destination_id:
            $exists: false
 
  - name: push_ticket
    type: call_method
    error_handling: ignore
    config: |
      {
        "connector": "destination",
        "method": "proxy_create",
        "args": { "resource": "tickets", "data": input }
      }
 
loop:
  sleep_ms: 15000
  check_num_times: 3
  max_duration_ms: 3600000
  steps:
    - name: check_remaining
      type: call_method
      config:
        connector: staging
        method: find_one
        args:
          collection: tickets
          filter:
            destination_id:
              $exists: false
    - name: should_continue
      type: transform
      config: "steps.check_remaining.output != null"

After each pipeline drain, Envoy sleeps sleep_ms, runs the loop steps, and reruns the pipeline when the final output is truthy. A falsy result increments a false counter; the loop exits once it reaches check_num_times. Inside loop.steps, JSONata can read variables.envoy_iteration, variables.envoy_last_iteration_at, and variables.envoy_false_count.

until_empty

Instead of writing check steps, reference an existing main step that returns remaining work:

loop:
  sleep_ms: 15000
  until_empty:
    from_step: find_pending

The named step is re-executed in isolation each iteration; truthy output means more work remains. Because it reruns every iteration, make from_step a lightweight existence check — a find with limit: 1 or a count — not a full extraction.

Use output: none or ndjson for looped tasks. output: json accumulates all items from all iterations in memory before returning.

Task-level masking

The masking block controls the $mask family of JSONata functions ($mask, $maskEmail, $maskPhone, $maskName, $maskText, $maskingEnabled):

masking:
  enabled: ${MASK_PII}
  seed: ${MASK_SEED}

• enabled — boolean toggle. Environment variables here resolve leniently: an unset variable means false, so the same task YAML is safe in production where the toggle is not defined.
• seed — optional string that makes masking deterministic: the same value and seed always produce the same token, so masked test data stays joinable across tasks.

When masking is disabled, every $mask* call returns its input unchanged. Use masking when running real migration tasks against test environments without leaking production PII. Deterministic masking preserves equality and is guessable for low-entropy values — it protects test data but is not cryptographic anonymization. See Working with JSONata for the function signatures.

How failures surface at run level

A step failure with error_handling: fail fails the Run; ignored failures appear as warnings in the Run's logs and the Run itself completes. See Run lifecycle for run states.

When the task runs inside a Job, the Job Template's on_failure policy (pause, abort, or continue) decides what happens to the rest of the DAG when a task Run fails. See Job templates and DAGs.

Verify it worked

Force a failure on a bounded input — for example, point the destination at an invalid resource for one record — and confirm:

• With fail, the Run fails immediately and the error appears in the logs.
• With ignore, the log shows Step failed (ignored) at warn level, the remaining items complete, and any guarded marker steps were skipped for the failed item.
• A loop exits when its continuation check stays false check_num_times times, or when max_iterations or max_duration_ms is reached.

Related

• Idempotent sync and the mapping DB — make ignored failures safe to rerun
• Testing and debugging tasks — where ignored errors are logged and how to inspect them
• Task YAML schema — every top-level and step-level key
• Job templates and DAGs — on_failure behavior across a DAG