Migration playbook

This guide walks through building a real migration — helpdesk tickets from a source system to a destination — the way production Envoy migrations are structured. Use it when you are planning a multi-resource migration and need to know which tasks to build, in what order, and which steps to reach for.

The shape is always the same: pull each resource into a staging store, validate and transform, push idempotently to the destination while recording what happened, then run bounded delta passes until cutover.

Prerequisites

• Source and destination connectors, plus a staging store (a mongodb or SQL connector) and, if you use the mapping steps, a mapping DB.
• Idempotent sync and the mapping DB — the write patterns this playbook builds on.
• Error handling, retries, and loops — error_handling: ignore and guarded markers.

Plan resources and order

Migrate parents before children: users and contacts before tickets, tickets before comments and attachments. Each independent resource gets its own pull task and its own push task — a task pipeline is a single linear stream with one source, so independent resources cannot share a task. Combine the tasks into a Job Template DAG so independent pulls run in parallel and pushes depend on the pulls they need (see Job templates and DAGs).

A dependent pull — comments fetched per pulled ticket — stays inside one task: the parent call_method list step streams tickets, and a per-item call_method step referencing input fetches each ticket's children.

Stage the source data

Pull each resource into a staging collection, stamped with scope fields so reruns upsert instead of duplicating. This excerpt is adapted from a real Zendesk ticket pull; the source is an incremental-export list call, and the write is an upsert keyed on the source ID:

name: pull-tickets
description: Pull source tickets into the staging store (rerun-safe)
output: none
 
connectors:
  source:
    type: truto
  staging:
    type: mongodb
 
arguments:
  start_time:
    type: number
    default: 0
    description: Unix timestamp for incremental-export start_time
 
steps:
  - name: list_tickets
    type: call_method
    config: |
      {
        "connector": "source",
        "method": "proxy_list",
        "args": {
          "resource": "incremental-export",
          "query": {
            "start_time": arguments.start_time > 0 ? arguments.start_time : 0
          }
        }
      }
 
  - name: upsert_ticket
    type: call_method
    config: |
      {
        "connector": "staging",
        "method": "update_one",
        "args": {
          "collection": "tickets",
          "filter": { "id": input.id },
          "update": {
            "$set": $merge([input, { "synced_at": { "$date": $now() } }]),
            "$setOnInsert": { "comments_synced": false }
          },
          "upsert": true
        }
      }

Incremental re-pulls pass a later start_time; the upsert makes every pull rerun-safe regardless.

Validate before you push

Run a validate_source step against staged data (or the source CSV) before any destination write. It checks primary_key uniqueness and required_columns presence, with a mode controlling whether problems fail the task or are reported. Catching a duplicate primary key here is far cheaper than discovering duplicate destination records later. See validate_source.

Shape the payload

Two steps cover most transformation work:

• map_fields — declarative field renames with optional per-field transform and skip expressions, driven by a mapping config.
• transform — a full JSONata expression when the payload needs computed values, tag building, or conditionals.

Resolve cross-store references — "this ticket's requester is which destination contact?" — with a lookup step: a keys fallback chain where each entry names a connector and a match, with required dropping items that resolve nothing.

Push idempotently and record the outcome

Prefer sync_record for the write: mapping lookup, hash-based change detection, create or update, and mapping upsert in one step.

Production push tasks that need custom payload assembly use the guarded three-step shape instead. This excerpt is condensed from a real Gorgias ticket push. The transform carries the source ID and hash forward, because after create_ticket runs, input is the create response and the original item's fields are gone from it:

- name: build_payload
  type: transform
  config: |
    {
      "ticket_payload": {
        "subject": input.subject,
        "status": input.status,
        "external_id": $string(input.id)
      },
      "hash": $hash({ "subject": input.subject, "status": input.status, "updated_at": input.updated_at }),
      "source_id": input.id
    }
 
- name: create_ticket
  type: call_method
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "proxy_create",
      "args": { "resource": "tickets", "data": input.ticket_payload }
    }
 
- name: mark_synced
  type: call_method
  config: |
    (
      $destId := steps.create_ticket.output.id
        ? $string(steps.create_ticket.output.id)
        : "errored";
      {
        "connector": "staging",
        "method": "update_one",
        "args": {
          "collection": "tickets",
          "filter": { "id": steps.build_payload.output.source_id },
          "update": {
            "$set": {
              "destination_id": $destId,
              "payload_hash": steps.build_payload.output.hash,
              "synced_at": { "$date": $now() }
            }
          }
        }
      }
    )

Three rules this shape enforces:

1. Mark by the carried key, not input. mark_synced filters on steps.build_payload.output.source_id — after a call_method, input is that method's result.
2. Guard against ignored failures. With error_handling: ignore, a failed create still flows downstream; the marker writes "errored" instead of a bogus success so reruns can find and retry exactly the failed rows.
3. Filter readiness by the marker. The push task's source query selects rows where destination_id does not exist, so a rerun processes only unfinished work.

When the staging row itself needs a simple synced flag or destination ID written back, the mark_processed step does the writeback declaratively (connector, table or collection, fields).

Handle attachments

Attachments migrate in two phases with object storage as the bridge, using transfer_file (source and destination endpoints, skip_if_exists):

• Pull: stream each file from the source to object storage under a stable key built from the parent and file IDs. Stable keys let reruns find cached files instead of re-downloading.
• Push: upload the cached file to the destination's file endpoint, then reference the returned hosted URL in the record payload. Never put presigned storage URLs in destination payloads.

Gate attachment steps with error_handling: ignore — a failed attachment must not fail the parent record unless that is an explicit business rule.

Delta passes and reruns

Rerunning a push task is safe: the readiness filter skips everything already marked, and the mapping or destination_id marker prevents duplicate creates. Change detection is what makes a delta pass cheap. This is the delta shape from the real Gorgias delta task, condensed:

- name: recompute_hash
  type: transform
  config: |
    (
      $newHash := $hash({ "subject": input.subject, "status": input.status, "updated_at": input.updated_at });
      {
        "new_hash": $newHash,
        "old_hash": input.payload_hash,
        "destination_id": input.destination_id,
        "source_id": input.id,
        "needs_update": $newHash != input.payload_hash
      }
    )
 
- name: update_ticket
  type: call_method
  skip_if: "input.needs_update = false"
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "proxy_update",
      "args": { "resource": "tickets", "id": input.destination_id, "data": { "status": input.status } }
    }

The critical constraint is hash parity: the delta task must hash the exact same canonical payload subset — same fields, same normalization, same array-building order — as the initial push. A single field-order difference makes the delta update every record on every run. Cross-check by computing both hashes on one real record before trusting the delta. Advance the stored hash only after the update actually succeeded (skip_if on the update step's output).

Delta tasks are one-shot launch-day tasks, not loops — job reruns handle retries. Incremental pulls, where the source supports them, are driven by a task argument such as the start_time timestamp in the pull excerpt above; the production tasks do not use cursor state for this.

Which step when

Rewritten from the pattern selection guide the in-product AI assistant uses:

Verify it worked

• Run each push task twice against the same staged data. The second run creates nothing new; counts in the staging markers are unchanged.
• Check the staging store: every row has a destination_id that is a real ID, an explicit error marker, or a deliberate skip marker — no row is silently unaccounted for.
• Compute the push hash and the delta hash for one real record and confirm they match before running the delta at scale.

Related

• Idempotent sync and the mapping DB — the mapping and hashing mechanics used throughout
• Job templates and DAGs — orchestrate the pull and push tasks as one Job
• Error handling, retries, and loops — ignored failures and guarded markers in detail
• Steps overview — the full step catalog