# Migration playbook

> Source: https://docs.clonepartner.com/building/migration-playbook/

<!-- sources: references/envoy/patterns.md, references/envoy/prompt/decision-table.md, tasks/migrations/zendesk/pull-tickets.yaml, tasks/migrations/gorgias/push-tickets.yaml, tasks/migrations/gorgias/delta-tickets.yaml, references/envoy/task-structure.md -->

# Migration playbook

This guide walks through building a real migration — helpdesk tickets from a source system to a destination — the way production Envoy migrations are structured. Use it when you are planning a multi-resource migration and need to know which tasks to build, in what order, and which steps to reach for.

The shape is always the same: pull each resource into a staging store, validate and transform, push idempotently to the destination while recording what happened, then run bounded delta passes until cutover.

## Prerequisites

- Source and destination connectors, plus a staging store (a `mongodb` or SQL connector) and, if you use the mapping steps, a mapping DB.
- [Idempotent sync and the mapping DB](/building/idempotent-sync/) — the write patterns this playbook builds on.
- [Error handling, retries, and loops](/building/error-handling/) — `error_handling: ignore` and guarded markers.

## Plan resources and order

Migrate parents before children: users and contacts before tickets, tickets before comments and attachments. Each independent resource gets its own pull task and its own push task — a task pipeline is a single linear stream with one source, so independent resources cannot share a task. Combine the tasks into a Job Template DAG so independent pulls run in parallel and pushes depend on the pulls they need (see [Job templates and DAGs](/building/job-templates-and-dags/)).

A dependent pull — comments fetched per pulled ticket — stays inside one task: the parent `call_method` list step streams tickets, and a per-item `call_method` step referencing `input` fetches each ticket's children.

## Stage the source data

Pull each resource into a staging collection, stamped with scope fields so reruns upsert instead of duplicating. This excerpt is adapted from a real Zendesk ticket pull; the source is an incremental-export list call, and the write is an upsert keyed on the source ID:

```yaml
name: pull-tickets
description: Pull source tickets into the staging store (rerun-safe)
output: none

connectors:
  source:
    type: truto
  staging:
    type: mongodb

arguments:
  start_time:
    type: number
    default: 0
    description: Unix timestamp for incremental-export start_time

steps:
  - name: list_tickets
    type: call_method
    config: |
      {
        "connector": "source",
        "method": "proxy_list",
        "args": {
          "resource": "incremental-export",
          "query": {
            "start_time": arguments.start_time > 0 ? arguments.start_time : 0
          }
        }
      }

  - name: upsert_ticket
    type: call_method
    config: |
      {
        "connector": "staging",
        "method": "update_one",
        "args": {
          "collection": "tickets",
          "filter": { "id": input.id },
          "update": {
            "$set": $merge([input, { "synced_at": { "$date": $now() } }]),
            "$setOnInsert": { "comments_synced": false }
          },
          "upsert": true
        }
      }
```

Incremental re-pulls pass a later `start_time`; the upsert makes every pull rerun-safe regardless.

## Validate before you push

Run a `validate_source` step against staged data (or the source CSV) before any destination write. It checks `primary_key` uniqueness and `required_columns` presence, with a `mode` controlling whether problems fail the task or are reported. Catching a duplicate primary key here is far cheaper than discovering duplicate destination records later. See [validate_source](/reference/steps/validate-source/).

## Shape the payload

Two steps cover most transformation work:

- [map_fields](/reference/steps/map-fields/) — declarative field renames with optional per-field `transform` and `skip` expressions, driven by a `mapping` config.
- [transform](/reference/steps/transform/) — a full JSONata expression when the payload needs computed values, tag building, or conditionals.

Resolve cross-store references — "this ticket's requester is which destination contact?" — with a [lookup](/reference/steps/lookup/) step: a `keys` fallback chain where each entry names a connector and a `match`, with `required` dropping items that resolve nothing.

## Push idempotently and record the outcome

Prefer [sync_record](/reference/steps/sync-record/) for the write: mapping lookup, hash-based change detection, create or update, and mapping upsert in one step.

Production push tasks that need custom payload assembly use the guarded three-step shape instead. This excerpt is condensed from a real Gorgias ticket push. The `transform` carries the source ID and hash forward, because after `create_ticket` runs, `input` is the create response and the original item's fields are gone from it:

```yaml
- name: build_payload
  type: transform
  config: |
    {
      "ticket_payload": {
        "subject": input.subject,
        "status": input.status,
        "external_id": $string(input.id)
      },
      "hash": $hash({ "subject": input.subject, "status": input.status, "updated_at": input.updated_at }),
      "source_id": input.id
    }

- name: create_ticket
  type: call_method
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "proxy_create",
      "args": { "resource": "tickets", "data": input.ticket_payload }
    }

- name: mark_synced
  type: call_method
  config: |
    (
      $destId := steps.create_ticket.output.id
        ? $string(steps.create_ticket.output.id)
        : "errored";
      {
        "connector": "staging",
        "method": "update_one",
        "args": {
          "collection": "tickets",
          "filter": { "id": steps.build_payload.output.source_id },
          "update": {
            "$set": {
              "destination_id": $destId,
              "payload_hash": steps.build_payload.output.hash,
              "synced_at": { "$date": $now() }
            }
          }
        }
      }
    )
```

Three rules this shape enforces:

1. **Mark by the carried key, not `input`.** `mark_synced` filters on `steps.build_payload.output.source_id` — after a `call_method`, `input` is that method's result.
2. **Guard against ignored failures.** With `error_handling: ignore`, a failed create still flows downstream; the marker writes `"errored"` instead of a bogus success so reruns can find and retry exactly the failed rows.
3. **Filter readiness by the marker.** The push task's source query selects rows where `destination_id` does not exist, so a rerun processes only unfinished work.

When the staging row itself needs a simple synced flag or destination ID written back, the [mark_processed](/reference/steps/mark-processed/) step does the writeback declaratively (`connector`, `table` or `collection`, `fields`).

## Handle attachments

Attachments migrate in two phases with object storage as the bridge, using [transfer_file](/reference/steps/transfer-file/) (`source` and `destination` endpoints, `skip_if_exists`):

- **Pull:** stream each file from the source to object storage under a stable key built from the parent and file IDs. Stable keys let reruns find cached files instead of re-downloading.
- **Push:** upload the cached file to the destination's file endpoint, then reference the returned hosted URL in the record payload. Never put presigned storage URLs in destination payloads.

Gate attachment steps with `error_handling: ignore` — a failed attachment must not fail the parent record unless that is an explicit business rule.

## Delta passes and reruns

Rerunning a push task is safe: the readiness filter skips everything already marked, and the mapping or `destination_id` marker prevents duplicate creates. Change detection is what makes a **delta** pass cheap. This is the delta shape from the real Gorgias delta task, condensed:

```yaml
- name: recompute_hash
  type: transform
  config: |
    (
      $newHash := $hash({ "subject": input.subject, "status": input.status, "updated_at": input.updated_at });
      {
        "new_hash": $newHash,
        "old_hash": input.payload_hash,
        "destination_id": input.destination_id,
        "source_id": input.id,
        "needs_update": $newHash != input.payload_hash
      }
    )

- name: update_ticket
  type: call_method
  skip_if: "input.needs_update = false"
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "proxy_update",
      "args": { "resource": "tickets", "id": input.destination_id, "data": { "status": input.status } }
    }
```

The critical constraint is **hash parity**: the delta task must hash the exact same canonical payload subset — same fields, same normalization, same array-building order — as the initial push. A single field-order difference makes the delta update every record on every run. Cross-check by computing both hashes on one real record before trusting the delta. Advance the stored hash only after the update actually succeeded (`skip_if` on the update step's output).

Delta tasks are one-shot launch-day tasks, not loops — job reruns handle retries. Incremental pulls, where the source supports them, are driven by a task argument such as the `start_time` timestamp in the pull excerpt above; the production tasks do not use cursor state for this.

## Which step when

Rewritten from the pattern selection guide the in-product AI assistant uses:

| You want to… | Use |
|---|---|
| Skip already-migrated records / dedup | [sync_record](/reference/steps/sync-record/), or [get_destination_id](/reference/steps/get-destination-id/) + `skip_if` + [set_destination_id](/reference/steps/set-destination-id/) with a mapping DB |
| Detect changed records | [compute_hash](/reference/steps/compute-hash/) + hash comparison via the mapping lookup |
| Dedup + create/update + mapping in one step | [sync_record](/reference/steps/sync-record/) |
| Map or rename fields | [map_fields](/reference/steps/map-fields/) |
| Validate source schema before migrating | [validate_source](/reference/steps/validate-source/) |
| Stage or upsert rows to SQL/Mongo | [upsert_record](/reference/steps/upsert-record/) |
| Bootstrap SQL tables at task start | [ensure_table](/reference/steps/ensure-table/) |
| Cross-store foreign-key lookup | [lookup](/reference/steps/lookup/) |
| Write a synced flag or destination ID back to the source | [mark_processed](/reference/steps/mark-processed/) |
| Stream a file between connectors | [transfer_file](/reference/steps/transfer-file/) |
| Call a connector per item and keep the item's fields | `$callMethod` inside a [transform](/reference/steps/transform/) — bounded single-record lookups only |
| Pull a whole resource | [call_method](/reference/steps/call-method/) step as the pipeline source (streams with backpressure) |
| Drop items from the stream | [filter](/reference/steps/filter/) — `skip_if` skips the step but keeps the item flowing |
| Repeat until a query returns no rows | `loop.until_empty` — see [Error handling, retries, and loops](/building/error-handling/) |

## Verify it worked

- Run each push task twice against the same staged data. The second run creates nothing new; counts in the staging markers are unchanged.
- Check the staging store: every row has a `destination_id` that is a real ID, an explicit error marker, or a deliberate skip marker — no row is silently unaccounted for.
- Compute the push hash and the delta hash for one real record and confirm they match before running the delta at scale.

:::callout{type="tip"}
Reset failed rows for a retry by clearing their error marker in the staging store — the readiness filter picks them up on the next run. Nothing else needs to change.
:::

## Related

- [Idempotent sync and the mapping DB](/building/idempotent-sync/) — the mapping and hashing mechanics used throughout
- [Job templates and DAGs](/building/job-templates-and-dags/) — orchestrate the pull and push tasks as one Job
- [Error handling, retries, and loops](/building/error-handling/) — ignored failures and guarded markers in detail
- [Steps overview](/reference/steps/) — the full step catalog