Build an idempotent migration

A migration is idempotent when rerunning the same source state does not create duplicate destination records or corrupt progress. Envoy provides streaming, hash, mapping, and upsert primitives, but the task author must define identity and retry behavior correctly.

Establish the identity contract

For each resource, document:

• the immutable source identifier;
• destination uniqueness rules;
• the mapping-store resource name;
• the fields included in the canonical payload hash;
• how deletes or deactivations are handled;
• what counts as complete.

Do not derive identity from mutable labels, timestamps, or array order unless that order is normalized.

Prefer sync_record

sync_record combines mapping lookup, content hashing, create/update selection, and mapping persistence:

- name: sync_contact
  type: sync_record
  error_handling: ignore
  config: |
    {
      "mapping": {
        "connector": "mapping_db",
        "resource": "contact",
        "source_id": $string(input.id)
      },
      "hash": {
        "fields": ["name", "email"],
        "algorithm": "sha256"
      },
      "destination": {
        "connector": "crm",
        "create": {
          "method": "create",
          "resource": "contacts",
          "args": {
            "name": input.name,
            "email": input.email
          }
        },
        "update": {
          "method": "update",
          "resource": "contacts",
          "args": {
            "name": input.name,
            "email": input.email
          }
        }
      }
    }

When the stored hash matches, the step skips the destination write. When the payload changes, it uses the mapped destination ID for an update.

Use the lower-level get_destination_id, compute_hash, connector write, and set_destination_id steps only when the composite step cannot express the required workflow.

Keep hashes canonical

The initial load and delta task must hash the exact same payload:

• same fields;
• same default values;
• same text normalization;
• same array sorting;
• same date representation;
• same conditional fields.

Compute the hash after normalization and before the destination write. A small difference between initial and delta hash logic causes every record to look changed.

Guard success and failure markers

When a destination write uses error_handling: ignore, its output may be absent. Never mark success unconditionally:

- name: create_ticket
  type: call_method
  error_handling: ignore
  config: |
    {
      "connector": "destination",
      "method": "create",
      "args": {
        "resource": "tickets",
        "data": input.payload
      }
    }
 
- name: mark_success
  type: call_method
  skip_if: "$not(steps.create_ticket.output.id)"
  config: |
    {
      "connector": "staging",
      "method": "update_one",
      "args": {
        "collection": "tickets",
        "filter": { "id": steps.build_payload.output.source_id },
        "update": { "$set": {
          "destination_id": $string(steps.create_ticket.output.id),
          "payload_hash": steps.build_payload.output.hash
        } }
      }
    }

Add a separate failure marker with the error category and retry eligibility. Preserve enough source identity to retry without replaying unrelated rows.

Design retries

Classify failures:

• transient provider or network failures;
• rate limits;
• invalid source data;
• destination validation failures;
• missing relationships;
• configuration errors.

Retry transient failures with bounded attempts and provider-aware pacing. Do not repeatedly send a permanently invalid payload. A timeout after a create is ambiguous: query by the stable external key before retrying.

Relationships and ordering

Migrate parent resources before children. Store parent source-to-destination mappings, then resolve foreign keys through bounded lookups. Use a job DAG when stages have independent observability or need pause/resume boundaries.

For attachments, use a stable intermediate object key so a rerun can reuse a previously transferred file. Do not let one failed attachment invalidate an otherwise valid parent unless that is an explicit business rule.

Reconciliation

Counts alone are insufficient. Compare:

• source records in scope;
• accepted, skipped, failed, and pending records;
• unique mapped source IDs;
• destination records addressable by those mappings;
• representative transformed fields;
• parent/child and attachment counts;
• a deterministic sample and known edge cases.

Run the migration twice against the same source snapshot. The second run should create nothing new and update only records whose canonical payload changed.

Cutover checklist

1. Freeze or record the source high-water mark.
2. Run validation and a bounded sample.
3. Verify mapping-store backup and destination uniqueness.
4. Complete the initial load.
5. Reconcile records and relationships.
6. Run delta logic against changed source records.
7. Reconcile again.
8. Preserve run IDs, configuration snapshots, and reports.

See Mapping database and sync_record and Task patterns.