Mapping database and sync_record

A mapping store records the relationship between a stable source ID and destination ID for a resource. It can also store a content hash used to detect meaningful changes.

Data-storage connectors that implement mapping methods can serve as the mapping store. Keep the store durable, backed up, and shared by every executor that runs the migration.

Mapping key

Each record is identified by:

resource + source_id

Choose a stable resource name and immutable source ID. Convert IDs to strings consistently.

Low-level steps

get_destination_id

- name: existing
  type: get_destination_id
  config: |
    {
      "connector": "mapping_db",
      "resource": "contact",
      "source_id": $string(input.id)
    }

It returns:

{
  "destination_id": null,
  "hash": null
}

set_destination_id

- name: remember
  type: set_destination_id
  skip_if: "$not(steps.create.output.id)"
  config: |
    {
      "connector": "mapping_db",
      "resource": "contact",
      "source_id": $string(steps.normalize.output.id),
      "destination_id": $string(steps.create.output.id),
      "hash": steps.hash.output.hash
    }

The step refuses an empty source or destination ID. Keep the guard because an ignored destination error can leave no output.

sync_record

sync_record implements the standard composite flow:

1. read the mapping;
2. compute a canonical content hash when configured;
3. skip an unchanged mapped record;
4. create an unmapped record;
5. update a mapped record when hash changed and an update method exists;
6. persist destination ID and hash;
7. optionally write fields back to staging.

- name: sync
  type: sync_record
  config: |
    {
      "mapping": {
        "connector": "mapping_db",
        "resource": "contact",
        "source_id": $string(input.source_id)
      },
      "hash": {
        "from": "payload",
        "fields": ["name", "email"],
        "algorithm": "sha256"
      },
      "destination": {
        "connector": "destination",
        "create": {
          "method": "create",
          "resource": "contacts",
          "args": input.payload,
          "id": "id"
        },
        "update": {
          "method": "update",
          "resource": "contacts",
          "args": input.payload
        }
      },
      "writeback": {
        "connector": "staging",
        "table": "contacts",
        "key_column": "source_id",
        "key": input.source_id,
        "fields": {
          "migration_status": "complete"
        }
      }
    }

hash.algorithm supports sha256 and md5; use sha256 unless compatibility requires otherwise.

If an existing mapping has no update method, no hash, or an unchanged hash, the record passes through without a destination write.

Destination ID extraction

For creates, set destination.create.id when the response ID field is not a standard supported name. An empty extracted ID is an error and the mapping is not written.

For updates, the existing destination ID is supplied to the destination call when appropriate.

Writeback

Writeback supports a SQL table or MongoDB collection. It adds destination_id and, when available, hash unless those fields are explicitly supplied.

Writeback occurs after the destination operation and mapping persistence. Design reconciliation for the rare case where destination and mapping succeed but writeback fails.

Backup and reconciliation

Back up the mapping store before cutover. Verify:

• no duplicate (resource, source_id) mappings;
• every destination ID is non-empty;
• hashes use the current canonical payload;
• mappings resolve to existing destination records;
• stale mappings are handled explicitly.

See Build an idempotent migration.