Documentation by ClonePartner

Start typing to search.

Task Reference

mark_processed

View Markdown

Write a processed flag, destination ID, or hash back onto the source row or document so delta passes can skip completed records.

mark_processed

Updates the source row or document that produced the current item — typically to set a processed flag, the destination ID, or a content hash — and passes the item through unchanged.

When to use it

Use mark_processed as the last write in a pipeline whose source is a database you can write to, so a later delta run can filter out completed records at the source. When the destination write itself needs dedup and change tracking, the writeback block of sync_record covers the same update in one step.

Configuration

Key Type Required Default Description
connector string Yes — SQL (sqlite, mssql, mysql, postgres) or MongoDB Connector slot. Other types fail with mark_processed requires a SQL or MongoDB connector.
table string For SQL — Table to update. One of table or collection is required.
collection string For MongoDB — Collection to update. One of table or collection is required.
key_column string No id Column/field that identifies the row or document.
key any No — Key value. Omitted, the input item's key_column field is used. A missing key fails with mark_processed requires key or key_column on input.
fields object No — Columns/fields to write.
set object No — Alias for fields. Used only when fields is not set.
unset string[] No — MongoDB only: fields to remove from the document via $unset.

Example

name: push_and_mark
connectors:
  source:
    type: sqlite
  destination:
    type: freshdesk
steps:
  - name: read_pending
    type: call_method
    config:
      connector: source
      method: select
      args:
        sql: SELECT * FROM contacts WHERE synced IS NULL
  - name: create_contact
    type: transform
    config: |
      {
        "id": input.id,
        "created": $callMethod({
          "connector": "destination",
          "method": "create",
          "resource": "api/v2/contacts",
          "data": { "name": input.name }
        })
      }
  - name: mark_synced
    type: mark_processed
    config: |
      {
        "connector": "source",
        "table": "contacts",
        "fields": {
          "synced": true,
          "destination_id": $string(input.created.id)
        }
      }

Behavior notes

  • SQL Connectors call the Connector's update method with table, data (the fields), and a filter of { key_column: key }. When fields resolves to an empty object, no update is issued.
  • MongoDB Connectors call update_one with $set built from fields and $unset built from unset (each listed field mapped to ''). When both are empty, no update is issued.
  • The key value defaults to the input item's key_column field (here id, carried through the transform). When a call_method step has replaced the item — for example with a create response that has no source key — pass key explicitly instead.
  • The step returns the input item unchanged, so it can sit anywhere in the pipeline.
  • sync_record — combines the destination write with an equivalent writeback
  • filter — skip already-marked rows at the start of a delta pass
  • Migration playbook — delta-pass patterns that rely on marking