Documentation by ClonePartner

Start typing to search.

Task Reference

validate_source

View Markdown

Check source rows for a usable primary key and required columns, per row or as a single batch report, before writing anything.

validate_source

Validates each source row against a primary-key and required-column contract, either annotating rows as they stream or producing one summary report for the whole batch.

When to use it

Use validate_source at the front of a migration pipeline to catch broken exports — null primary keys, missing columns — before any record reaches the destination. To drop rows on arbitrary conditions rather than this fixed contract, use a filter step.

Configuration

Key Type Required Default Description
primary_key string Yes — Column that must be present and non-empty on every row. null, missing, and '' all count as a null primary key.
required_columns string[] No — Columns that must exist on each row (presence check, not a value check).
reject_null_pk boolean No true Row mode only: when not set to false, invalid rows are dropped from the stream. Set false to keep them, annotated.
mode string No report row for streaming per-row validation, report for a single batch summary.

Example

name: validate_contacts_export
connectors:
  source:
    type: csv
steps:
  - name: read_rows
    type: call_method
    config:
      connector: source
      method: read
      args:
        resource: contacts
  - name: validate
    type: validate_source
    spool: true
    config:
      primary_key: contact_id
      required_columns:
        - email
        - name

Behavior notes

row mode

Each row is checked and annotated with a _validation object: { valid, null_pk, missing_columns }. Invalid rows (null primary key or missing required columns) are dropped unless reject_null_pk: false, in which case they pass through with valid: false so a later step can route them. Memory use is constant — rows stream through one at a time.

report mode (default)

The step consumes the whole batch and emits a single report:

{
  "status": "ok",
  "messages": [],
  "data": {
    "row_count": 1200,
    "null_pk_count": 0,
    "missing_columns": [],
    "column_count": 14
  }
}
  • status is error when any message is present; messages are Missing required columns: <names> and <n> row(s) have null/empty primary key.
  • missing_columns reports the first row's missing set; column_count is the widest row seen.
  • Report mode requires array input: set spool: true on this step (or an upstream step) so all rows arrive as one batch. Non-array input fails with validate_source report mode requires array input. Only counters are kept in memory, not the rows.