# Task YAML schema

> Source: https://docs.clonepartner.com/reference/task-yaml/

<!-- sources: src/task-parser.ts, src/task-validator.ts, references/envoy/task-structure.md -->

# Task YAML schema

This schema governs task files — the YAML pipelines stored in `tasks/` or in the `tasks` table. `src/task-parser.ts` parses a task into a `TaskDefinition`, and `src/task-validator.ts` checks it statically before any run. Connector credentials never appear in task YAML: a task declares typed connector slots, and actual configurations are injected at runtime.

## Top-level keys

| Key | Type | Required | Description |
|---|---|---|---|
| `name` | string | No | Task name. Defaults to the file name without its extension. |
| `description` | string | No | Human-readable description. |
| `output` | string | No | Output format: `ndjson` (default), `json`, or `none`. |
| `lookup_connector` | string | No | Connector slot backing `lookup` steps and the `$callMethod` / `$getDestinationId` JSONata helpers. |
| `masking` | object | No | PII masking toggle: `enabled` and optional `seed`. |
| `cursor_store` | string | No | Connector slot used to persist step cursors. |
| `arguments` | object | No | Typed runtime parameters, keyed by name. |
| `connectors` | object | No | Connector slots, keyed by name, each with a `type`. |
| `variables` | object | No | Typed constants, keyed by name, each with `type` and `value`. |
| `metrics` | array | No | Metrics backend configurations (`backend` + optional `config`). |
| `steps` | array | Yes | Pipeline steps, executed as a single linear stream. |
| `loop` | object | No | Repeats the main pipeline in one run. |

Any other top-level key produces the validation warning `Unknown top-level key: '<key>'`.

## `output`

Controls what happens to items that reach the end of the pipeline:

- `ndjson` (default) — each item is written as one JSON line to stdout.
- `json` — all items are collected and written as a single JSON array to stdout.
- `none` — no stdout output; use when the last step writes to a Connector.

`output` is a top-level key only. There is no per-step output control — putting `output` (or `quiet`, `silent`, and similar keys) on a step raises a validation warning.

## `connectors`

Each slot declares a name and a type — nothing more. Actual configuration is injected at runtime from `connectors.yaml` (CLI) or the `connectors` table (server), so tasks stay portable.

```yaml
connectors:
  source:
    type: csv
  destination:
    type: sqlite
```

Inline `config` under a slot is flagged: `Connector '<name>' has inline config — connectors should declare type only; config is injected at runtime`. A slot no step references produces the warning `Connector '<name>' is defined but never referenced by any step`.

## `arguments`

Runtime parameters supplied when a run starts — from a Job Template task node, CLI `--arguments`, the server API, or a UI form. Each argument declares:

| Field | Type | Required | Description |
|---|---|---|---|
| `type` | string | Yes | `string`, `number`, `boolean`, `string[]`, or `number[]`. |
| `description` | string | No | Help text shown to operators. |
| `required` | boolean | No | Reject the run when the value is missing. |
| `default` | any | No | Value used when the argument is omitted. |
| `enum` | array | No | Allowed values; anything else is rejected. |

```yaml
arguments:
  account_id:
    type: string
    required: true
    description: Integrated account to pull from
  batch_limit:
    type: number
    default: 100
```

At run start every provided value is coerced to its declared type, so `arguments.<name>` inside JSONata is always the declared type. Missing required arguments fail with `Argument "<name>" is required but was not provided`; values outside `enum` fail with `Argument "<name>" must be one of [...] but got <value>`. Arguments also accept UI widget fields — see [Arguments schema](/reference/arguments/).

## `variables`

Typed constants defined inside the task, read in JSONata as `variables.<name>`. Each entry must have `type` and `value`; `description` is optional.

```yaml
variables:
  target_status:
    type: string
    value: active
    description: Status to filter by
  batch_size:
    type: number
    value: 100
```

Valid types match argument types: `string`, `number`, `boolean`, `string[]`, `number[]`. A variable never referenced in any expression produces a validation warning.

## `steps`

An array of step objects. Only these keys are valid on a step — anything else raises `Unknown step key: '<key>'`:

| Key | Type | Required | Description |
|---|---|---|---|
| `name` | string | Yes | Unique within the task; read downstream as `steps.<name>.output`. |
| `type` | string | Yes | A registered step type — see [Steps overview](/reference/steps/). |
| `config` | object or string | Depends on type | Static YAML object, or a JSONata expression string evaluated per item. |
| `run_if` | string | No | JSONata condition; the step runs only when truthy. When falsy, the item passes through unchanged. |
| `skip_if` | string | No | JSONata condition; the step is skipped when truthy. The item passes through unchanged. |
| `error_handling` | string | No | `ignore`, `ignore_rest`, or `fail` (default). See below. |
| `concurrency` | number | No | Max concurrent executions for this step. Defaults to 1 (sequential). Upstream pauses with backpressure at the limit. |
| `batch` | number or string | No | Collect output into arrays of this size before emitting downstream. A JSONata expression string is resolved on the first item. The final batch may be smaller. |
| `spool` | boolean or string | No | Collect all output into a single array, emitted once when the stream ends. A JSONata expression string is resolved on the first item. |
| `debug` | boolean | No | Print this step's input, config, and output to stderr. |
| `cursor` | object | No | Cursor persistence for the step: an object with a `key` string, stored via the `cursor_store` connector. |

```yaml
steps:
  - name: read_users
    type: call_method
    config:
      connector: source
      method: read
      args:
        resource: users

  - name: upsert_user
    type: call_method
    concurrency: 5
    error_handling: ignore
    config: |
      {
        "connector": "destination",
        "method": "upsert",
        "args": { "table": "users", "data": input, "conflict_columns": ["id"] }
      }
```

A string `config` is compiled as JSONata and evaluated per item with `{ input, arguments, steps, variables, job_config }`. An object `config` is passed verbatim and never evaluated — a JSONata reference such as `arguments.x` inside a static object is a bug the validator flags. Step-specific `config` keys are documented on each step's page; see [Steps overview](/reference/steps/).

### `error_handling`

Controls what happens when a step throws for an item:

| Value | Behavior |
|---|---|
| `fail` (default) | The error stops the entire task. |
| `ignore` | Logs a warning and emits an empty item so downstream steps still run for it. |
| `ignore_rest` | Logs a warning and emits nothing — remaining steps are skipped for this item, but the task continues with other items. |

Any other value fails validation with `Invalid error_handling: '<value>' (must be ignore, ignore_rest, or fail)`. For retry patterns and when to choose each policy, see [Error handling, retries, and loops](/building/error-handling/).

## `loop`

Repeats the main pipeline within a single run — for polling, delta sync, or drain-until-empty workloads.

```yaml
loop:
  sleep_ms: 30000
  max_iterations: 100
  max_duration_ms: 3600000
  until_empty:
    from_step: read_users
```

| Key | Type | Required | Description |
|---|---|---|---|
| `sleep_ms` | integer | Yes | Milliseconds to sleep between iterations. Must be >= 10000. |
| `check_num_times` | integer | No | Consecutive falsy checks tolerated before exiting. Must be >= 1 and requires `loop.steps` or `loop.until_empty`. |
| `max_iterations` | integer | No | Safety cap on main pipeline runs. |
| `max_duration_ms` | integer | No | Wall-clock limit from task start. Must be >= 10000. |
| `steps` | array | No | Continuation-check steps run after each drain; the final step's output is coerced to a boolean. |
| `until_empty` | object | No | `{ from_step: "<step name>" }` — re-run the main pipeline while that step still returns rows. |

Loop steps accept the same keys as main steps except `cursor`. Ending `loop.steps` with a `call_method` step produces a warning, because its output is coerced to a boolean ambiguously — prefer ending with a `transform`.

## `masking`

Toggles deterministic PII masking for `$mask` bindings in JSONata:

```yaml
masking:
  enabled: ${MASK_PII}
  seed: ${MASK_SEED}
```

`enabled` is coerced to a boolean (`true`, `1`, `yes`, `on` count as true). Environment variables in this block resolve leniently: an unset variable becomes an empty string, so masking is simply off where the toggle is not defined. Keys other than `enabled` and `seed` produce the warning `Unknown key '<key>' in masking block`.

## `metrics`

An array of metrics backend configurations. Each entry must have a `backend` string and may carry a backend-specific `config` object; environment variables inside `config` are resolved at parse time.

```yaml
metrics:
  - backend: statsd
    config:
      host: ${STATSD_HOST}
```

Available backends and their config keys are documented in [Metrics config](/reference/metrics/).

## Validation errors

Static validation (`envoy validate` and the server's validate endpoint) emits these messages verbatim. Errors block the run; warnings do not.

**`Task must have a steps array`** — the top-level `steps` key is missing or not a list. Add at least one step under `steps:`.

**`Task must have at least one step`** — `steps` is an empty array. A pipeline needs a source step.

**`Invalid output format: '<value>' (must be ndjson, json, or none)`** — `output` is set to something else. Use one of the three formats.

**`Connector '<name>' must have a type`** — a connector slot is missing its `type`. Every slot needs exactly one `type` string.

**`Unknown connector type: '<type>'. Available: <list>`** — the slot's `type` is not a registered connector. Pick a type from the listed registry.

**`Unknown step type: '<type>'`** — the step's `type` is not one of the 15 registered types. Check spelling against [Steps overview](/reference/steps/).

**`Duplicate step name: '<name>'`** — two steps share a name. Names must be unique because downstream expressions address them as `steps.<name>.output`.

**`Transform step config must be a JSONata expression string`** — a `transform` step has an object `config`. Use a `config: |` block string.

**`Invalid JSONata expression: <message>`** — a string config, `run_if`, or `skip_if` fails to compile. Fix the expression; the message includes the JSONata error code and token.

**`Connector '<name>' not defined in task`** — a step's config references a connector slot that is not declared under `connectors`. Declare the slot or fix the name.

**`Method '<method>' is not available on <type> connector. Available: <list>`** — a `call_method` step names a method the connector type does not expose. Use one of the listed methods.

**`concurrency must be a positive integer`** — `concurrency` is zero, negative, or not an integer.

**`batch must be a positive integer or JSONata expression`** — `batch` is neither a positive integer nor an expression string.

**`spool must be a boolean or JSONata expression`** — `spool` is neither a boolean nor an expression string.

**`loop.sleep_ms must be >= 10000 (minimum 10 seconds between iterations)`** — the loop sleep is under 10 seconds. Raise it to avoid tight loops.

**`loop.check_num_times requires loop.steps or loop.until_empty`** — a check count is set without anything to check. Add continuation steps or an `until_empty` block.

**`loop.until_empty references unknown step '<name>'`** — `from_step` does not match any main step name.

**`cursor_store references undefined connector '<name>'`** — the `cursor_store` slot is not declared under `connectors`. The same applies to `lookup_connector` (`lookup_connector references undefined connector '<name>'`).

**`Static config contains a JSONata reference ('<leaf>'). Static object configs are passed verbatim and never evaluated — convert this step's config to a JSONata string (config: |).`** — an object config embeds `input.x`, `arguments.x`, `variables.x`, or `steps.x`. Change the config to a JSONata string so the reference is evaluated.

## Related

- [Steps overview](/reference/steps/) — all 15 step types and how to choose one
- [JSONata reference](/reference/jsonata/) — expression contexts and helper functions
- [Task anatomy and authoring workflow](/building/task-anatomy/) — how to build and test a task
- [Error handling, retries, and loops](/building/error-handling/) — the guide behind `error_handling`, `loop`, and `masking`