Task YAML schema

This schema governs task files — the YAML pipelines stored in tasks/ or in the tasks table. src/task-parser.ts parses a task into a TaskDefinition, and src/task-validator.ts checks it statically before any run. Connector credentials never appear in task YAML: a task declares typed connector slots, and actual configurations are injected at runtime.

Top-level keys

Any other top-level key produces the validation warning Unknown top-level key: '<key>'.

output

Controls what happens to items that reach the end of the pipeline:

• ndjson (default) — each item is written as one JSON line to stdout.
• json — all items are collected and written as a single JSON array to stdout.
• none — no stdout output; use when the last step writes to a Connector.

output is a top-level key only. There is no per-step output control — putting output (or quiet, silent, and similar keys) on a step raises a validation warning.

connectors

Each slot declares a name and a type — nothing more. Actual configuration is injected at runtime from connectors.yaml (CLI) or the connectors table (server), so tasks stay portable.

connectors:
  source:
    type: csv
  destination:
    type: sqlite

Inline config under a slot is flagged: Connector '<name>' has inline config — connectors should declare type only; config is injected at runtime. A slot no step references produces the warning Connector '<name>' is defined but never referenced by any step.

arguments

Runtime parameters supplied when a run starts — from a Job Template task node, CLI --arguments, the server API, or a UI form. Each argument declares:

arguments:
  account_id:
    type: string
    required: true
    description: Integrated account to pull from
  batch_limit:
    type: number
    default: 100

At run start every provided value is coerced to its declared type, so arguments.<name> inside JSONata is always the declared type. Missing required arguments fail with Argument "<name>" is required but was not provided; values outside enum fail with Argument "<name>" must be one of [...] but got <value>. Arguments also accept UI widget fields — see Arguments schema.

variables

Typed constants defined inside the task, read in JSONata as variables.<name>. Each entry must have type and value; description is optional.

variables:
  target_status:
    type: string
    value: active
    description: Status to filter by
  batch_size:
    type: number
    value: 100

Valid types match argument types: string, number, boolean, string[], number[]. A variable never referenced in any expression produces a validation warning.

steps

An array of step objects. Only these keys are valid on a step — anything else raises Unknown step key: '<key>':

steps:
  - name: read_users
    type: call_method
    config:
      connector: source
      method: read
      args:
        resource: users
 
  - name: upsert_user
    type: call_method
    concurrency: 5
    error_handling: ignore
    config: |
      {
        "connector": "destination",
        "method": "upsert",
        "args": { "table": "users", "data": input, "conflict_columns": ["id"] }
      }

A string config is compiled as JSONata and evaluated per item with { input, arguments, steps, variables, job_config }. An object config is passed verbatim and never evaluated — a JSONata reference such as arguments.x inside a static object is a bug the validator flags. Step-specific config keys are documented on each step's page; see Steps overview.

error_handling

Controls what happens when a step throws for an item:

Any other value fails validation with Invalid error_handling: '<value>' (must be ignore, ignore_rest, or fail). For retry patterns and when to choose each policy, see Error handling, retries, and loops.

loop

Repeats the main pipeline within a single run — for polling, delta sync, or drain-until-empty workloads.

loop:
  sleep_ms: 30000
  max_iterations: 100
  max_duration_ms: 3600000
  until_empty:
    from_step: read_users

Loop steps accept the same keys as main steps except cursor. Ending loop.steps with a call_method step produces a warning, because its output is coerced to a boolean ambiguously — prefer ending with a transform.

masking

Toggles deterministic PII masking for $mask bindings in JSONata:

masking:
  enabled: ${MASK_PII}
  seed: ${MASK_SEED}

enabled is coerced to a boolean (true, 1, yes, on count as true). Environment variables in this block resolve leniently: an unset variable becomes an empty string, so masking is simply off where the toggle is not defined. Keys other than enabled and seed produce the warning Unknown key '<key>' in masking block.

metrics

An array of metrics backend configurations. Each entry must have a backend string and may carry a backend-specific config object; environment variables inside config are resolved at parse time.

metrics:
  - backend: statsd
    config:
      host: ${STATSD_HOST}

Available backends and their config keys are documented in Metrics config.

Validation errors

Static validation (envoy validate and the server's validate endpoint) emits these messages verbatim. Errors block the run; warnings do not.

Task must have a steps array — the top-level steps key is missing or not a list. Add at least one step under steps:.

Task must have at least one step — steps is an empty array. A pipeline needs a source step.

Invalid output format: '<value>' (must be ndjson, json, or none) — output is set to something else. Use one of the three formats.

Connector '<name>' must have a type — a connector slot is missing its type. Every slot needs exactly one type string.

Unknown connector type: '<type>'. Available: <list> — the slot's type is not a registered connector. Pick a type from the listed registry.

Unknown step type: '<type>' — the step's type is not one of the 15 registered types. Check spelling against Steps overview.

Duplicate step name: '<name>' — two steps share a name. Names must be unique because downstream expressions address them as steps.<name>.output.

Transform step config must be a JSONata expression string — a transform step has an object config. Use a config: | block string.

Invalid JSONata expression: <message> — a string config, run_if, or skip_if fails to compile. Fix the expression; the message includes the JSONata error code and token.

Connector '<name>' not defined in task — a step's config references a connector slot that is not declared under connectors. Declare the slot or fix the name.

Method '<method>' is not available on <type> connector. Available: <list> — a call_method step names a method the connector type does not expose. Use one of the listed methods.

concurrency must be a positive integer — concurrency is zero, negative, or not an integer.

batch must be a positive integer or JSONata expression — batch is neither a positive integer nor an expression string.

spool must be a boolean or JSONata expression — spool is neither a boolean nor an expression string.

loop.sleep_ms must be >= 10000 (minimum 10 seconds between iterations) — the loop sleep is under 10 seconds. Raise it to avoid tight loops.

loop.check_num_times requires loop.steps or loop.until_empty — a check count is set without anything to check. Add continuation steps or an until_empty block.

loop.until_empty references unknown step '<name>' — from_step does not match any main step name.

cursor_store references undefined connector '<name>' — the cursor_store slot is not declared under connectors. The same applies to lookup_connector (lookup_connector references undefined connector '<name>').

Static config contains a JSONata reference ('<leaf>'). Static object configs are passed verbatim and never evaluated — convert this step's config to a JSONata string (config: |). — an object config embeds input.x, arguments.x, variables.x, or steps.x. Change the config to a JSONata string so the reference is evaluated.

Related

• Steps overview — all 15 step types and how to choose one
• JSONata reference — expression contexts and helper functions
• Task anatomy and authoring workflow — how to build and test a task
• Error handling, retries, and loops — the guide behind error_handling, loop, and masking