# Task structure and execution model

> Source: https://docs.clonepartner.com/reference/tasks/task-structure/

A task is a YAML-defined streaming pipeline. At runtime, Envoy resolves connector slots, validates arguments, builds the step chain, and passes each source result through the remaining steps with backpressure.

## Top-level fields

```yaml
name: tickets/migrate
description: Migrate tickets
output: none

connectors:
  source:
    type: truto
  destination:
    type: postgres

arguments:
  account_id:
    type: string
    required: true

variables:
  destination_table:
    type: string
    value: tickets
    description: Destination table name

steps:
  - name: list
    type: call_method
    config: {}
```

| Field | Purpose |
|---|---|
| `name` | Stable task name |
| `description` | User-facing purpose |
| `output` | Final output format: commonly `none`, `ndjson`, or `json` |
| `connectors` | Logical slots with required connector types |
| `arguments` | Validated runtime input schema |
| `variables` | Authored constants available to expressions |
| `metrics` | Optional per-task metrics backend definitions |
| `masking` | Deterministic masking toggle and seed |
| `lookup_connector` | Legacy connector slot used by `$destinationId()` |
| `steps` | Ordered pipeline definitions |
| `loop` | Optional repeated-pipeline configuration |

Connector slots do not contain credentials. Server runs resolve them from a stored connector mapping; CLI runs resolve them from connector configuration.

## Step fields

Every step has:

- `name` — unique within the task;
- `type` — registered step type;
- `config` — YAML object or JSONata expression string.

Common optional fields:

- `run_if` and `skip_if`;
- `error_handling`;
- `concurrency`;
- `batch`;
- `spool`;
- `debug`.

See [Step types](/reference/tasks/step-types).

## Execution context

JSONata expressions can read:

- `input` — the current item;
- `arguments` — validated task arguments;
- `variables` — top-level constants;
- `job_config` — the job-run configuration snapshot;
- `steps.<name>.output` — output captured from an earlier step for the current item;
- `steps.<name>.error` — earlier step error information where available.

Use [JSONata reference](/reference/jsonata) for expression rules.

## Pipeline behavior

The first streaming step produces records. Each later step receives one current item and emits:

- one value, which becomes the next `input`;
- an array, which is unwrapped into individual downstream items;
- no value, which follows the step contract for that type.

Use a `filter` step to remove items intentionally. A skipped step passes its current input through unchanged.

`call_method` can stream an async connector result. Backpressure limits how quickly upstream pages are consumed.

## Concurrency, batch, and spool

`concurrency` permits parallel handling at that step while preserving bounded flow. Consider provider limits and write idempotency.

`batch` groups items into fixed-size arrays. Downstream configuration must expect an array.

`spool` collects the complete incoming stream before continuing. It is unbounded and should be reserved for an operation that truly needs the entire dataset.

## Loop blocks

A top-level loop repeats the complete main pipeline after a required sleep. It can run continuously, stop after continuation steps return false, or use `until_empty` with a lightweight named-step check. Use iteration or duration caps for bounded operations.

See [Masking, loops, and error handling](/reference/masking-loops-and-error-handling).

## Output formats

- `none` suppresses final data output; use it for side-effect pipelines.
- `ndjson` emits one JSON object per line and is suitable for large streams.
- `json` collects final results into one JSON array and should be limited to bounded output.

Debug messages go to stderr and structured application logs, not the task data stream.

## Snapshots

Standalone task runs snapshot the task version and connector mapping. A job snapshots its template and current task definitions when the job is created. Later edits do not mutate an existing run.

## Validation boundaries

Static validation checks YAML shape, known connector and step types, references, argument definitions, and expression compilation. It cannot prove:

- credentials are valid;
- fields exist in live data;
- external permissions are sufficient;
- expressions return the intended runtime type;
- writes are idempotent.

Follow validation with a bounded runtime test. See [Author a production task](/guides/task-authoring).