# Task anatomy and authoring workflow > Source: https://docs.clonepartner.com/building/task-anatomy/ # Task anatomy and authoring workflow A Task is a self-contained YAML file that defines one streaming pipeline: connector slots, typed inputs, and an ordered list of steps that each record flows through. This page walks through a complete task file and the write → validate → debug loop for authoring one. ## Prerequisites - The `envoy` CLI, or access to the Task editor in the web UI - A [Connector](/concepts/connectors-and-slots/) configuration for each connector type your task needs (stored in the server, or in a `connectors.yaml` file for CLI runs) ## A complete annotated task ```yaml name: stage-contacts description: Read contacts from CSV and upsert them into SQLite output: none # ndjson (default), json, or none connectors: # slots: name + type only, never credentials source: type: csv staging: type: sqlite arguments: # caller-supplied at run time dry_run: type: boolean default: true description: Skip the write step when true variables: # typed values defined inside the task target_status: type: string value: active description: Status to keep steps: - name: read_contacts type: call_method config: # static object — no input reference needed connector: source method: read args: resource: contacts - name: only_target_status type: filter config: "input.status = variables.target_status" - name: normalize type: transform config: | { "id": $string(input.id), "name": $trim(input.name), "email": $lowercase(input.email), "status": input.status } - name: write_contact type: call_method skip_if: "arguments.dry_run" config: | { "connector": "staging", "method": "upsert", "args": { "table": "contacts", "data": input, "conflict_columns": ["id"] } } ``` The top-level keys, all parsed by the task parser: | Key | Purpose | |---|---| | `name` | Task name. Defaults to the filename when omitted. | | `description` | Human-readable description. | | `output` | What happens to items after the last step: `ndjson` (default, one JSON line per item to stdout), `json` (collect all items into one array), or `none` (no stdout output — use when the last step writes to a Connector). | | `connectors` | Connector slots, keyed by name, each with a `type` only. Real configuration is injected at runtime from `connectors.yaml` (CLI) or the server database (Jobs and standalone runs). | | `arguments` | Typed runtime parameters passed in from a job template task node, CLI `--arguments`, or the server API. | | `variables` | Typed values with defaults defined inside the YAML, read as `variables.{name}`. | | `steps` | The pipeline. The only required key. | | `loop` | Optional block that re-runs the pipeline repeatedly in one run (polling and daemon-style tasks). | | `masking` | Optional PII masking toggle (`enabled`, `seed`) for the `$mask` helpers. | | `metrics` | Optional metrics backend configurations. | | `lookup_connector` | Connector slot used by `lookup` steps and the `$destinationId` helper. | | `cursor_store` | Connector slot name used for cursor persistence. | Each step takes `name`, `type`, and `config`, plus the optional keys `run_if`, `skip_if`, `error_handling`, `concurrency`, `batch`, `spool`, `debug`, and `cursor`. A step `config` is either a static YAML object (passed to the step as-is) or a JSONata string (evaluated per item — required whenever the config references `input`, `arguments`, `variables`, `steps`, or `job_config`). See the [Task YAML schema](/reference/task-yaml/) for the exhaustive key-by-key reference. `output` is a top-level key only. There is no per-step output control — the validator warns if you put `output` (or `quiet`, `silent`, and similar keys) on a step. ## The authoring workflow 1. Write the YAML. Start from the source step, confirm its output shape, then add downstream steps one at a time. 2. Validate it. From the CLI: ```bash envoy validate stage-contacts ``` Validation checks YAML syntax, task structure, step types, connector references, and that every JSONata expression compiles. It does not run anything: expressions that compile can still fail at evaluation time against a real record. 3. Run it against real data with debug output: ```bash envoy stage-contacts --config connectors.yaml --debug ``` `--debug` prints each step's input, config, and output to stderr. Combine it with a `dry_run`-style argument (as in the example above) so the write step is skipped while you inspect the transformed records. 4. Iterate: adjust expressions, re-validate, re-run. Enable writes only once the debug output looks right for a small, representative set of records. In the web UI, the Task editor gives you the same loop: edit the YAML, validate, and trigger a run from the task's detail page. See [Tasks and task runs](/using/tasks-and-runs/). ## Preserve streaming Every item flows through the pipeline independently — the source step's records are not collected into memory before downstream steps run. Design tasks to keep that property: - **Return-value contract.** A step that returns an array fans out: each element is emitted downstream as a separate item. Returning `undefined` or `[]` drops the item. Connector methods that return collections (`find`, `read`, `list`) stream each record as it arrives. - **`concurrency`** sets how many items a step processes in parallel. It uses backpressure — upstream is paused when the limit is reached, so memory stays bounded. - **`batch`** collects a step's output into arrays of N before pushing downstream (the final batch may be smaller). Use it for bulk writes. - **`spool`** collects all output into a single array, emitted once when the stream ends. It is intentionally unbounded — use it only when a downstream step genuinely needs the whole dataset (for example, counting or aggregation). - One task, one source. A pipeline is a single linear stream, so pull independent resources in separate tasks and combine them with a [Job template](/building/job-templates-and-dags/). ## Conditional step execution `run_if` and `skip_if` are JSONata expressions evaluated per item. When `run_if` is false or `skip_if` is true, the step's processing is skipped and the item passes through to the next step unchanged — it is not removed from the stream. To drop items, use a `filter` step instead (see [Transforming and filtering](/building/transforming-and-filtering/)). ```yaml - name: write_contact type: call_method skip_if: "arguments.dry_run" ``` Setting both `run_if` and `skip_if` on one step produces a validation warning; it is almost always unintentional. ## Common validation errors Messages come verbatim from the validator: - `Task must have a steps array` — the file has no `steps` key, or `steps` is not a YAML list. Every task needs at least one step. - `Duplicate step name: 'write_contact'` — two steps share a `name`. Step names must be unique because they key `steps.{name}.output` references. - `Connector 'staging' not defined in task` — a step config references a connector slot that is missing from the top-level `connectors` block. Add the slot or fix the name in the config. ## Verify it worked A valid task prints a pass summary and exits with code 0: ```bash envoy validate stage-contacts ``` ``` Validating task: stage-contacts PASS YAML syntax valid PASS Task structure valid (4 steps) PASS Step types valid PASS Connector references valid PASS JSONata expressions compile Task is valid! ``` :::callout{type="warning"} Validation only compiles JSONata expressions — evaluation-time errors surface only when a real item flows through. Always run the task with `--debug` against a small real dataset before trusting it. ::: ## Related - [Task YAML schema](/reference/task-yaml/) — every top-level and step key in detail - [Reading and writing data](/building/reading-and-writing-data/) — `call_method`, streaming, and batching - [Working with JSONata](/building/working-with-jsonata/) — the expression language used in step configs - [Tasks and task runs](/using/tasks-and-runs/) — the Task editor and run surfaces in the UI