# Job templates and DAGs

> Source: https://docs.clonepartner.com/building/job-templates-and-dags/

<!-- sources: references/envoy/jobs.md, src/job-parser.ts, src/server/api/jobs.ts, src/server/runner/job-manager.ts, jobs/fiserv-migration.yaml, jobs/zendesk-to-gorgias.yaml -->

# Job templates and DAGs

A Job Template is a YAML file that composes Tasks into a DAG: it declares type-only connector slots, wires each task node to those slots, and sets a failure policy. Write one when a pipeline spans more than one Task — validation before migration, parallel pulls feeding a load, or a reusable Task run twice with different arguments.

## Prerequisites

- One or more Tasks already saved in the task library — a template only references Tasks by name.
- Familiarity with [task anatomy](/building/task-anatomy/) and connector slots.

## Anatomy of a job template

The parser (`parseJobDefinition` in `src/job-parser.ts`) accepts these top-level keys:

| Key | Type | Required | Description |
|---|---|---|---|
| `name` | string | Yes | Template name. Defaults to the file name when omitted in a file-based template. |
| `description` | string | No | Human-readable description. |
| `connectors` | object | No | Type-only connector slots: slot name → `{ type }`. No credentials. |
| `tasks` | object | Yes | DAG nodes keyed by task key. |
| `on_failure` | string | No | `pause` (default), `abort`, or `continue`. |
| `arguments` | object | No | Create-time parameters substituted into the YAML as `${VAR}` when a Job is created. |
| `job_config` | object | No | Runtime-editable configuration schema copied to each Job. |

See [Job template YAML](/reference/job-templates/) for the exhaustive schema, [Template arguments](/building/template-arguments/) for `arguments`, and [Job config](/building/job-config/) for `job_config`.

## A complete template

Adapted from the Fiserv migration template in the Envoy repo — a setup Task and a validation Task both gate the migration Task:

```yaml
name: fiserv-migration
description: Migrate from Enact CRM to D365

connectors:
  d365:
    type: dynamics365
  mapping_db:
    type: sqlite
  csv_source:
    type: csv

tasks:
  gen_lookup:
    task: migrations/gen-contact-lookup
    type: command
    connector_mapping:
      mapping_db: mapping_db
      csv_source: csv_source

  validate_csv:
    task: migrations/validate-contact-csv
    type: validation
    connector_mapping:
      csv_source: csv_source

  migrate_contacts:
    task: migrations/migrate-contact
    depends_on: [gen_lookup, validate_csv]
    connector_mapping:
      source: csv_source
      destination: d365
      mapping_db: mapping_db

on_failure: pause
```

```mermaid The validation task gates the migration
flowchart LR
  G[gen_lookup<br/>command] --> M[migrate_contacts]
  V[validate_csv<br/>validation] --> M
```

The scheduler topologically sorts the DAG into levels and runs each level's nodes in parallel. A circular dependency is rejected with `Circular dependency detected among tasks`.

## Task nodes

Each entry under `tasks:` is a DAG node (`JobTaskNode`). The map key is the node's `key` — the identity used by `depends_on`, run history, triggers, and per-job overrides.

### `task`

The referenced Task's registered name. Required.

```yaml
tasks:
  migrate_contacts:
    task: migrations/migrate-contact-mssql
```

### `type`

`command` or `validation`; omit for a regular migration Task. Command and validation Tasks return a `TaskResult` (`status: pass | fail | warn` plus `messages`); a validation Task whose result is `status: "fail"` stops the Job. Regular Tasks are evaluated by exit code.

```yaml
  validate_contact_csv:
    task: migrations/validate-contact-csv
    type: validation
```

### `depends_on`

Array of task keys this node waits for. Nodes with no `depends_on` are roots and start immediately; siblings at the same level run in parallel. Every listed key must exist in `tasks:` — an unknown key fails parsing with `Task "<key>" depends on undefined task "<dep>"`.

```yaml
  migrate_contacts:
    task: migrations/migrate-contact-mssql
    depends_on: [validate_contact_csv]
```

### `arguments`

Key-value pairs delivered to the Task as runtime arguments, readable in the Task's JSONata as `arguments.<key>`. Values are passed verbatim — they are not JSONata and not interpolated. Only `${VAR}` placeholders (declared in the template-level `arguments:`) are substituted when the Job is created; see [Template arguments](/building/template-arguments/).

Because arguments differentiate invocations, one Task file can appear as several nodes. From `jobs/zendesk-to-gorgias.yaml`, the same `gorgias/push-tickets` Task runs in `create` mode, then again in `delta` mode:

```yaml
  push_tickets:
    task: gorgias/push-tickets
    depends_on: [push_contacts]
    connector_mapping:
      truto: truto
      mongo: mongo
    arguments:
      mode: create
      truto_migration_job_name: ${TRUTO_MIGRATION_JOB_NAME}

  delta_tickets:
    task: gorgias/push-tickets
    depends_on: [push_tickets]
    connector_mapping:
      truto: truto
      mongo: mongo
    arguments:
      mode: delta
      truto_migration_job_name: ${TRUTO_MIGRATION_JOB_NAME}
```

### `connector_mapping`

Maps the Task's connector slot names to the template's connector names — see the next section.

## Wire slots with `connector_mapping`

A Task declares its own connector slots (`connectors:` in the task YAML); a template declares its own slots at the top level. `connector_mapping` on each node wires the two, task slot → template connector:

```yaml
# Task YAML declares:    connectors: { source: { type: csv }, dest: { type: sqlite } }
# Template declares:     connectors: { csv_source: { type: csv }, db: { type: sqlite } }

connector_mapping:
  source: csv_source
  dest: db
```

Every template connector referenced in a mapping must exist in the template's `connectors:` block; the parser rejects `Task "<key>" connector_mapping references undefined connector "<name>"`. When a Job is created from the template, each template connector is bound to a stored Connector by database ID, and the mapped configs are inlined into the resolved task YAML at run time.

## Failure policies

`on_failure` decides what happens to a running Job Run when any Task in a level fails:

- `pause` (default) — the Job Run stops after the current level and its status becomes `paused`. Fix the problem and resume; completed Tasks are not re-run.
- `abort` — the Job Run stops after the current level and its status becomes `failed`.
- `continue` — remaining levels keep executing despite the failure.

Both `paused` and `failed` Job Runs can be resumed from the UI or API.

## Override a task inside a job

A Job (an instantiated template) can hold a per-DAG-node task YAML override that wins over the pinned task version at run time, for that Job only. Use it when one customer needs a Task tweaked without forking it in the task library.

- Overrides are keyed by DAG node key, so two nodes referencing the same Task can be overridden independently.
- The task library is untouched; other Jobs and the Task's version history are unaffected.
- At run time (and on resume), the override YAML is used as the base, then connectors are inlined and node arguments applied exactly as for the pinned version.
- The run's provenance still records the pinned `task_id` / `task_version` (the library Task), not the override — keep this in mind when reading run history.

The admin-only API:

- `GET /api/jobs/:id/tasks/:taskKey/yaml` → `{ task_key, task_name, base_version, yaml_content, is_overridden }`. Returns the override if present, otherwise the pinned task version YAML.
- `PUT /api/jobs/:id/tasks/:taskKey/override` with body `{ yaml_content }`. The YAML is validated with the same task parser and validator as the task library. Every connector slot in the node's `connector_mapping` must be declared in the override's `connectors:` section, and the override must not declare slots the node does not map — both cases return 400 naming the slots.
- `DELETE /api/jobs/:id/tasks/:taskKey/override` — removes the override; future runs use the pinned task version again. 404 if no override exists.
- `GET /api/jobs/:id` includes `overridden_task_keys` so clients can badge customized Tasks.

In the UI, each task row on the Job Details page has an edit action that opens the per-job task YAML editor; overridden Tasks show a **Customized** badge, and the editor's ⋮ menu has **Reset to original**. See [Jobs and job templates](/using/jobs/) for the operator side.

## Verify it worked

Validate the template before saving it:

```bash
envoy job validate jobs/fiserv-migration.yaml
```

Validation catches `depends_on` references to undefined task keys, `connector_mapping` references to undefined template connectors, invalid `on_failure` values, and `job_config` schema errors. Then create a Job from the template, bind Connectors, and start a run; the Job Run detail page shows each node executing in dependency order.

:::callout{type="warning"}
A node's `task:` value must exactly match the Task's registered name (for example `migrations/validate-contact-csv`). A mismatch fails Job creation with a `VALIDATION_ERROR` saying the task was not found in the database. If you rename a Task, update every template node that references it.
:::

## Related

- [Template arguments](/building/template-arguments/) — create-time values substituted via `${VAR}`
- [Job config: forms and field mapping](/building/job-config/) — runtime-editable per-Job configuration
- [Job template YAML](/reference/job-templates/) — every key, exhaustively
- [Jobs and job templates](/using/jobs/) — creating and running Jobs in the UI