Job templates and DAGs

A Job Template is a YAML file that composes Tasks into a DAG: it declares type-only connector slots, wires each task node to those slots, and sets a failure policy. Write one when a pipeline spans more than one Task — validation before migration, parallel pulls feeding a load, or a reusable Task run twice with different arguments.

Prerequisites

• One or more Tasks already saved in the task library — a template only references Tasks by name.
• Familiarity with task anatomy and connector slots.

Anatomy of a job template

The parser (parseJobDefinition in src/job-parser.ts) accepts these top-level keys:

See Job template YAML for the exhaustive schema, Template arguments for arguments, and Job config for job_config.

A complete template

Adapted from the Fiserv migration template in the Envoy repo — a setup Task and a validation Task both gate the migration Task:

name: fiserv-migration
description: Migrate from Enact CRM to D365
 
connectors:
  d365:
    type: dynamics365
  mapping_db:
    type: sqlite
  csv_source:
    type: csv
 
tasks:
  gen_lookup:
    task: migrations/gen-contact-lookup
    type: command
    connector_mapping:
      mapping_db: mapping_db
      csv_source: csv_source
 
  validate_csv:
    task: migrations/validate-contact-csv
    type: validation
    connector_mapping:
      csv_source: csv_source
 
  migrate_contacts:
    task: migrations/migrate-contact
    depends_on: [gen_lookup, validate_csv]
    connector_mapping:
      source: csv_source
      destination: d365
      mapping_db: mapping_db
 
on_failure: pause

The scheduler topologically sorts the DAG into levels and runs each level's nodes in parallel. A circular dependency is rejected with Circular dependency detected among tasks.

Task nodes

Each entry under tasks: is a DAG node (JobTaskNode). The map key is the node's key — the identity used by depends_on, run history, triggers, and per-job overrides.

task

The referenced Task's registered name. Required.

tasks:
  migrate_contacts:
    task: migrations/migrate-contact-mssql

type

command or validation; omit for a regular migration Task. Command and validation Tasks return a TaskResult (status: pass | fail | warn plus messages); a validation Task whose result is status: "fail" stops the Job. Regular Tasks are evaluated by exit code.

  validate_contact_csv:
    task: migrations/validate-contact-csv
    type: validation

depends_on

Array of task keys this node waits for. Nodes with no depends_on are roots and start immediately; siblings at the same level run in parallel. Every listed key must exist in tasks: — an unknown key fails parsing with Task "<key>" depends on undefined task "<dep>".

  migrate_contacts:
    task: migrations/migrate-contact-mssql
    depends_on: [validate_contact_csv]

arguments

Key-value pairs delivered to the Task as runtime arguments, readable in the Task's JSONata as arguments.<key>. Values are passed verbatim — they are not JSONata and not interpolated. Only ${VAR} placeholders (declared in the template-level arguments:) are substituted when the Job is created; see Template arguments.

Because arguments differentiate invocations, one Task file can appear as several nodes. From jobs/zendesk-to-gorgias.yaml, the same gorgias/push-tickets Task runs in create mode, then again in delta mode:

  push_tickets:
    task: gorgias/push-tickets
    depends_on: [push_contacts]
    connector_mapping:
      truto: truto
      mongo: mongo
    arguments:
      mode: create
      truto_migration_job_name: ${TRUTO_MIGRATION_JOB_NAME}
 
  delta_tickets:
    task: gorgias/push-tickets
    depends_on: [push_tickets]
    connector_mapping:
      truto: truto
      mongo: mongo
    arguments:
      mode: delta
      truto_migration_job_name: ${TRUTO_MIGRATION_JOB_NAME}

connector_mapping

Maps the Task's connector slot names to the template's connector names — see the next section.

Wire slots with connector_mapping

A Task declares its own connector slots (connectors: in the task YAML); a template declares its own slots at the top level. connector_mapping on each node wires the two, task slot → template connector:

# Task YAML declares:    connectors: { source: { type: csv }, dest: { type: sqlite } }
# Template declares:     connectors: { csv_source: { type: csv }, db: { type: sqlite } }
 
connector_mapping:
  source: csv_source
  dest: db

Every template connector referenced in a mapping must exist in the template's connectors: block; the parser rejects Task "<key>" connector_mapping references undefined connector "<name>". When a Job is created from the template, each template connector is bound to a stored Connector by database ID, and the mapped configs are inlined into the resolved task YAML at run time.

Failure policies

on_failure decides what happens to a running Job Run when any Task in a level fails:

• pause (default) — the Job Run stops after the current level and its status becomes paused. Fix the problem and resume; completed Tasks are not re-run.
• abort — the Job Run stops after the current level and its status becomes failed.
• continue — remaining levels keep executing despite the failure.

Both paused and failed Job Runs can be resumed from the UI or API.

Override a task inside a job

A Job (an instantiated template) can hold a per-DAG-node task YAML override that wins over the pinned task version at run time, for that Job only. Use it when one customer needs a Task tweaked without forking it in the task library.

• Overrides are keyed by DAG node key, so two nodes referencing the same Task can be overridden independently.
• The task library is untouched; other Jobs and the Task's version history are unaffected.
• At run time (and on resume), the override YAML is used as the base, then connectors are inlined and node arguments applied exactly as for the pinned version.
• The run's provenance still records the pinned task_id / task_version (the library Task), not the override — keep this in mind when reading run history.

The admin-only API:

• GET /api/jobs/:id/tasks/:taskKey/yaml → { task_key, task_name, base_version, yaml_content, is_overridden }. Returns the override if present, otherwise the pinned task version YAML.
• PUT /api/jobs/:id/tasks/:taskKey/override with body { yaml_content }. The YAML is validated with the same task parser and validator as the task library. Every connector slot in the node's connector_mapping must be declared in the override's connectors: section, and the override must not declare slots the node does not map — both cases return 400 naming the slots.
• DELETE /api/jobs/:id/tasks/:taskKey/override — removes the override; future runs use the pinned task version again. 404 if no override exists.
• GET /api/jobs/:id includes overridden_task_keys so clients can badge customized Tasks.

In the UI, each task row on the Job Details page has an edit action that opens the per-job task YAML editor; overridden Tasks show a Customized badge, and the editor's ⋮ menu has Reset to original. See Jobs and job templates for the operator side.

Verify it worked

Validate the template before saving it:

envoy job validate jobs/fiserv-migration.yaml

Validation catches depends_on references to undefined task keys, connector_mapping references to undefined template connectors, invalid on_failure values, and job_config schema errors. Then create a Job from the template, bind Connectors, and start a run; the Job Run detail page shows each node executing in dependency order.

Related

• Template arguments — create-time values substituted via ${VAR}
• Job config: forms and field mapping — runtime-editable per-Job configuration
• Job template YAML — every key, exhaustively
• Jobs and job templates — creating and running Jobs in the UI