Documentation by ClonePartner

Start typing to search.

Core Concepts

Tasks, jobs, and templates

View Markdown

Understand the three-layer model: portable Tasks, Job Templates that compose them into DAGs, and Jobs that bind Connectors.

Tasks, jobs, and templates

Why three layers

Envoy keeps pipeline logic strictly separate from deployment-specific configuration. The logic — what to read, how to transform it, where to write it — is written once as portable YAML. The configuration — which actual database, which credentials, which account — is supplied where the pipeline is deployed. Three layers make that split explicit.

flowchart LR T1[Task A] --> JT[Job Template] T2[Task B] --> JT JT -->|Create Job: bind Connectors + arguments| J[Job] C1[Connector: prod CRM] --> J C2[Connector: prod DB] --> J J -->|Run| JR[Job Run]
From portable YAML to a concrete run

Task

A Task is one streaming pipeline: an ordered list of steps that records flow through. Tasks declare connector slots — a slot name plus a connector type, never credentials — along with optional variables, arguments, and output settings. Because nothing environment-specific appears in the YAML, the same task runs on a laptop via the CLI, in a test deployment, and in production, unchanged.

Tasks live as YAML files in the tasks/ directory or in the server's tasks table, where every save creates a new entry in task_versions.

Job Template

A Job Template composes tasks into a directed acyclic graph. Each task node has a key, the task it points to, optional depends_on edges, and optional per-node arguments. The template's other keys:

  • connectors declares the template-level connector slots, and each task node's connector_mapping wires the task's slot names to those template names. Many tasks can share one template connector.
  • arguments declares create-time parameters. Their values are substituted into ${VAR} placeholders when a Job is created and baked into the Job's snapshot — right for things fixed at creation, like an account ID or a migration job ID.
  • job_config declares runtime-editable settings with defaults — field mappings, sync options, stage toggles. Operators can change these on the Job after creation, and each run snapshots the values it ran with.
  • on_failure sets the failure policy for runs: pause (default), abort, or continue.

Templates live as YAML in the jobs/ directory or the job_templates table.

Job

A Job is an instantiated template: a persistent record that binds each template connector name to an actual stored Connector by ID, carries the argument-substituted snapshot of the template, and holds the current job_config values. Jobs are created in the UI with the Create Job dialog (or via the API) and run any number of times. Each execution is a Job Run, which fans out into one task run per DAG node.

Jobs, job runs, and their per-task links live in the jobs, job_runs, and job_run_tasks tables; the individual task executions live in runs.

The layers at a glance

Layer Contains Stored in Environment-specific?
Task Steps, connector slots, variables tasks/ files or tasks + task_versions No
Job Template Task DAG, connector mapping, arguments, job config defaults jobs/ files or job_templates No
Job Connector bindings, argument values, editable job config jobs table Yes
Job Run / Run Execution state, logs, metrics job_runs, job_run_tasks, runs Yes

Standalone task runs

A task does not need a template to run. The UI's Run Now on a task (and the equivalent API call) binds Connectors to the task's slots directly for that one run, and the CLI does the same with a connectors.yaml file. Templates earn their keep when tasks need ordering, shared connectors, arguments, or operator-editable configuration.