# Architecture

> Source: https://docs.clonepartner.com/concepts/architecture/

Envoy separates portable pipeline definitions from deployment-specific configuration across three product layers and two runtime roles.

## Three product layers

### Task

A task is one streaming pipeline. Its YAML declares:

- typed connector slots;
- optional runtime arguments and variables;
- an ordered list of steps;
- output and execution options.

A task contains no credentials or environment-specific connection details.

### Job template

A job template is a reusable directed acyclic graph (DAG) of tasks. It:

- declares template-level connector slots;
- maps each task's slots to template slots;
- defines dependencies between task nodes;
- may collect create-time arguments;
- may define runtime-editable `job_config`.

### Job

A job is an instantiated template. It binds concrete connector records to the template slots and stores the values chosen at creation. A job can be run repeatedly, scheduled, customized, and paired with dashboards.

```mermaid Definition and execution flow
flowchart TD
  task["Task definitions"] --> template["Job template"]
  template --> job["Job"]
  connectors["Stored connectors"] --> job
  job --> jobrun["Job run"]
  jobrun --> taskruns["Task runs"]
```

## Arguments and job configuration

These serve different lifecycles:

- **Template arguments** are selected when the job is created. `${NAME}` placeholders are substituted into the saved job snapshot.
- **Job configuration** remains editable after creation. Current values are snapshotted when each run starts and exposed to task JSONata as `job_config`.

Use arguments for identifiers that define the job instance. Use job configuration for field mappings, toggles, stages, batch settings, and other behavior operators may change later.

## Two runtime roles

### Control plane

The control plane serves the UI and REST API. It owns:

- authentication and authorization;
- task, template, job, connector, trigger, and dashboard records;
- scheduling and orchestration;
- connector secret retrieval;
- run and job-run state;
- dispatch and status callbacks.

Before dispatch, it resolves the selected connector configurations and prepares a fully resolved task definition.

### Executor

The executor runs task subprocesses. It:

- accepts dispatched work;
- enforces a concurrency limit;
- starts one isolated process per task run;
- streams logs, metrics, heartbeats, and status back to the control plane;
- reaches source and destination systems.

An executor does not resolve `${ENV_VAR}` values from a CLI connector file. It receives resolved connector configuration from the control plane.

## State and data boundaries

Envoy distinguishes product state from data moved by connectors:

- **State backend**: users, sessions, tasks, versions, connectors, templates, jobs, runs, triggers, dashboards, metrics configuration, and secret metadata.
- **Run log store**: per-run output, in files or a configured database/log backend.
- **Connector data**: source and destination systems, mounted files, object stores, and connector-specific SQLite databases.
- **Secret provider**: encrypted in the state database or stored in Azure Key Vault.

The state backend is not a warehouse for records moving through a task. Streaming records normally exist only in the executor process and the systems named by connector calls.

## Streaming execution

Each task is a Node.js stream pipeline with backpressure. Steps can run sequentially or with bounded concurrency. Large connector reads should yield records incrementally instead of collecting a complete dataset.

Arrays returned by a step are emitted item by item unless batching or spooling collects them. Spooling deliberately buffers and should be reserved for bounded use cases.

See [Task structure and execution model](/reference/tasks/task-structure), [Step types](/reference/tasks/step-types), and [Author a production task](/guides/task-authoring) for the execution contract and applied patterns.

## Deployment shapes

The same product model is used in:

- direct-binary local execution;
- a single server using local subprocess execution;
- remote Docker with one control plane and one HTTP executor;
- Azure Container Apps with external control-plane ingress and internal executor ingress.

Read [Control plane and executors](/deployment/control-plane-and-executors) and [Run lifecycle](/concepts/run-lifecycle) next.