Architecture

Envoy separates portable pipeline definitions from deployment-specific configuration across three product layers and two runtime roles.

Three product layers

Task

A task is one streaming pipeline. Its YAML declares:

• typed connector slots;
• optional runtime arguments and variables;
• an ordered list of steps;
• output and execution options.

A task contains no credentials or environment-specific connection details.

Job template

A job template is a reusable directed acyclic graph (DAG) of tasks. It:

• declares template-level connector slots;
• maps each task's slots to template slots;
• defines dependencies between task nodes;
• may collect create-time arguments;
• may define runtime-editable job_config.

Job

A job is an instantiated template. It binds concrete connector records to the template slots and stores the values chosen at creation. A job can be run repeatedly, scheduled, customized, and paired with dashboards.

Arguments and job configuration

These serve different lifecycles:

• Template arguments are selected when the job is created. ${NAME} placeholders are substituted into the saved job snapshot.
• Job configuration remains editable after creation. Current values are snapshotted when each run starts and exposed to task JSONata as job_config.

Use arguments for identifiers that define the job instance. Use job configuration for field mappings, toggles, stages, batch settings, and other behavior operators may change later.

Two runtime roles

Control plane

The control plane serves the UI and REST API. It owns:

• authentication and authorization;
• task, template, job, connector, trigger, and dashboard records;
• scheduling and orchestration;
• connector secret retrieval;
• run and job-run state;
• dispatch and status callbacks.

Before dispatch, it resolves the selected connector configurations and prepares a fully resolved task definition.

Executor

The executor runs task subprocesses. It:

• accepts dispatched work;
• enforces a concurrency limit;
• starts one isolated process per task run;
• streams logs, metrics, heartbeats, and status back to the control plane;
• reaches source and destination systems.

An executor does not resolve ${ENV_VAR} values from a CLI connector file. It receives resolved connector configuration from the control plane.

State and data boundaries

Envoy distinguishes product state from data moved by connectors:

• State backend: users, sessions, tasks, versions, connectors, templates, jobs, runs, triggers, dashboards, metrics configuration, and secret metadata.
• Run log store: per-run output, in files or a configured database/log backend.
• Connector data: source and destination systems, mounted files, object stores, and connector-specific SQLite databases.
• Secret provider: encrypted in the state database or stored in Azure Key Vault.

The state backend is not a warehouse for records moving through a task. Streaming records normally exist only in the executor process and the systems named by connector calls.

Streaming execution

Each task is a Node.js stream pipeline with backpressure. Steps can run sequentially or with bounded concurrency. Large connector reads should yield records incrementally instead of collecting a complete dataset.

Arrays returned by a step are emitted item by item unless batching or spooling collects them. Spooling deliberately buffers and should be reserved for bounded use cases.

See Task structure and execution model, Step types, and Author a production task for the execution contract and applied patterns.

Deployment shapes

The same product model is used in:

• direct-binary local execution;
• a single server using local subprocess execution;
• remote Docker with one control plane and one HTTP executor;
• Azure Container Apps with external control-plane ingress and internal executor ingress.

Read Control plane and executors and Run lifecycle next.