CLI commands, configuration, and logs

Run and validate Envoy tasks and jobs, provide connector configuration, select output formats, and separate data from diagnostics.

The Envoy runtime CLI executes task files and job templates directly. Customer builds may rename the binary; the examples use envoy.

Run envoy --help against your build for the authoritative command list.

Run a task

The task name or path is the first positional argument:

envoy csv-to-sqlite \
  --config connectors.yaml \
  --arguments '{"input_file":"records.csv"}'

Options:

--config, -c — connector configuration file;
--arguments, -a — JSON object passed to declared task arguments;
--output, -o — ndjson, json, or none;
--debug — human-readable step diagnostics on stderr;
--log-level — structured file log threshold.

Task YAML can choose its own output format. A CLI --output value overrides it.

Validate a task

envoy validate tasks/csv-to-sqlite.yaml

Validation checks the task definition without running connector methods. A valid result does not prove credentials, network access, source shape, or destination permissions.

Validate a job template

envoy job validate jobs/kitchen-sink.yaml

This checks the template structure, dependency graph, connector mappings, argument placeholders, and job-configuration schema.

Start a control plane

Customer server builds expose the server entry point through the same binary:

envoy server --config /path/to/envoy-server.yaml

When --config is omitted, the server entry point looks for envoy-server.yaml in the working directory. See Server configuration.

Start an executor

envoy executor \
  --control-plane https://envoy.example.com \
  --port 9090 \
  --max-concurrent 10 \
  --advertise-url http://executor.internal:9090

Executor options:

--control-plane — required control-plane URL; repeat the option or provide comma-separated values;
--port — executor listen port;
--max-concurrent — maximum active task processes;
--labels — JSON object used for executor selection;
--advertise-url — executor URL reachable from the control plane.

ENVOY_CONTROL_PLANES can supply comma-separated control-plane URLs. The executor key is supplied through deployment configuration, not a command-line option.

See Control plane and executors.

Run selected job nodes

envoy job run jobs/kitchen-sink.yaml \
  --config connectors.yaml \
  --tasks pull_accounts,push_accounts

Dependencies of selected nodes are resolved by default. --skip-deps bypasses that behavior:

envoy job run jobs/kitchen-sink.yaml \
  --config connectors.yaml \
  --tasks push_accounts \
  --skip-deps

Use --skip-deps only when required upstream state already exists and the selected task is independently safe.

--max-concurrent limits how many task nodes in one dependency level execute concurrently. It does not replace step-level concurrency inside each task.

Job state and resume

CLI job runs write state files under .envoy/job-runs by default. Inspect one with:

envoy job status .envoy/job-runs/example-1720000000000.json

Resume a paused or failed run:

envoy job resume .envoy/job-runs/example-1720000000000.json \
  --config connectors.yaml

Back up the state file when it is needed for an operational recovery. Resume uses the recorded graph state; it does not create a new server-side job-run record.

`connectors.yaml`

CLI connector configuration uses names matching task or template slots:

connectors:
  input_csv:
    type: csv
    config:
      directory: ./data/input
 
  output_db:
    type: sqlite
    config:
      path: ./data/output.db

Credentials can reference environment variables:

connectors:
  source:
    type: postgres
    config:
      host: ${PGHOST}
      user: ${PGUSER}
      password: ${PGPASSWORD}
      database: ${PGDATABASE}

Missing variables and connector type mismatches fail before task execution. Keep the file outside source control and restrict file permissions.

The CLI does not read connector records stored by the control plane. Likewise, the server does not use a CLI connectors.yaml to bind stored jobs.

Standard output and standard error

The streams have distinct purposes:

stdout is task data;
stderr is user-facing status and debug output;
logs/envoy.log is structured JSON logging.

This makes NDJSON safe to pipe:

envoy export-records --config connectors.yaml --output ndjson \
  > records.ndjson

Do not merge stderr into an output data file.

ndjson streams one envelope per output record and is appropriate for large results. json collects the result before printing and can consume substantial memory. none suppresses task data while execution and metrics still occur.

Prefer ndjson for pipelines and none when the destination connector is the only intended output.

Debug and structured logs

Use --debug for temporary, human-readable step input, config, and output details. A task step can also enable its own debug output.

Debug data can contain source records. Enable it narrowly, protect the log directory, and remove retained logs according to policy.

See Troubleshooting, Task structure, and Job templates.