Documentation by ClonePartner

Start typing to search.

Operations

Metrics and observability

View Markdown

Operate Envoy metrics backends, read run summaries through the metrics API, and separate structured logs from measurements.

Metrics and observability

Envoy records task, step, and connector measurements during execution, uses them for run summaries, and can send them to configured backends. This page covers the operations view; the per-backend YAML schema lives in Metrics config.

What is measured

Task measurements include total duration, output item count, and completion status. Step measurements include wall-clock duration, per-item execution duration, items in and out, execution errors, in-flight calls, and items skipped by conditions. Connector measurements include call count, duration, and errors.

Backends

The backend registry (src/metrics/index.ts) registers five types: noop, sqlite, statsd, http, and mssql. When no backend is configured, the runtime uses a no-op collector — tasks run, but nothing is recorded.

  • sqlite writes to a local metrics database (separate from the state database).
  • mssql writes to a SQL Server database, which works across remote executors.
  • statsd emits UDP packets to a compatible agent (Datadog Agent, VictoriaMetrics, Telegraf, Graphite). UDP is fire-and-forget: lost packets do not fail the task.
  • http posts metrics to the control plane's ingest endpoint (POST /api/executor-callbacks/metrics/ingest, authenticated with the executor key), which stores them through the server's enabled backends.

Server metrics configuration

The control plane maintains metrics-backend rows editable under Settings. The API (src/server/api/metrics.ts):

  • GET /api/metrics/config — list configured backends;
  • POST /api/metrics/config, PUT /api/metrics/config/:id, DELETE /api/metrics/config/:id — manage them (requires the superadmin or admin role);
  • GET /api/metrics/data/summary/:taskId and GET /api/metrics/data/timeseries/:taskId — read stored measurements for run views.

Enabled server configuration is injected into a task run when the task YAML does not define its own metrics section. A new server creates a default durable backend matching its state configuration: SQLite deployments get a sqlite backend at ./metrics.db; deployments with an mssql section get an MSSQL backend and the SQLite default is disabled (it would not work from remote executors).

After changing a backend, run a small Task and confirm the run detail shows a metrics summary.

Per-task configuration

A task can declare its own backends, which overrides the injected server config:

metrics:
  - backend: sqlite
    config:
      path: ./metrics.db
 
  - backend: statsd
    config:
      host: ${STATSD_HOST}
      port: 8125
      prefix: envoy

Use a task-level section when the task must override deployment defaults or runs entirely outside the control plane. Key tables per backend are in Metrics config.

Run summaries

The run detail page reads metrics for that run and displays aggregate task and step values. Use it to identify a step with disproportionate wall time, item loss or amplification between steps, high connector call counts, errors hidden by per-item continue behavior, and concurrency that is not producing throughput.

Metrics describe what was measured; logs explain individual failures. Review both.

Structured logs

Structured server logs are separate from metrics and from per-run logs. The unified pino logger (src/logger.ts) writes JSON to logs/envoy.log with levels trace, debug, info, warn, error, fatal, and silent. Per-run task output goes to the configured run log store instead — see Live run monitoring.

Capacity signals

Useful starting signals: task duration by task name, step items per second, connector error count and latency, executor active tasks versus capacity, pending and failed run counts, and state and log-store disk usage.

Do not infer record correctness from a success counter. Add explicit validation steps and destination checks to migration workflows.

Data management

Metrics storage has its own retention behavior; SQLite backends support age-based pruning in their backend config. Deleting a run through Envoy also removes associated run metrics from the server metrics store on a best-effort basis. Back up a metrics database only when historical performance data is part of the recovery objective.