# Metrics and observability

> Source: https://docs.clonepartner.com/operations/metrics/

<!-- sources: src/metrics/index.ts, src/server/api/metrics.ts, references/envoy/metrics.md, src/logger.ts -->

# Metrics and observability

Envoy records task, step, and connector measurements during execution, uses them for run summaries, and can send them to configured backends. This page covers the operations view; the per-backend YAML schema lives in [Metrics config](/reference/metrics/).

## What is measured

Task measurements include total duration, output item count, and completion status. Step measurements include wall-clock duration, per-item execution duration, items in and out, execution errors, in-flight calls, and items skipped by conditions. Connector measurements include call count, duration, and errors.

## Backends

The backend registry (`src/metrics/index.ts`) registers five types: `noop`, `sqlite`, `statsd`, `http`, and `mssql`. When no backend is configured, the runtime uses a no-op collector — tasks run, but nothing is recorded.

- `sqlite` writes to a local metrics database (separate from the state database).
- `mssql` writes to a SQL Server database, which works across remote executors.
- `statsd` emits UDP packets to a compatible agent (Datadog Agent, VictoriaMetrics, Telegraf, Graphite). UDP is fire-and-forget: lost packets do not fail the task.
- `http` posts metrics to the control plane's ingest endpoint (`POST /api/executor-callbacks/metrics/ingest`, authenticated with the executor key), which stores them through the server's enabled backends.

## Server metrics configuration

The control plane maintains metrics-backend rows editable under **Settings**. The API (`src/server/api/metrics.ts`):

- `GET /api/metrics/config` — list configured backends;
- `POST /api/metrics/config`, `PUT /api/metrics/config/:id`, `DELETE /api/metrics/config/:id` — manage them (requires the `superadmin` or `admin` role);
- `GET /api/metrics/data/summary/:taskId` and `GET /api/metrics/data/timeseries/:taskId` — read stored measurements for run views.

Enabled server configuration is injected into a task run when the task YAML does not define its own `metrics` section. A new server creates a default durable backend matching its state configuration: SQLite deployments get a `sqlite` backend at `./metrics.db`; deployments with an `mssql` section get an MSSQL backend and the SQLite default is disabled (it would not work from remote executors).

After changing a backend, run a small Task and confirm the run detail shows a metrics summary.

## Per-task configuration

A task can declare its own backends, which overrides the injected server config:

```yaml
metrics:
  - backend: sqlite
    config:
      path: ./metrics.db

  - backend: statsd
    config:
      host: ${STATSD_HOST}
      port: 8125
      prefix: envoy
```

Use a task-level section when the task must override deployment defaults or runs entirely outside the control plane. Key tables per backend are in [Metrics config](/reference/metrics/).

## Run summaries

The run detail page reads metrics for that run and displays aggregate task and step values. Use it to identify a step with disproportionate wall time, item loss or amplification between steps, high connector call counts, errors hidden by per-item continue behavior, and concurrency that is not producing throughput.

Metrics describe what was measured; logs explain individual failures. Review both.

## Structured logs

Structured server logs are separate from metrics and from per-run logs. The unified pino logger (`src/logger.ts`) writes JSON to `logs/envoy.log` with levels `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `silent`. Per-run task output goes to the configured run log store instead — see [Live run monitoring](/operations/live-run-monitoring/).

## Capacity signals

Useful starting signals: task duration by task name, step items per second, connector error count and latency, executor active tasks versus capacity, pending and failed run counts, and state and log-store disk usage.

Do not infer record correctness from a success counter. Add explicit validation steps and destination checks to migration workflows.

## Data management

Metrics storage has its own retention behavior; SQLite backends support age-based pruning in their backend config. Deleting a run through Envoy also removes associated run metrics from the server metrics store on a best-effort basis. Back up a metrics database only when historical performance data is part of the recovery objective.

## Related

- [Metrics config](/reference/metrics/) — per-backend YAML key tables
- [Retention](/operations/retention/) — how run deletion interacts with metrics
- [Live run monitoring](/operations/live-run-monitoring/) — live counters on the run page