Documentation by ClonePartner

Start typing to search.

Operations

Live run monitoring

View Markdown

Understand how live status, metric, and log updates reach the Run detail page, configure proxies for WebSockets, and interpret gaps.

Live run monitoring

Envoy run pages combine durable run records with live updates. The control plane publishes status changes and tails metrics and logs while an authorized browser is subscribed. Live updates improve operator response time; they are not the durable source of truth and do not control task execution.

How updates reach the browser

The Run detail page opens a WebSocket to /api/ws and subscribes to the run:<id> topic for the run it is viewing (useRunSocket.ts), receiving three message types: run:status, run:log, and run:metrics. On the server, a log tailer polls the configured log store with readSince(runId, cursor) and pushes new lines to subscribers; unused tailers stop after subscribers leave.

Every run log store — file, db, mssql, postgres, and victoria_logs — implements the same RunLogStore interface with readSince cursors, so live tailing works regardless of the configured backend.

The REST API also serves logs directly (GET /api/runs/:id/logs in src/server/api/runs.ts): for terminal runs (or with the snapshot query parameter) it returns the stored log as text, optionally capped with tail bytes; for running tasks it streams new lines as server-sent events, polling the log store every 250 ms and closing with a done event when the run reaches a terminal status.

What updates live

Depending on the page and run type: task and job status transitions, per-step counters and rates, newly persisted log content, and job task transitions.

Network requirements

Your reverse proxy and load balancer must support:

  • WebSocket upgrade headers on /api/ws;
  • long-lived connections and idle timeouts longer than expected quiet periods;
  • authenticated browser connections;
  • the public origin and path used by the browser.

For multiple control-plane replicas, follow the deployment's supported event-distribution and session-routing design. An in-memory event published on one replica is not automatically visible to a client connected to another.

Verify live monitoring

  1. Start a Task that runs long enough to observe.
  2. Open its run page and confirm status changes without a reload.
  3. Confirm metrics increase and log lines arrive in order.
  4. Disconnect the browser network briefly, reconnect, and confirm stored state catches up.
  5. Complete the run and reload the page.

Repeat through the production reverse-proxy path, not only against a local port.

Interpreting gaps

A quiet page can mean: the task is legitimately idle, no new metrics were flushed, log persistence failed, the WebSocket disconnected, a proxy timed out the connection, or the control plane or executor lost contact.

Refresh the run and compare durable status, stored logs, lease state, and executor health. Do not stop or rerun a Job based only on a stale browser counter.

Operational response

When a run appears stuck:

  1. verify the run's durable status and the last status and log timestamps;
  2. inspect control-plane and executor health;
  3. confirm the external connector is responding;
  4. wait for configured lease or timeout behavior where applicable;
  5. stop only after considering partial external writes, and reconcile state before rerunning.

Data protection

Live logs can include connector and record-level errors. Use role-based access, HTTPS, and approved support-sharing practices. Do not add raw secret values to task debug output.