Documentation by ClonePartner

Start typing to search.

Operations

VictoriaLogs

View Markdown

Configure VictoriaLogs as the run-log backend with url, tenant IDs, extra fields, and decolorize, and verify ingestion and tailing.

VictoriaLogs

Envoy can persist task-run logs in VictoriaLogs instead of local files or the application database. Use it when control-plane replicas need a shared, scalable log backend. This backend stores run logs only — not the structured server log or Envoy database state.

Configure the backend

runs:
  log_store: victoria_logs
 
victoria_logs:
  url: https://victoria-logs.example.com
  account_id: 0
  project_id: 0
  decolorize: true
  extra_fields:
    environment: production
    service: envoy

The keys, verified in src/server/config.ts and src/server/runner/log-store.ts:

Key Type Required Default Description
url string Yes — VictoriaLogs base URL. Boot fails with runs.log_store is "victoria_logs" but no victoria_logs.url is provided when missing.
account_id number No 0 Tenant account ID, sent as the AccountID header when non-zero.
project_id number No 0 Tenant project ID, sent as the ProjectID header when non-zero.
extra_fields map of string No — Static fields attached to every inserted record — useful for environment-level filtering.
decolorize boolean No true Strip terminal color codes from the message field on ingest.

Do not put credentials in the URL. Configure authentication at a proxy or an infrastructure layer that injects it.

Data model and live tailing

Envoy writes JSON lines with run_id as the stream field. The store assigns strictly increasing microsecond timestamps and serializes writes per run, so cursor-based live tailing (readSince) never misses out-of-order inserts. Deleting a run's logs issues a delete query scoped to that run_id.

Executor-side backend

In HTTP executor mode, the control plane can instruct executors to write run logs directly to VictoriaLogs — the executor implements a victoria_logs log backend (src/executor/server.ts) alongside http, sqlite, and mssql. The VictoriaLogs endpoint must then be reachable from the executor network, not only from the control plane.

Verify the integration

  1. Start Envoy and confirm configuration validation succeeds.
  2. Run a small Task that emits several log lines.
  3. Open the run and verify stored logs render.
  4. Keep the page open and verify live lines arrive in order.
  5. Reload after completion and confirm the same lines are queryable.
  6. Verify extra_fields values on records inside VictoriaLogs.

Retention and deletion

Envoy's run-retention process calls the log store's delete path, but VictoriaLogs deletion must be explicitly enabled by its administrator. If deletion is disabled or forbidden, removal is best-effort and the backend's own retention policy remains authoritative. Align Envoy run retention, VictoriaLogs retention, and backup or legal-hold requirements — deleting an Envoy run record does not guarantee immediate physical deletion from an external log backend.

Troubleshooting

  • Insert errors — verify URL, proxy authentication, network policy, and VictoriaLogs health.
  • Query errors — verify tenant IDs and read permissions.
  • Live logs stop — reload the run to distinguish WebSocket trouble from backend query failure.
  • Logs remain after retention — verify VictoriaLogs deletion is enabled or shorten backend retention.