Documentation by ClonePartner

Start typing to search.

Troubleshooting

Run and executor issues

View Markdown

Diagnose queued or stuck runs, executor registration failures, interrupted runs, and scheduler in-flight expiry.

Run and executor issues

Dispatch problems have three usual sources: the control plane's own concurrency gate, executor availability, and the lease mechanism that detects lost work. For each entry below, check the control plane's structured log (logs/envoy.log and container stderr), the executor's container stderr, and the run's own logs in the UI.

Run stays pending and never starts

Symptom: A run sits in pending. In HTTP mode the control-plane log may show No healthy executors available with sufficient capacity.

Cause: Either the control plane's runs.max_concurrent gate is full (new runs wait until an active run finishes), or no registered executor is healthy with free capacity — offline executor, all slots busy (--max-concurrent, default 10), or an executor key mismatch preventing registration.

Resolution:

  1. Open Executors: at least one executor must be healthy with active tasks below capacity.
  2. Check active runs against runs.max_concurrent (default 50; 0 disables the gate).
  3. If no executor is listed, see the registration entry below.
  4. Do not create repeat runs while capacity is unavailable; that hides the original dispatch failure. Note that a pending run untouched for 90 seconds is failed by the reaper (PENDING_GRACE_MS).

Executor does not appear on the Executors page

Symptom: A started executor never shows up, or shows unhealthy immediately.

Cause: The executor could not register or heartbeat: missing ENVOY_EXECUTOR_KEY (the process exits at startup with [executor] ENVOY_EXECUTOR_KEY environment variable is required), a key that does not match the control plane's executor.config.executor_key, no control-plane URL, or a wrong URL.

Resolution:

  1. Check executor stderr for the startup error. Provide at least one --control-plane URL or ENVOY_CONTROL_PLANES; otherwise the command exits with Error: at least one --control-plane URL or ENVOY_CONTROL_PLANES is required.
  2. Confirm both sides use the same executor key value.
  3. Confirm the control-plane URL is reachable from the executor container, and the executor's --advertise-url is reachable from the control plane (service DNS name, not localhost).
  4. Heartbeats run every 15 seconds — a healthy executor appears within one interval.

Run fails with "Executor became unavailable — run interrupted"

Symptom: A running task flips to failed with the error message Executor became unavailable — run interrupted.

Cause: The run's lease expired. Executor heartbeats renew leases for the run IDs they report; a lease lives 60 seconds (LEASE_TTL_MS, four missed 15-second heartbeats). When the executor dies, restarts, or loses network, the run reaper (sweeping every 30 seconds) fails the orphaned run rather than leaving it running forever.

Resolution:

  1. Check whether the executor restarted or lost connectivity at that time (container events, deploy history).
  2. Review the run's logs for partial external writes before rerunning — the task process was killed mid-flight.
  3. Prevent recurrence during deploys with the drain gate: WAIT_FOR_DRAIN=true ./scripts/deploy-remote.sh.

Logs or completion arrive late after a control-plane restart

Symptom: After a control-plane restart, runs briefly show stale logs or a non-terminal status even though the executor finished the task.

Cause: When callbacks fail, the executor spools log chunks to disk (.envoy/.log-spool) and journals completions in a local SQLite journal (data/executor-journal.db), retrying delivery until the control plane returns.

Resolution: Wait for the executor's retry to drain, then reload the run. If a run was already reaped while the executor kept working, the executor is fenced on its next heartbeat: the control plane returns those run IDs in cancelRunIds and the executor kills the zombie processes.

Triggered job blocked, then suddenly fires again

Symptom: A Trigger skips executions while an old job run is stuck in flight, then resumes after roughly an hour.

Cause: The scheduler will not start a new triggered run while one is in flight. scheduler.max_inflight_age_minutes (default 60) caps how long an abandoned in-flight job run blocks its schedule; past that age the Trigger fires again.

Resolution:

  1. Find the stuck job run and resolve it — stop it or let the reaper fail it.
  2. Tune scheduler.max_inflight_age_minutes in server config if your longest legitimate job runs exceed the default.

Where to find each log

  • Dispatch decisions and reaper activity — control-plane stderr and logs/envoy.log ([run-reaper] lines).
  • Executor lifecycle, registration, spooling — executor container stderr ([executor] lines).
  • Task output — the run's logs in the UI, stored in the configured run log store.