# Run and executor issues

> Source: https://docs.clonepartner.com/troubleshooting/runs-and-executors/

<!-- sources: src/server/runner/manager.ts, src/server/runner/adapters/http.ts, src/server/scheduler/run-reaper.ts, src/server/scheduler/lease.ts, src/executor/server.ts, src/server/api/executors.ts, src/server/config.ts -->

# Run and executor issues

Dispatch problems have three usual sources: the control plane's own concurrency gate, executor availability, and the lease mechanism that detects lost work. For each entry below, check the control plane's structured log (`logs/envoy.log` and container stderr), the executor's container stderr, and the run's own logs in the UI.

## Run stays pending and never starts

**Symptom:** A run sits in `pending`. In HTTP mode the control-plane log may show `No healthy executors available with sufficient capacity`.

**Cause:** Either the control plane's `runs.max_concurrent` gate is full (new runs wait until an active run finishes), or no registered executor is healthy with free capacity — offline executor, all slots busy (`--max-concurrent`, default 10), or an executor key mismatch preventing registration.

**Resolution:**

1. Open **Executors**: at least one executor must be healthy with active tasks below capacity.
2. Check active runs against `runs.max_concurrent` (default 50; `0` disables the gate).
3. If no executor is listed, see the registration entry below.
4. Do not create repeat runs while capacity is unavailable; that hides the original dispatch failure. Note that a pending run untouched for 90 seconds is failed by the reaper (`PENDING_GRACE_MS`).

## Executor does not appear on the Executors page

**Symptom:** A started executor never shows up, or shows unhealthy immediately.

**Cause:** The executor could not register or heartbeat: missing `ENVOY_EXECUTOR_KEY` (the process exits at startup with `[executor] ENVOY_EXECUTOR_KEY environment variable is required`), a key that does not match the control plane's `executor.config.executor_key`, no control-plane URL, or a wrong URL.

**Resolution:**

1. Check executor stderr for the startup error. Provide at least one `--control-plane` URL or `ENVOY_CONTROL_PLANES`; otherwise the command exits with `Error: at least one --control-plane URL or ENVOY_CONTROL_PLANES is required`.
2. Confirm both sides use the same executor key value.
3. Confirm the control-plane URL is reachable from the executor container, and the executor's `--advertise-url` is reachable from the control plane (service DNS name, not `localhost`).
4. Heartbeats run every 15 seconds — a healthy executor appears within one interval.

## Run fails with "Executor became unavailable — run interrupted"

**Symptom:** A running task flips to `failed` with the error message `Executor became unavailable — run interrupted`.

**Cause:** The run's lease expired. Executor heartbeats renew leases for the run IDs they report; a lease lives 60 seconds (`LEASE_TTL_MS`, four missed 15-second heartbeats). When the executor dies, restarts, or loses network, the run reaper (sweeping every 30 seconds) fails the orphaned run rather than leaving it running forever.

**Resolution:**

1. Check whether the executor restarted or lost connectivity at that time (container events, deploy history).
2. Review the run's logs for partial external writes before rerunning — the task process was killed mid-flight.
3. Prevent recurrence during deploys with the drain gate: `WAIT_FOR_DRAIN=true ./scripts/deploy-remote.sh`.

## Logs or completion arrive late after a control-plane restart

**Symptom:** After a control-plane restart, runs briefly show stale logs or a non-terminal status even though the executor finished the task.

**Cause:** When callbacks fail, the executor spools log chunks to disk (`.envoy/.log-spool`) and journals completions in a local SQLite journal (`data/executor-journal.db`), retrying delivery until the control plane returns.

**Resolution:** Wait for the executor's retry to drain, then reload the run. If a run was already reaped while the executor kept working, the executor is fenced on its next heartbeat: the control plane returns those run IDs in `cancelRunIds` and the executor kills the zombie processes.

## Triggered job blocked, then suddenly fires again

**Symptom:** A Trigger skips executions while an old job run is stuck in flight, then resumes after roughly an hour.

**Cause:** The scheduler will not start a new triggered run while one is in flight. `scheduler.max_inflight_age_minutes` (default 60) caps how long an abandoned in-flight job run blocks its schedule; past that age the Trigger fires again.

**Resolution:**

1. Find the stuck job run and resolve it — stop it or let the reaper fail it.
2. Tune `scheduler.max_inflight_age_minutes` in server config if your longest legitimate job runs exceed the default.

## Where to find each log

- **Dispatch decisions and reaper activity** — control-plane stderr and `logs/envoy.log` (`[run-reaper]` lines).
- **Executor lifecycle, registration, spooling** — executor container stderr (`[executor]` lines).
- **Task output** — the run's logs in the UI, stored in the configured run log store.

:::callout{type="tip"}
Collect the run ID, executor URL, and the timestamps of the last heartbeat and status change before escalating. Do not edit run rows in the database by hand.
:::

## Related

- [Control plane and executors](/deployment/control-plane-and-executors/) — registration, heartbeats, leases, and callbacks
- [Live run monitoring](/operations/live-run-monitoring/) — distinguishing a stale page from a stuck run
- [Server configuration](/deployment/server-configuration/) — `runs.max_concurrent`, `executor`, and `scheduler` keys