Documentation by ClonePartner

Start typing to search.

Deployment

Backups and upgrades

View Markdown

Back up Envoy state per store type, replicate SQLite with Litestream, protect encryption keys, and upgrade without losing active runs.

Backups and upgrades

A recoverable Envoy backup includes more than a container image. Protect the application state, the encryption material required to read it, and every file-backed connector database your workloads depend on.

What the state contains

The application database (store.type: sqlite, mssql, or postgres) holds all product state in one schema: users and sessions, connectors (with encrypted secrets), tasks and task versions, job templates, jobs and job runs, runs and run log chunks (for the db log store), triggers, dashboards, metrics configuration, API keys, explorer presets, and AI conversations and references. There is no per-feature database to inventory separately — back up the one configured store plus the assets below.

Backup inventory

For a standard remote Docker deployment:

Asset Typical location Why it matters
Server configuration /opt/envoy/state/envoy-server.yaml Backend, auth, retention, executor, and branding settings
SQLite state /opt/envoy/state/envoy-server.db All product state listed above
Data files /opt/envoy/data CSV/YAML inputs and outputs
Connector databases /opt/envoy/dbs Mapping, staging, and workload-specific SQLite data
Encryption key Protected secret store Required to decrypt in-database secrets
Executor key Protected secret store Required for split-runtime communication

If state or logs live in PostgreSQL or MSSQL, use that service's backup facilities instead of assuming /app/state contains them.

SQLite with Litestream

The Docker entrypoint generates a Litestream configuration when the store is SQLite and the replica settings are complete. For an S3-compatible target:

LITESTREAM_S3_ENDPOINT=https://<endpoint>
LITESTREAM_S3_BUCKET=envoy-backups
LITESTREAM_S3_ACCESS_KEY_ID=<key-id>
LITESTREAM_S3_SECRET_ACCESS_KEY=<secret>
LITESTREAM_S3_PATH=production/envoy-server.db

For Azure Blob Storage, set LITESTREAM_STORAGE_ACCOUNT (and optionally LITESTREAM_BLOB_CONTAINER, default envoy-litestream) instead. If LITESTREAM_S3_ENDPOINT is set but bucket or credentials are missing, the entrypoint logs a skip message and starts without replication.

To also replicate a directory of workload SQLite databases:

LITESTREAM_DBS_DIR=/app/dbs
LITESTREAM_S3_DBS_PATH=production/dbs
LITESTREAM_DBS_PATTERN=*.db

The directory watcher (watch: true) discovers matching databases created after startup. Replication runs with a 10-second sync interval, hourly snapshots, and 24-hour replica retention as generated.

On startup, if the state database file does not exist and a Litestream config is present, the entrypoint runs litestream restore -if-replica-exists before starting the server. It never overwrites an existing database.

Managed databases

For PostgreSQL, MSSQL, and Azure SQL:

  • enable the provider's automated backups and document point-in-time recovery limits;
  • protect connection credentials separately;
  • include the server YAML and encryption key in the recovery set;
  • confirm run-log and metrics databases are covered if they use a separate backend.

Azure Key Vault recovery settings are independent from Azure SQL recovery — enable soft-delete and purge protection per policy.

Upgrade preparation

Before changing an image or binary:

  1. Review release-specific migration instructions.
  2. Take or verify a current backup.
  3. Record the deployed image tag and configuration.
  4. Check for active task and job runs.
  5. Confirm the new build includes every connector type used by stored Tasks and Jobs.
  6. Plan control-plane and executor restart order.

Review Seed import — startup import is create-only, so bundled definition changes do not update existing database records.

Drain active work

Do not recreate an executor while it owns active task processes. The remote deployment script checks activeRunsByLease on /api/health and can wait:

WAIT_FOR_DRAIN=true ./scripts/deploy-remote.sh

If a run must be stopped, stop it through Envoy and wait for a terminal status before deploying.

Apply the upgrade

./scripts/deploy-remote.sh
DEPLOY_SCOPE=cp ./scripts/deploy-remote.sh
DEPLOY_SCOPE=ex ./scripts/deploy-remote.sh

The Azure script supports the same DEPLOY_SCOPE values; Terraform users should review terraform plan before applying.

Verify after upgrade

  • /api/health reports a healthy server;
  • expected executors are healthy and heartbeating;
  • the UI loads under the configured browser origin;
  • stored Connectors can be decrypted and tested;
  • one bounded run completes with logs and metrics;
  • Triggers remain enabled as intended;
  • public dashboard controls have not become more permissive.

Rollback

Rollback normally means redeploying the prior immutable image against the same state and keys. Restore an old database only when schema compatibility requires it and the release's rollback procedure says so. Never run docker compose down -v as an upgrade or rollback step.