Start typing to search.

Building Pipelines

Bundles: export and import

View Markdown

Export portable NDJSON bundles, preview imports, choose conflict and version policies, and promote tasks and job templates safely.

Bundles: export and import

Envoy bundles move task definitions and job templates between control planes without moving connector credentials, Jobs, runs, or customer data. A bundle is newline-delimited JSON (NDJSON), so large exports and import results can stream.

Use bundles for promotion from development to production, controlled handoff between deployments, and backup of authored definitions. Use normal state backups for disaster recovery.

What a bundle contains

A bundle can include:

  • current task YAML and task metadata (name, description, status, current_version);
  • connector slot names and types parsed from each task's YAML, but no connector configurations;
  • optional task version history (include_versions);
  • job template YAML;
  • a manifest line describing the export and its filters.

Tasks appear before job templates, so an import can validate template task references. Dashboard instances, instantiated Jobs, connector bindings, runtime configuration values, secrets, and run history are not included.

Export from the UI

On Tasks, select one or more rows and choose Export, or use a row's overflow menu Export action for a single Task. When the whole result set is selected, the export covers all Tasks matching the current search and status filter.

The UI export includes current task versions only. Use the API when you also need job templates or complete task version history.

Export through the API

Create an API key as described in Using the API, then request a bundle (admin only):

curl --fail-with-body \
  -H "Authorization: Bearer $ENVOY_API_KEY" \
  "https://your-envoy.example.com/api/export?include_tasks=true&include_job_templates=true&include_versions=true" \
  --output envoy-bundle.ndjson

Useful filters include task_name, job_template_name, include_archived, and status. Repeat the singular name parameters when exporting several exact names:

/api/export?task_name=pull-tickets&task_name=push-tickets

Treat bundles as configuration artifacts. Review and protect them even though they do not contain connector secrets; YAML can still reveal resource names, business rules, and data shapes.

Always preview an import

From Tasks, choose Import, pick the .ndjson file with Choose file, select policies, and run Preview import. The preview (a server-side dry run) parses and validates every line without writing rows.

Review the Dry-run preview before confirming. The summary counts each action:

  • create — a new task or template;
  • update — changed content, status, or imported history;
  • unchanged — equivalent YAML and status already exist;
  • skip — excluded by policy;
  • error — invalid or unavailable on the target.

An import rejects task YAML that fails validation, connector types not registered on the target, inconsistent version history, invalid DAGs, and job templates that reference missing tasks. The Confirm import button stays disabled while the preview has errors.

Choose an import mode

The Existing items option maps to the API's mode:

Mode UI label Use it when
create_only Create new only The target must never alter an existing named definition
upsert_current Create or update current version The bundle's current definition should become the target's next current version
upsert_all_versions Create or update + fill in missing versions You also need missing historical task versions

For name collisions (When names collide, conflict_policy), choose skip (Keep existing), overwrite (Overwrite with bundle), or error (Fail that item). overwrite creates a new task version when task content differs; it does not rewrite an existing version.

Archived task policy (Archived tasks in the bundle, archived_task_policy) is separate:

  • skip (Don't import) ignores archived tasks in the bundle;
  • restore (Import as active) imports them as active;
  • keep_archived (Import as archived) preserves archived status.

By default, import fails a task whose YAML name: differs from the name in the bundle. Enable Allow YAML name mismatches (normalize_names) only after deliberately reviewing the mismatch; the bundle name is kept.

API preview and import

Preview first with dry_run=true (admin only):

curl --fail-with-body \
  -X POST \
  -H "Authorization: Bearer $ENVOY_API_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @envoy-bundle.ndjson \
  "https://your-envoy.example.com/api/import?mode=upsert_current&conflict_policy=overwrite&dry_run=true"

The response is NDJSON: one result line per emitted item and a final summary line with created, updated, skipped, unchanged, and errors counts. If the preview has no errors, repeat with dry_run=false.

The same import path also backs seed import: a deployment can load tasks and job templates into a fresh instance at server startup (skipped when ENVOY_SKIP_SEED_IMPORT=true). See Seed import.

Promotion checklist

  1. Export only the intended tasks and templates.
  2. Store the bundle in an approved location.
  3. Confirm the target has every required connector type.
  4. Run a dry-run import and save its summary.
  5. Import definitions.
  6. Create or update target Connector records separately.
  7. Instantiate a fresh Job and bind target Connectors.
  8. Run a bounded test and verify idempotency before scheduling.