Building Pipelines
Job config: forms and field mapping
Model runtime-editable forms and field mappings with job_config, read values from task JSONata, and rely on per-run snapshots.
Job config: forms and field mapping
job_config is a runtime-editable configuration schema declared at the top level of a job template. Use it for settings an operator may change after a Job is created: migration stages, dry-run switches, notifications, batching choices, and field mappings.
It is distinct from template arguments: arguments are create-time values baked into the Job snapshot via ${VAR} substitution and fixed once the Job exists; job_config stays editable, so each Job can diverge from its template.
Lifecycle
- A job template declares the
job_configschema and defaults. - Creating a Job copies that schema into the Job snapshot, and defaults seed the Job's initial values.
- Operators edit values on the Job Details page or through dashboard input widgets.
- Starting a Job Run snapshots the current values onto
runs.job_config, so every run has an auditable copy of the configuration it used. - Tasks read the snapshot in JSONata as
job_config.<key>.
Editing a Job's configuration after a run starts does not change that run.
Schema rules
Each top-level key under job_config: becomes both the stored key and the JSONata key. Keys must match ^[A-Za-z0-9_-]+$. Only two entry types are allowed — form and field_mapping — reusing the dashboard input-widget shapes from src/dashboard-parser.ts. Do not put id or config_key inside an entry; the map key supplies both.
Define a form
A form entry stores one object keyed by field key:
job_config:
sync_options:
type: form
title: Sync options
description: Settings applied to the next run
fields:
- key: dry_run
label: Preview only
type: boolean
default: true
- key: batch_size
label: Batch size
type: number
required: true
default: 100
- key: stage
label: Current step
type: select
required: true
default: connect_accounts
options:
- { label: Connect accounts, value: connect_accounts }
- { label: Migrate tickets, value: migrate_tickets }Stored value for this entry:
{
"dry_run": true,
"batch_size": 100,
"stage": "connect_accounts"
}Each field supports these keys (FormField in src/dashboard-parser.ts):
| Key | Type | Description |
|---|---|---|
key |
string | Required. The key the value is stored under. |
label |
string | Display label. |
type |
string | text, number, boolean, select, or multi_select. |
required |
boolean | The field must be present when saving. |
default |
any | Seeds the Job's initial value. |
format |
string | email or url, validated on save. |
options |
array | Inline choices as { label, value } for select / multi_select. |
options_source |
object | Connector-backed choices (an OptionSource — must call a read-only method). |
depends_on |
string[] | Sibling field keys; options refetch and the value resets when a dependency changes. |
hide_if_expr |
string | JSONata; hides the field when it evaluates true. |
A form entry may also declare validation_expr (JSONata run against the value before persisting; a non-empty [{ field, message }] result rejects the save) and transform_expr (reshapes the value just before it is written).
Define a field mapping
A field_mapping entry stores an array of { source_field, target_field, transform? } rows. Use it when an operator maps source fields to destination fields:
job_config:
ticket_field_map:
type: field_mapping
title: Ticket field mapping
allow_transform: true
default:
- { source_field: subject, target_field: title }
source:
options_source:
connector: zendesk
method: proxy_list
args: { resource: ticket_fields }
label_expr: $.title
value_expr: $.key
search: { on_frontend: true }
target:
options_source:
connector: gorgias
method: proxy_list
args: { resource: ticket-fields }
label_expr: $.label
value_expr: $.name
search: { on_frontend: true }Entry keys (DashboardFieldMappingWidget in src/dashboard-parser.ts):
| Key | Type | Description |
|---|---|---|
source / target |
object | Required. Each an OptionSource supplying the pickable fields for that side. |
default |
array | Seeds initial mapping rows; must be an array. |
allow_transform |
boolean | Enables the per-row JSONata transform editor (operator-only). |
validation_expr |
string | JSONata run against the value before persisting. |
cardinality |
string | one_to_one (default, unique targets) or many_to_one. |
extra_columns |
array | Typed per-row fields: { key, label?, type, options?, required? } with type text, select, or boolean. |
status_expr |
string | JSONata per row → { label, tone } badge. |
row_action |
string | Action widget id rendered per row on a dashboard. |
Connector-backed option sources must use read-only methods; writes are rejected. Saves are validated structurally: source_field and target_field must be non-empty strings, target_field values must be unique (unless cardinality: many_to_one), and the value is capped at 1000 rows and 256 KB of JSON.
Where operators edit values
- The Job Details page renders the schema-backed configuration editor for every key declared in the job template snapshot — see Jobs and job templates.
- Dashboard
formandfield_mappingwidgets can read and write the same keys for guided or customer-facing flows — see Authoring dashboards.
For automation, read GET /api/jobs/{id}/config (returns { schema, config }) before writing, then save one key with PUT /api/jobs/{id}/config/{key} and body { value }. Values are validated against the Job's schema on save.
Read job_config from a task
Every step expression can reference job_config.<key>:
- name: write
type: call_method
run_if: "job_config.sync_options.dry_run = false"
config: |
{
"connector": "destination",
"method": "upsert",
"args": {
"table": "tickets",
"data": input
}
}A form value is an object, so nested reads like job_config.sync_options.batch_size work directly. A field_mapping value is an array of rows; the map_fields step expects an object keyed by source field, so convert the rows in the step config, with a static fallback so a freshly created Job is runnable before anyone edits the mapping:
- name: map
type: map_fields
config: |
{
"mapping": $exists(job_config.ticket_field_map)
? job_config.ticket_field_map{
source_field: {
"field": target_field,
"transform": transform
}
}
: {
"subject": { "field": "title" }
}
}Avoid these mistakes
- Do not store credentials or private tokens in
job_config. - Do not use
job_configfor an identity that must be fixed at Job creation — use template arguments. - Do not assume a later edit changes historical runs; each run keeps its own snapshot.
- Do not use connector write methods to populate option lists — only read-only methods are accepted.
Related
- job_config schema — the exhaustive field reference
- Template arguments — create-time values, the contrast to job_config
- Authoring dashboards — form and field_mapping widgets as editors
- Jobs and job templates — where operators edit configuration