# Working with JSONata

> Source: https://docs.clonepartner.com/building/working-with-jsonata/

<!-- sources: references/envoy/jsonata.md, src/jsonata-helpers.ts -->

# Working with JSONata

JSONata is the expression language that makes task steps dynamic — it builds step configs from the current record, decides whether steps run, and shapes output. This page covers where expressions appear, what data they can see, and the Envoy-specific helper functions.

For the language itself (operators, path syntax, standard functions), see the [JSONata documentation](https://jsonata.org/). For the exhaustive function list, see the [JSONata reference](/reference/jsonata/).

## Prerequisites

- A task file to work in — see [Task anatomy and authoring workflow](/building/task-anatomy/)
- Basic JSONata path syntax from the [JSONata documentation](https://jsonata.org/)

## Where JSONata appears in a task

Any step key that accepts an expression string is evaluated as JSONata:

- **`config`** — when a step's `config` is a string (YAML `|` block or quoted string) instead of a static object, it is evaluated per item and the result is passed to the step. `transform` and `filter` configs are always JSONata strings.
- **`run_if` / `skip_if`** — boolean expressions that decide per item whether the step executes.
- **`batch` / `spool`** — can be expressions instead of a literal number or boolean.

A static object `config` is passed to the step verbatim and never evaluated. If you write `arguments.account_id` inside a static object, the step receives that literal string — the validator flags this with an error telling you to convert the config to a JSONata string (`config: |`).

## Context roots

Every expression is evaluated with these roots in scope:

| Root | Description |
|---|---|
| `input` | Current input item (output of the previous step) |
| `arguments` | Runtime arguments — from job template task node `arguments`, CLI `--arguments`, or server API body |
| `steps` | Previous step outputs: `steps.{stepName}.output` |
| `variables` | Task variables: `variables.{varName}` |
| `job_config` | Per-job runtime configuration from the job template `job_config:` schema or compatible dashboard input widgets: `job_config.{key}` |

Short valid examples of each:

```jsonata
input.email
```

```jsonata
arguments.account_id
```

```jsonata
steps.find_user.output.name
```

```jsonata
variables.batch_size
```

```jsonata
job_config.sync_enabled
```

`input` always holds the output of the immediately preceding step. To reach further back, use `steps.{name}.output`:

```yaml
- name: combine
  type: transform
  config: |
    {
      "order_id": input.order_id,
      "user_name": steps.find_user.output.name
    }
```

## Helper functions

Envoy registers these custom functions on top of standard JSONata and the `@truto/truto-jsonata` extensions:

| Function | Purpose |
|---|---|
| `$callMethod({connector, method, ...args})` | Call a connector method from within an expression. Bounded lookups only — it buffers the entire result into memory. |
| `$oid(value)` | Coerce a 24-character hex string or ObjectId-like value to a BSON `ObjectId` for MongoDB filters and writes. |
| `$isNullish(value)` | `true` for both `null` and `undefined`. |
| `$hash(obj, algorithm?)` | Stable hash of a value (`sha256` default, `md5` optional). Keys are sorted before hashing; returns `null` for nullish input. |
| `$fromJson(value)` | Parse a JSON string into an object/value. Non-strings pass through; unparseable strings return `null`. |
| `$formData(descriptor)` | Build a multipart `FormData` from `{ fields?, files? }` for file-upload calls. |
| `$setBlobName(blob, filename)` | Set the filename on a `Blob`. |
| `$fixEmail(str, returnNullOnInvalid?)` | Normalize a malformed email; pass `true` to get `null` for invalid addresses. |
| `$stripInvisibleChars(str)` | Remove zero-width and invisible characters from a string. |
| `$parsePhone(str, region?)` | Parse a phone number into `{ country, nationalNumber, number, isValid }`; `null` if unparseable. Default region `US`. |
| `$isValidPhone(str, region?)` | `true` if the string is a valid phone number. |
| `$formatPhoneE164(str, region?)` | Format a valid phone number as E.164; `null` if invalid. |
| `$mostSimilar(str, candidates, threshold?)` | Best Jaro–Winkler match above the threshold (default `0.8`); `null` if none. |
| `$adfToHtml(value)` | Convert Atlassian Document Format (an ADF object or JSON string) to HTML. Non-ADF strings pass through unchanged. |
| `$mask(value, type?, options?)` | Redact a PII value with a deterministic token (see the masking notes in the [JSONata reference](/reference/jsonata/)). |
| `$maskEmail(value, options?)` / `$maskPhone(value)` / `$maskName(value)` / `$maskText(value)` | Typed masking shortcuts. |
| `$maskingEnabled()` | `true` when masking is enabled for this task. |
| `$destinationId(resource, sourceId)` | Legacy shorthand for a destination-id lookup via the task's `lookup_connector` slot. |

### `$callMethod` — per-item bounded lookups

Use `$callMethod` inside a `transform` when you need a connector call per item and want to keep the original item's fields:

```yaml
- name: count_per_collection
  type: transform
  config: |
    {
      "collection": input.name,
      "count": $callMethod({
        "connector": "mongo",
        "method": "count",
        "collection": input.name
      }).count
    }
```

Unlike the `call_method` step, `$callMethod` takes flat args — method arguments sit directly on the object, not under `args`. Reserve it for bounded single-record lookups (`find_one`, `count`, grouped `aggregate`): for collection methods, it drains every page into memory, so use a `call_method` step instead (see [Reading and writing data](/building/reading-and-writing-data/)).

### `$oid` — MongoDB ObjectId handling

When one MongoDB call returns a document and a later step reuses its `_id`, pass it through `$oid()` so native and serialized ObjectId values are handled consistently:

```jsonata
$oid(steps.load_config.output._id)
```

### `$isNullish` — missing-or-null checks

In JSONata, `undefined = null` evaluates to `false`, so absent keys slip through `= null` checks. `$isNullish(x)` is `true` for both missing and explicitly-null values:

```yaml
- name: has_email
  type: filter
  config: "$not($isNullish(input.email))"
```

## Common pitfalls

These expressions compile cleanly and fail only at evaluation time, so `envoy validate` does not catch them. Test every expression against a real record.

- **`not(x)` throws `T1005`.** JSONata has no bare `not`. Use `$not(x)`.
- **`$contains(arr, x)` throws `T0410`** for arrays — `$contains` is string-only. Use `x in arr` or `$count(arr[$ = x]) > 0`.
- **`undefined = null` evaluates to `false`**, so absent keys slip through null checks. Use `$isNullish(x)`.
- **A single-element result collapses from array to object.** When an API needs an array, wrap it: `$append([], $maybeOne)`.
- **Multi-statement expressions need a parenthesized sequence:** `( $a := 1; $b := 2; { ... } )`. Statements directly inside `{ }` are a parse error.
- **Inside a predicate `[ ... ]`, bare identifiers resolve against the current context item**, not your outer variables. Bind values you need to `$`-prefixed variables first.
- **The `%` parent operator does not work across nested `$map` closures.** Bind what you need first (`$tid := ticket_id;`), then use `$tid` inside the inner function.
- **Referencing a later step** (`steps.{name}` for a step defined below the current one) produces a validation warning — the step has no output yet when the expression runs.

## Verify it worked

Run the task with `--debug` and inspect each step's evaluated config and output on stderr:

```bash
envoy my-task --config connectors.yaml --debug
```

Every expression should produce the shape you expect for a real record — compile-clean is not the same as correct.

:::callout{type="warning"}
`envoy validate` only compiles expressions. Evaluation-time errors such as `D3030`, `T0410`, and `T1005` appear only when a real item flows through the step.
:::

## Related

- [JSONata reference](/reference/jsonata/) — the exhaustive function and masking reference
- [Transforming and filtering](/building/transforming-and-filtering/) — the steps where most expressions live
- [Task anatomy and authoring workflow](/building/task-anatomy/) — where configs and conditions fit in a task