Documentation by ClonePartner

Start typing to search.

Task Reference

transfer_file

View Markdown

Download one file per pipeline item from a source Connector and upload it to a destination Connector, with an optional skip when it already exists.

transfer_file

Moves one file per input item: downloads it through a source Connector method and uploads it through a destination Connector method, passing the item through unchanged.

When to use it

Use transfer_file to migrate attachments or documents where each pipeline item describes one file (its URL and target key). For very large objects, prefer a connector-native server-side copy — the file body is held in memory during the upload.

Configuration

Key Type Required Default Description
source.connector string Yes — Connector the file is downloaded from.
source.method string No proxy_custom Download method to call.
source.args object No — Extra arguments merged into the download call.
source.url string No — Download URL/path. Takes precedence over url_field.
source.url_field string No url Input field holding the download URL when source.url is not set.
source.url_query_param string No — Pass the URL's path into args.query[<param>] instead of args.path.
source.url_query_base_param string No — With url_query_param, put an absolute URL's origin into this query parameter.
destination.connector string Yes — Connector the file is uploaded to.
destination.method string No upload Upload method to call.
destination.args object No — Extra arguments merged into the upload call.
destination.key string No — Upload key. Takes precedence over key_field. A missing key fails with transfer_file requires a destination key.
destination.key_field string No key Input field holding the upload key when destination.key is not set.
destination.content_type string No — Content type for the upload. When the input item has a field with this name, that field's value is used; otherwise the literal string is sent.
destination.size_field string No — Input field holding the expected file size, used by skip_if_exists.
skip_if_exists boolean No — Call head on the destination first and skip the transfer when the object exists (and, with size_field, the size matches).

Example

name: migrate_attachments
connectors:
  source:
    type: freshdesk
  destination:
    type: s3
  staging:
    type: sqlite
steps:
  - name: read_attachments
    type: call_method
    config:
      connector: staging
      method: select
      args:
        sql: SELECT file_name, download_url, content_type, size FROM attachments
  - name: copy_file
    type: transfer_file
    config: |
      {
        "source": {
          "connector": "source",
          "method": "execute",
          "args": { "method": "GET" },
          "url_field": "download_url"
        },
        "destination": {
          "connector": "destination",
          "key_field": "file_name",
          "content_type": "content_type",
          "size_field": "size"
        },
        "skip_if_exists": true
      }

Behavior notes

  • One file per item. Memory is bounded by the size of a single file, not the dataset — the pipeline streams across files. The body is materialized in memory for the upload because storage upload methods require a concrete body.
  • URL handling. By default an absolute URL is reduced to its path and query and passed to the download method as args.path (unless args.path or args.url is already set). With url_query_param, the path goes into args.query[<param>] instead, and url_query_base_param receives the origin — for APIs that take the download path as a query parameter. When the method is proxy_custom, args.method defaults to GET.
  • Body normalization. The download result is normalized into an uploadable body: strings and byte buffers pass through, an async-iterable stream (for example from download_stream) is collected into one buffer, a storage { content, encoding } envelope is decoded (base64 content becomes bytes), and an object with a body field is unwrapped.
  • Skip check. With skip_if_exists, the destination's head method is called with the key. The transfer is skipped when the object exists and — if size_field is set and resolves to a number — the reported size matches; a size mismatch forces a re-transfer.
  • The upload call receives the destination args plus key and body, and content_type when configured. The step returns the input item unchanged.