Documentation by ClonePartner

Start typing to search.

Connector Reference

S3 and R2

View Markdown

Configure the s3 connector for AWS S3, Cloudflare R2, or any S3-compatible store, and use its upload, download, list, and read methods.

S3 and R2

The s3 connector works with AWS S3, Cloudflare R2, and other S3-compatible object stores: upload, download, list, delete, presign, and read objects with optional format parsing.

When to use it

Use s3 when task data lives in an S3-compatible bucket — file drops to ingest, exports to publish, or attachments to move with transfer_file. For Azure Storage, use Azure Blob Storage instead.

Configuration

Key Type Required Default Description
access_key_id string Yes — AWS/S3-compatible access key ID (stored as a secret)
secret_access_key string Yes — AWS/S3-compatible secret access key (stored as a secret)
region string No — Region (e.g. us-east-1, weur). Required for AWS S3.
bucket string Yes — Default bucket name
endpoint string No — Custom endpoint URL (for R2, MinIO, or other S3-compatible stores)

For Cloudflare R2, set endpoint to your account's R2 endpoint (https://<account-id>.r2.cloudflarestorage.com) and use the R2 region name (for example weur). For AWS S3, leave endpoint unset and set region.

Methods

Method Arguments Returns Behavior
upload key (required), body (required), encoding, content_type, metadata object Uploads content; set encoding: base64 when body is base64-encoded binary
download key (required), encoding object Downloads the object as text (default) or base64
download_stream key (required) array Streams the object body as chunks
get_presigned_url key (required), expires_in_seconds object Generates a presigned URL (default expiry 518400 seconds = 6 days)
head key (required) object Returns object metadata without downloading it
delete key (required) object Deletes the object
list prefix, delimiter array Streams all matching objects across pages; with delimiter, also yields grouped folder prefixes
list_page prefix, delimiter, continuation_token, max_keys object Returns one page of prefixes and objects plus a continuation cursor (default 100 keys, max 1000)
read key (required), format, escape, from_record, delimiter, separator array or object Reads an object and parses it: text (default), json, yaml, or csv; CSV is streamed record by record

Example

name: ingest_r2_drop
connectors:
  source:
    type: s3
steps:
  - name: read_orders
    type: call_method
    config: |
      {
        "connector": "source",
        "method": "read",
        "args": { "key": "drops/orders.csv", "format": "csv" }
      }

The matching connectors.yaml entry for CLI runs (R2 shown; omit endpoint for AWS S3):

source:
  type: s3
  config:
    access_key_id: ${R2_ACCESS_KEY_ID}
    secret_access_key: ${R2_SECRET_ACCESS_KEY}
    region: weur
    bucket: envoy-data
    endpoint: https://your-account-id.r2.cloudflarestorage.com

Behavior notes

  • list, download_stream, and CSV-format read return async iterators, so downstream steps consume results with backpressure instead of buffering.
  • CSV parsing uses the header row as column names, skips empty lines, and coerces the literal cell values NULL and null to JSON null. delimiter accepts named values tab (or \t), comma, semicolon, and pipe; separator is an alias, and the escape character defaults to a double quote.
  • from_record resumes CSV parsing from a specific record number for restartable imports.
  • Use list_page for UI-style browsing with cursors; use list inside pipelines that process every object.
  • The connector supports file uploads from the connector editor (any extension).