# S3 and R2

> Source: https://docs.clonepartner.com/connectors/s3/

<!-- sources: src/connectors/s3.ts, src/connectors/csv-options.ts, references/envoy/connectors.md -->

# S3 and R2

The `s3` connector works with AWS S3, Cloudflare R2, and other S3-compatible object stores: upload, download, list, delete, presign, and read objects with optional format parsing.

## When to use it

Use `s3` when task data lives in an S3-compatible bucket — file drops to ingest, exports to publish, or attachments to move with [transfer_file](/reference/steps/transfer-file/). For Azure Storage, use [Azure Blob Storage](/connectors/azure-blob/) instead.

## Configuration

| Key | Type | Required | Default | Description |
|---|---|---|---|---|
| `access_key_id` | string | Yes | — | AWS/S3-compatible access key ID (stored as a secret) |
| `secret_access_key` | string | Yes | — | AWS/S3-compatible secret access key (stored as a secret) |
| `region` | string | No | — | Region (e.g. `us-east-1`, `weur`). Required for AWS S3. |
| `bucket` | string | Yes | — | Default bucket name |
| `endpoint` | string | No | — | Custom endpoint URL (for R2, MinIO, or other S3-compatible stores) |

For Cloudflare R2, set `endpoint` to your account's R2 endpoint (`https://<account-id>.r2.cloudflarestorage.com`) and use the R2 region name (for example `weur`). For AWS S3, leave `endpoint` unset and set `region`.

## Methods

| Method | Arguments | Returns | Behavior |
|---|---|---|---|
| `upload` | `key` (required), `body` (required), `encoding`, `content_type`, `metadata` | object | Uploads content; set `encoding: base64` when `body` is base64-encoded binary |
| `download` | `key` (required), `encoding` | object | Downloads the object as `text` (default) or `base64` |
| `download_stream` | `key` (required) | array | Streams the object body as chunks |
| `get_presigned_url` | `key` (required), `expires_in_seconds` | object | Generates a presigned URL (default expiry 518400 seconds = 6 days) |
| `head` | `key` (required) | object | Returns object metadata without downloading it |
| `delete` | `key` (required) | object | Deletes the object |
| `list` | `prefix`, `delimiter` | array | Streams all matching objects across pages; with `delimiter`, also yields grouped folder prefixes |
| `list_page` | `prefix`, `delimiter`, `continuation_token`, `max_keys` | object | Returns one page of prefixes and objects plus a continuation cursor (default 100 keys, max 1000) |
| `read` | `key` (required), `format`, `escape`, `from_record`, `delimiter`, `separator` | array or object | Reads an object and parses it: `text` (default), `json`, `yaml`, or `csv`; CSV is streamed record by record |

## Example

```yaml
name: ingest_r2_drop
connectors:
  source:
    type: s3
steps:
  - name: read_orders
    type: call_method
    config: |
      {
        "connector": "source",
        "method": "read",
        "args": { "key": "drops/orders.csv", "format": "csv" }
      }
```

The matching `connectors.yaml` entry for CLI runs (R2 shown; omit `endpoint` for AWS S3):

```yaml
source:
  type: s3
  config:
    access_key_id: ${R2_ACCESS_KEY_ID}
    secret_access_key: ${R2_SECRET_ACCESS_KEY}
    region: weur
    bucket: envoy-data
    endpoint: https://your-account-id.r2.cloudflarestorage.com
```

## Behavior notes

- `list`, `download_stream`, and CSV-format `read` return async iterators, so downstream steps consume results with backpressure instead of buffering.
- CSV parsing uses the header row as column names, skips empty lines, and coerces the literal cell values `NULL` and `null` to JSON `null`. `delimiter` accepts named values `tab` (or `\t`), `comma`, `semicolon`, and `pipe`; `separator` is an alias, and the escape character defaults to a double quote.
- `from_record` resumes CSV parsing from a specific record number for restartable imports.
- Use `list_page` for UI-style browsing with cursors; use `list` inside pipelines that process every object.
- The connector supports file uploads from the connector editor (any extension).

## Related

- [Connectors overview](/connectors/) — all compiled connector types
- [Azure Blob Storage](/connectors/azure-blob/) — the Azure equivalent
- [Reading and writing data](/building/reading-and-writing-data/) — calling connector methods from steps