R2

Cloudflare R2 is a scalable, durable, affordable and managed object storage drop-in replacement for existing S3 workflows.

omniload supports R2 as a data source.

URI Format

The URI for connecting to R2 is structured as follows.

r2://bucket/path/to/data.parquet?endpoint_url=https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com&access_key_id=YOUR_ACCESS_KEY&secret_access_key=YOUR_SECRET_ACCESS_KEY

URI components

access_key_id:

Your access key ID.

secret_access_key:

Your secret access key.

Set up an R2 integration

Before you use Cloudflare R2, make sure you have registered an account. Then, generate an API token to serve as the Access Key for usage with existing S3-compatible SDKs or XML APIs.

Examples

To integrate omniload with R2, you need your account id, access key, and secret.

Load Parquet data from R2 into DuckDB

The following command demonstrates how to copy data from a specified OSS location into a DuckDB database.

omniload ingest \
    --source-uri   'r2://?endpoint_url=https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com&access_key_id=YOUR_ACCESS_KEY&secret_access_key=YOUR_SECRET_ACCESS_KEY' \
    --source-table 'path/to/data.parquet' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'public.example'

Running the command creates a table named example within the public schema (or equivalent grouping) in the DuckDB database file located at demo.duckdb.

Tip

Here, instead of defining the remote resource exclusively per source URI using its <path> component, the bucket name and the file glob pattern are specified using the separate --source-table option. Both addressing variants are supported equally.

Backlog

Todo

• The endpoint_url doesn’t need to be conveyed fully. It is enough to let the user supply the account_id parameter in the connection URL.

• The temporary credentials API is not covered yet.