HDFS¶

Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system written in Java for the Hadoop framework.

omniload supports HDFS as a data source.

URI Format¶

The URI for connecting to HDFS is structured as follows.

hdfs://example.com:8020/path/to/data.parquet?user=test

URI components¶

host:: HDFS host to connect to. Set to “default” for fs.defaultFS from core-site.xml.
port:: HDFS port to connect to. Set to 0 for default or logical (HA) nodes. Type: int. Default: 8020.
user:: Username when connecting to HDFS; None implies login user.
replication:: Number of copies each block will have. Type: int. Default: 3.
buffer_size:: The size of the temporary read and write buffer in bytes. 0 means no buffering will happen. Type: int. Default: 0.
default_block_size:: The block size in bytes. None means the default configuration for HDFS, a typical block size is 128 MB. Type: int. Default: None.
kerb_ticket:: The path to the Kerberos ticket cache.
extra_conf:: [1] Optional extra key/value pairs for configuration; will override any hdfs-site.xml properties. Type: dict.

Examples¶

To integrate omniload with OSS, you need the server’s hostname (endpoint) and valid credentials.

Load Parquet data from HDFS into DuckDB¶

The following command demonstrates how to copy data from a specified OSS location into a DuckDB database.

omniload ingest \
    --source-uri   'hdfs://example.com:8020/?user=test' \
    --source-table 'path/to/data.parquet' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'public.example'

Running the command creates a table named example within the public schema (or equivalent grouping) in the DuckDB database file located at demo.duckdb.

Tip

Here, instead of defining the remote resource exclusively per source URI using its <path> component, the resource location is specified using the separate --source-table option. Both addressing variants are supported equally.

Backlog¶

Todo

PyArrow comes with bindings to the Hadoop File System, however you must still configure it properly. In this spirit, because not packaging efforts were poured into this, the HDFS connector can not be expected to work out of the box, for example when using the omniload OCI image.