Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

linux-china/ducklake-demo

Repository files navigation

DuckLake demo

DuckLake is an integrated data lake and catalog format.

DuckLake Specification

DuckLake architecture:

DuckLake Architecture

OpenTable Format

OpenTable Format

OpenTable format traits:

  • Reliable ACID transactions
  • Advanced data skipping
  • Time travel
  • Schema enforcement and evolution
  • Full CRUD operations

Features

  • PostgreSQL database for metadata storage
  • MinIO for data storage

DuckLake PostgreSQL MinIO

Data Lakes

  • lake1: a lake with DuckDB and MinIO
  • lake2: a lake with DuckDB and Local File System
  • lake2: a lake with PostgreSQL and MinIO

Please refer to justfile for commands.

Getting Started

  • Start PostgreSQL and MinIO using Docker Compose: docker-compose up -d
  • Use DuckDB CLI to create a new lake: duckdb --cmd "ATTACH 'ducklake:lakes/lake2.ducklake' AS lake2; use lake2;"
  • Create city table from demo.sql

PostgreSQL as catalog metadata store

With public schema by default:

ATTACH 'ducklake:postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456' as lake3 (DATA_PATH 's3://lake3');

Assigned schema:

ATTACH 'ducklake:postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456' as lake3 (DATA_PATH 's3://lake3', METADATA_SCHEMA 'ducklake');

Secret style:

-- secret for MinIO
create
secret ducklake_oss (
       type s3,
       key_id 'minioadmin',
       secret 'minioadmin',
       endpoint '127.0.0.1:9000',
       use_ssl false,
       url_style 'path'
);

CREATE
SECRET catalog1 (
	TYPE DUCKLAKE,
	METADATA_PATH 'postgres:dbname=ducklake host=127.0.0.1 port=55432 user=ducklake password=123456',
	DATA_PATH 's3://lake1',
    METADATA_SCHEMA 'ducklake'
);

ATTACH
'ducklake:catalog1' AS catalog1;

Attention: Please refer Add support for using secrets to manage DuckLake options and credentials

Ducklake Friends

  • Neon: Ship faster with Postgres
  • Tigris: Globally Distributed S3-Compatible Object Storage

References

About

DuckLake demo with PostgreSQL and MinIO

Topics

Resources

Stars

Watchers

Forks