Table of Contents

• About The Project
• Supported Flink Versions
• Installation
• DataStream API
  • Snippets
  • Examples
• Table API
  • Snippets
  • Examples
• Supported ClickHouse Types
• Configuration Options
  • Client Configuration
  • Sink Configuration
  • Sink Metrics
• Limitations
• Contributing

This is a repo of ClickHouse official Apache Flink Connector supported by the ClickHouse team. The connector supports two main Apache Flink APIs:

• DataStreamAPI
• Table API (This feature is not implemented yet and is planned for a future release)

Maven

<dependency>
    <groupId>com.clickhouse.flink</groupId>
    <artifactId>flink-connector-clickhouse-2.0.0</artifactId>
    <version>0.1.3</version>
    <classifier>all</classifier>
</dependency>

Maven

<dependency>
    <groupId>com.clickhouse.flink</groupId>
    <artifactId>flink-connector-clickhouse-1.17</artifactId>
    <version>0.1.3</version>
    <classifier>all</classifier>
</dependency>

Configure ClickHouseClient

ClickHouseClientConfig clickHouseClientConfig = new ClickHouseClientConfig(url, username, password, database, tableName);

If you are planning to insert RAW CSV data as is

Create an ElementConverter

ElementConverter<String, ClickHousePayload> convertorString = new ClickHouseConvertor<>(String.class);

Create the sink and set the format using setClickHouseFormat

ClickHouseAsyncSink<String> csvSink = new ClickHouseAsyncSink<>(
				convertorString,
				MAX_BATCH_SIZE,
				MAX_IN_FLIGHT_REQUESTS,
				MAX_BUFFERED_REQUESTS,
				MAX_BATCH_SIZE_IN_BYTES,
				MAX_TIME_IN_BUFFER_MS,
				MAX_RECORD_SIZE_IN_BYTES,
				clickHouseClientConfig
		);

csvSink.setClickHouseFormat(ClickHouseFormat.CSV);

Finally, connect your DataStream to the sink.

data.sinkTo(csvSink);

More examples and snippets can be found in our tests flink-connector-clickhouse-1.17 and flink-connector-clickhouse-2.0.0

We have created maven based example for easy start with ClickHouse Sink Different versions for Flink

• Flink 1.17+
• Flink 2.0.0+

For more detailed instructions, see the Example Guide

Table API is planned for a future release. This section will be updated once available.

Planned for a future release — this section will provide a usage snippet for configuring the Table API.

Planned for a future release — a complete end-to-end example will be added once the Table API becomes available.

• A ZoneId must also be provided when performing date operations.
• Precision and scale must also be provided when performing decimal operations.
• To use JSON type as a string, you need to enable enableJsonSupportAsString in ClickHouseClientConfig .

Our Sink is built on top of Flink’s AsyncSinkBase

Our Sink exposes additional metrics on top of Flink's existing metrics:

• Currently the sink does not support exactly-once semantics

• All projects in this repo are tested with all active LTS versions of ClickHouse.
• Support policy
• We recommend upgrading the connector continuously to not miss security fixes and new improvements
  • If you have an issue with migration - create and issue and we will respond!

Please see our contributing guide.