[Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode #9743

yzeng1618 · 2025-08-21T07:58:38Z

Purpose of this pull request

This pull request implements a comprehensive Hive Sink connector for Apache SeaTunnel with advanced auto-create table capabilities. The connector enables seamless data writing to Apache Hive tables with support for automatic schema management, partitioning, multiple storage formats, and cloud storage integration.
Key features include:
Auto-create table functionality with configurable schema save modes
Advanced partitioning support through partition_fields configuration
Multiple storage formats (PARQUET, ORC, TEXTFILE)

Does this PR introduce any user-facing change?

Yes, this PR introduces a new Hive Sink connector with the following user-facing changes:
New Configuration Options:
schema_save_mode: Controls table creation behavior (CREATE_SCHEMA_WHEN_NOT_EXIST, RECREATE_SCHEMA, ERROR_WHEN_SCHEMA_NOT_EXIST, IGNORE)
table_format: Specifies storage format (PARQUET, ORC, TEXTFILE)
partition_fields: List of partition columns for table partitioning
save_mode_create_template: Customizable table creation template

Example Configuration:
sink {
Hive {
metastore_uri = "thrift://metastore:9083"
table_name = "warehouse.user_data"
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
table_format = "PARQUET"
partition_fields = ["dt"]
}
}

How was this patch tested?

Unit Tests:
HiveSinkOptionsTest: Tests configuration option parsing and validation
HiveSaveModeHandlerTest: Tests table creation and schema management logic
HiveTypeConvertorTest: Tests data type conversion between source and Hive types
Local Tests

Check list

[*] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[*] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[*] If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

Hisoka-X · 2025-08-22T04:04:17Z

cc @liunaijie

Hisoka-X

Please add test case in HiveIT and update the docs.

liunaijie · 2025-08-23T05:24:53Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+    }
+
+    private void createTableUsingTemplate() throws TException {
+        processCreateTemplate();


I see a SQL template here, but the result not used.
Perhaps you meant to parse the sql result to a Table?

Hisoka-X · 2025-08-26T02:04:30Z

docs/en/connector-v2/sink/Hive.md

+
+
+
+### table_format [string]


Let's introduce save_mode_create_template instead of table_format and partition_fields
Please refer https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Doris.md#save_mode_create_template

Hisoka-X · 2025-08-31T03:04:48Z

seatunnel-connectors-v2/connector-hive/pom.xml

            <groupId>org.apache.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>${hive.exec.version}</version>
-            <scope>provided</scope>


Please keep hive-exec as provided. Because user could be different version

Hisoka-X · 2025-08-31T03:05:20Z

seatunnel-connectors-v2/connector-hive/pom.xml

                <exclusion>
                    <groupId>org.apache.parquet</groupId>
-                    <artifactId>parquet-hadoop-bundle</artifactId>
+                    <artifactId>*</artifactId>
                </exclusion>
+                <exclusion>
+                    <groupId>org.apache.avro</groupId>
+                    <artifactId>avro</artifactId>
+                </exclusion>
+                <!-- Exclude unnecessary dependencies -->


where is other version dependency?

Hisoka-X · 2025-08-31T03:05:37Z

seatunnel-connectors-v2/connector-hive/pom.xml

+            </exclusions>
+        </dependency>
+        <!-- Hive Common dependency - contains HiveConf class -->
+        <dependency>


Hisoka-X · 2025-08-31T03:05:43Z

seatunnel-connectors-v2/connector-hive/pom.xml

+        </dependency>
+
+        <!-- Hive MetaStore dependency - contains HiveMetaStoreClient, AlreadyExistsException and other metastore API classes -->
+        <dependency>


Hisoka-X · 2025-08-31T03:06:43Z

docs/en/connector-v2/sink/Hive.md

+
+
+
+### save_mode_create_template [string]


What's default value of save_mode_create_template?

Hisoka-X · 2025-08-31T03:06:57Z

docs/en/connector-v2/sink/Hive.md


 Flag to decide whether to use overwrite mode when inserting data into Hive. If set to true, for non-partitioned tables, the existing data in the table will be deleted before inserting new data. For partitioned tables, the data in the relevant partition will be deleted before inserting new data.

+### schema_save_mode [enum]


default value?

Hisoka-X · 2025-08-31T03:08:32Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java

        }
    }

+    public void createDatabaseIfNotExists(String db) throws TException {


How about let HiveMetaStoreProxy implements Catalog interface? So that we do need overwrite most method of HiveSaveModeHandler.

Hisoka-X · 2025-09-01T14:14:13Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

    private final List<String> partitionFields;

    private HiveMetaStoreProxy hiveMetaStoreProxy;
+    private Catalog optionalCatalog; // 可选的Catalog支持


Please remove chinese.

Hisoka-X · 2025-09-01T14:14:31Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveMetaStoreProxy.java


 @Slf4j
-public class HiveMetaStoreProxy implements Closeable, Serializable {
+public class HiveMetaStoreProxy implements Catalog, Closeable, Serializable {


Suggested change

public class HiveMetaStoreProxy implements Catalog, Closeable, Serializable {

public class HiveMetaStoreCatalog implements Catalog, Closeable, Serializable {

…eCreationORCFormat

liunaijie · 2025-09-10T09:43:07Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+        table.setTableName(tableName);
+        table.setOwner(System.getProperty("user.name", "seatunnel"));
+        table.setCreateTime((int) (System.currentTimeMillis() / 1000));
+        table.setTableType("MANAGED_TABLE");


For customTemplate, show we still need set the type to MANAGED_TABLE?

liunaijie · 2025-09-10T09:43:56Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+        sd.setCols(cols);
+
+        // Set table location
+        String tableLocation = HiveTableTemplateUtils.getDefaultTableLocation(dbName, tableName);


location should also extract from template

…nt of TableType, Location, and TblProperties

Hisoka-X · 2025-09-14T13:38:31Z

...-v2-e2e/connector-hive-e2e/src/test/java/org/apache/seatunnel/e2e/connector/hive/HiveIT.java

+        executeJob(
+                container,
+                "/auto_table_creation/fake_to_hive_create_when_not_exist.conf",
+                "/auto_table_creation/hive_auto_create_to_assert.conf");


why use new seatunnel job to verify table? Maybe we can just verify new table by hiveclient.

Hisoka-X · 2025-09-14T13:42:00Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+        // For Hive, data save mode is handled by the existing OVERWRITE parameter
+        // No additional data handling is needed here
+        log.info(
+                "Data save mode handling is managed by existing OVERWRITE parameter for table {}.{}",
+                dbName,
+                tableName);


Let's merge overwrite logic with datasavemode. We can set overwrite=true or datasavemode=TRUNCATE to do same thing.

Hisoka-X · 2025-09-14T13:45:32Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+        // Initialize partition fields from template if available
+        this.partitionFields = extractPartitionFieldsFromConfig();


I found this field only used by test case? We should use another way to verify partition fields in test case, not in runtime code.

Hisoka-X · 2025-09-14T13:46:17Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+        this.partitionFields = extractPartitionFieldsFromConfig();
+    }
+
+    public HiveSaveModeHandler(


Hisoka-X · 2025-09-14T13:47:38Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+    private final List<String> partitionFields;
+
+    private HiveMetaStoreCatalog hiveCatalog;
+    private Catalog optionalCatalog;


why not direct use hiveCatalog but optionalCatalog?

Hisoka-X · 2025-09-14T13:49:31Z

...e/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSaveModeHandler.java

+import java.util.List;
+
+@Slf4j
+public class HiveSaveModeHandler implements SaveModeHandler, AutoCloseable {


Why not implement DefaultSaveModeHandler? Many code is duplicated.

Hisoka-X · 2025-09-14T13:51:55Z

...-hive/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/sink/HiveSinkOptions.java

+    public static final Option<String> SAVE_MODE_CREATE_TEMPLATE =
+            Options.key("save_mode_create_template")
+                    .stringType()
+                    .noDefaultValue()


default value?

Carl-Zhou-CN · 2025-09-16T02:51:37Z

...ctor-hive-e2e/src/test/resources/auto_table_creation/fake_to_hive_create_when_not_exist.conf

+  FakeSource {
+    schema = {
+      fields {
+        pk_id = bigint


Could you help create a detection for all field types？

Carl-Zhou-CN · 2025-09-16T02:54:10Z

...ve/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveTypeConvertor.java

+            case TIMESTAMP:
+                return "timestamp";
+            case ROW:
+                return "struct";


Is there an error here when struct needs to map fields？

Carl-Zhou-CN · 2025-09-16T02:56:44Z

...ve/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveTypeConvertor.java

+            case ROW:
+                return "struct";
+            case ARRAY:
+                return "array";


Carl-Zhou-CN · 2025-09-16T02:57:02Z

...ve/src/main/java/org/apache/seatunnel/connectors/seatunnel/hive/utils/HiveTypeConvertor.java

+            case ARRAY:
+                return "array";
+            case MAP:
+                return "map";


Carl-Zhou-CN · 2025-09-20T13:42:37Z

@yzeng1618 hi,I have a small suggestion. You can use this identifier or "done" to mark that the problem has been solved

zengyi added 8 commits August 17, 2025 21:57

[feature] add hive sink

9aaec67

[feature] add hive sink

8866db9

[feature] fix details

9e5e078

[feature] ADD TABLE_FORMAT AND TIMEZONE

b848653

[feature] add partition_fields

b6e174b

[feature] fix details

5eb4aed

[feature] add e2e test

baaca77

[improve][start] fix detail

63dd6b6

github-actions bot added connectors-v2 e2e hive labels Aug 21, 2025

[feature] remove log and e2e

36280f2

github-actions bot removed the e2e label Aug 21, 2025

Hisoka-X reviewed Aug 22, 2025

View reviewed changes

liunaijie reviewed Aug 23, 2025

View reviewed changes

[feature] Add e2e and Simplify the code

2148423

github-actions bot added document e2e labels Aug 25, 2025

Hisoka-X reviewed Aug 26, 2025

View reviewed changes

[feature] add hiveautocreatetable by save_mode_create_template

7f53040

Hisoka-X reviewed Aug 31, 2025

View reviewed changes

zengyi added 4 commits August 31, 2025 11:57

Merge branch 'dev' into 9741-HiveAutoCreateTable

5766d1b

[feature] fix ci build

d5ad180

[feature] fix new

eb70617

[feature] add catalog option

cf9d827

Hisoka-X reviewed Sep 1, 2025

View reviewed changes

zengyi added 3 commits September 2, 2025 19:47

[feature] fix catalog option

5c39865

[feature] fix details

5a0f6c5

[feature] fix e2e details

abbfacb

zengyi added 10 commits September 4, 2025 17:39

[feature] fix

b090efb

[feature] fix 0904

d15cfa3

[feature] fix 0905

67b6e5a

[feature] fix 0905

8f6e920

[feature] fix 0905

3229234

[feature] fix 0905

0281c17

[feature] fix 0905

a468197

[feature] fix 0907

02eb6b4

[feature] remove testAutoTableCreationRecreateSchema and testAutoTabl…

0bd5a19

…eCreationORCFormat

[feature] fix details 0908

6dd1b37

yzeng1618 requested review from liunaijie and Hisoka-X September 8, 2025 05:43

[feature] rename HiveMetaStoreCatalog

4b98249

liunaijie reviewed Sep 10, 2025

View reviewed changes

zengyi added 3 commits September 11, 2025 11:39

[feature] Optimized the customTemplate scenario, as well as the conte…

3313dde

…nt of TableType, Location, and TblProperties

[feature] mvn spotless:apply

a466e5d

Merge branch 'dev' into 9741-HiveAutoCreateTable

6995fec

yzeng1618 requested a review from liunaijie September 11, 2025 03:45

[feature] Remove unnecessary comments and improve E2E tests

34800bb

Hisoka-X reviewed Sep 14, 2025

View reviewed changes

zengyi added 2 commits September 16, 2025 16:30

[feature] fix 0916

89563f8

[feature] fix 0916-1 add DataSaveMode

2873558

Carl-Zhou-CN reviewed Sep 16, 2025

View reviewed changes

yzeng1618 requested a review from Hisoka-X September 16, 2025 11:44

zengyi added 2 commits September 16, 2025 20:32

[feature] modify the code

ac0c8cc

[feature] fix detail

027da0d

Hisoka-X changed the title ~~[Feature][Connector-Hive] Hive Sink Connector with Auto-Create Table Support~~ [Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode Sep 17, 2025

yzeng1618 requested review from Carl-Zhou-CN September 17, 2025 03:59


		Flag to decide whether to use overwrite mode when inserting data into Hive. If set to true, for non-partitioned tables, the existing data in the table will be deleted before inserting new data. For partitioned tables, the data in the relevant partition will be deleted before inserting new data.

		### schema_save_mode [enum]

	public class HiveMetaStoreProxy implements Catalog, Closeable, Serializable {
	public class HiveMetaStoreCatalog implements Catalog, Closeable, Serializable {

		// Initialize partition fields from template if available
		this.partitionFields = extractPartitionFieldsFromConfig();

[Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode #9743

Are you sure you want to change the base?

[Feature][Connector-V2] Hive sink support SchemaSaveMode and DataSaveMode #9743

Uh oh!

Conversation

yzeng1618 commented Aug 21, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Hisoka-X commented Aug 22, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Carl-Zhou-CN commented Sep 20, 2025

Uh oh!

Uh oh!