[Feature][Engine]Metalake support for data source information storage and management #9688

wtybxqm · 2025-08-10T11:09:13Z

Purpose of this pull request

To address the problem: sensitive information such as data source usernames and passwords is directly written into task scripts
close #9687

Does this PR introduce any user-facing change?

yes
To enable Metalake, users need to configure three critical settings in seatunnel.sh: first set 'metalake_enable=true', then specify their 'metalake_url' and 'metalake_type'. Once activated, users can simply add a 'sourceId' field in their task script's source/sink configurations and replace sensitive credentials with secure placeholders like '${password}'.

How was this patch tested?

Integration testing has passed.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

…nel-env.cmd files; modify EnvCommonOptions class to include metalake configuration option

…metalake_enable=true

…ntroduce factory for type-based MetalakeClient creation; Fetch Metalake data by sourceId and replace Config placeholders

Hisoka-X · 2025-08-11T03:39:28Z

cc @liugddx

liugddx · 2025-08-11T06:37:54Z

Please add test cases and add documentation.

liugddx · 2025-09-03T14:41:59Z

seatunnel-api/pom.xml

+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+            <version>4.5.13</version>


Is this change necessary?

we need to send http request to get metainfo from gravitino, so a http tool is necessary

liugddx · 2025-09-03T14:51:30Z

...gine-e2e/connector-seatunnel-e2e-base/src/test/resources/mysql_to_console_with_metalake.conf

+}
+
+sink {
+  Console {


Please use the Assert connector.

I used the Assert connector instead of console

liugddx · 2025-09-03T15:08:41Z

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

+        boolean metalakeEnabled =
+                Boolean.parseBoolean(System.getenv().getOrDefault("METALAKE_ENABLED", "false"));
+        if (metalakeEnabled) {
+            this.seaTunnelJobConfig =
+                    getMetalakeConfig(ConfigBuilder.of(Paths.get(jobDefineFilePath), variables));
+        } else {
+            this.seaTunnelJobConfig = ConfigBuilder.of(Paths.get(jobDefineFilePath), variables);
+        }


We need to ensure that Flink/Spark/Zeta all support this feature simultaneously.

Therefore, it needs to be adapted in both the FlinkTaskExecuteCommandand SparkTaskExecuteCommand classes.

I added this feature in spark and flink engine and the test has passed

liugddx · 2025-09-03T15:10:39Z

seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/pom.xml

        </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>mysql</artifactId>


E2e test case need to support flink/spark/zeta, so they should be placed in the seatunnel-connector-v2-e2e.

I moved the metalakeIT to seatunnel-connector-v2-e2e

…or instead of console in the conf file for test

liugddx · 2025-09-10T11:30:15Z

docs/en/seatunnel-engine/metalake.md

Place the document in the concept directory.

ok, I moved the docs to concept directory

liugddx · 2025-09-10T11:30:41Z

docs/en/seatunnel-engine/metalake.md

The same as above

liugddx · 2025-09-10T11:34:44Z

...e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/MetalakeFlinkIT.java

E2e cases will execute all engines, so there is no need to create separate test cases for Flink and Spark.

I have removed the test cases

liugddx · 2025-09-10T11:35:44Z

...e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/MetalakeSparkIT.java

The same as above

…spark and flink

Hisoka-X · 2025-09-12T13:58:18Z

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

        return new ChangeStreamTableSourceCheckpoint(coordinatorState, subtaskState);
    }
+
+    private Config getMetalakeConfig(Config jobConfigTmp) {


Could you do some refactor? I see this method three times.

I have refactor this method

Hisoka-X · 2025-09-12T13:59:51Z

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

+                            if (strValue.startsWith("${") && strValue.endsWith("}")) {
+                                String placeholder = strValue.substring(2, strValue.length() - 1);
+
+                                if (metalakeJson.has(placeholder)) {
+                                    String replaced = metalakeJson.get(placeholder).asText();
+                                    tmp =
+                                            tmp.withValue(
+                                                    subKey,
+                                                    ConfigValueFactory.fromAnyRef(replaced));
+                                }


Please reuse PlaceholderUtils.

I reused PlaceholderUtils here

Hisoka-X · 2025-09-12T14:04:42Z

...jdbc-e2e-part-7/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/MetalakeIT.java

+import static org.apache.seatunnel.e2e.common.util.ContainerUtil.PROJECT_ROOT_PATH;
+import static org.awaitility.Awaitility.given;
+
+public class MetalakeIT extends SeaTunnelContainer {


I think we don't need too complex test case. Just verify the job config has right password/username after parse by metalake. Not need to run it. Also we can mock environment by

seatunnel/seatunnel-engine/seatunnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/SeaTunnelClientTest.java

Line 758 in dac8fda

@SetEnvironmentVariable(

This annotation set env in test environment, but we need set env in test container

Hisoka-X · 2025-09-12T14:05:07Z

docs/zh/concept/metalake.md

@@ -0,0 +1,69 @@
+# METALAKE
+
+由于seatunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。


Suggested change

由于seatunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。

由于SeaTunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。

It is fixed

Hisoka-X · 2025-09-12T14:05:34Z

docs/zh/concept/metalake.md

+
+由于seatunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。
+
+因此引入了metalake，将数据源的信息存储于Apache Gravitino等metalake中，任务脚本采用`sourId`和占位符的方法来代替原本的用户名和密码等信息，运行时seatunnel-engine通过http请求从metalake获取信息，根据占位符进行替换。


Suggested change

因此引入了metalake，将数据源的信息存储于Apache Gravitino等metalake中，任务脚本采用`sourId`和占位符的方法来代替原本的用户名和密码等信息，运行时seatunnel-engine通过http请求从metalake获取信息，根据占位符进行替换。

因此引入了metalake，将数据源的信息存储于Apache Gravitino等metalake中，任务脚本采用`sourceId`和占位符的方法来代替原本的用户名和密码等信息，运行时seatunnel-engine通过http请求从metalake获取信息，根据占位符进行替换。

It is fixed

Hisoka-X · 2025-09-12T14:06:19Z

docs/zh/concept/metalake.md

+        url = "jdbc:mysql://mysql-e2e:3306/seatunnel?useSSL=false&serverTimezone=UTC&allowPublicKeyRetrieval=true"
+        driver = "${jdbc-driver}"
+        connection_check_timeout_sec = 100
+        sourceId = "test_catalog"


why use sourceId not source_name?

source_name is the id of the catalog in Apache Gravitino, but maybe source_name is not used in other metalake type

…; fix some errors in the docs

Hisoka-X

Thanks @wtybxqm for update! It's looks great!

Hisoka-X · 2025-09-14T14:44:16Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/GravitinoClient.java

+                if (entity == null) {
+                    throw new RuntimeException("No response entity");
+                }
+                ObjectMapper mapper = new ObjectMapper();


Please reuse JsonUtils

I added a readTree method in JsonUtils and reuse it here

Hisoka-X · 2025-09-14T14:45:13Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+        String metalakeType = System.getenv("METALAKE_TYPE");
+        String metalakeUrl = System.getenv("METALAKE_URL");


Maybe user can overwrite it on job config env part is better. cc @liugddx

Maybe user can overwrite it on job config env part is better. cc @liugddx

Yes, user can retrieve variables from the environment (env), and if they are not present in the environment, they can obtain them from the System. Config envConfig = config.getConfig("env");

Hisoka-X · 2025-09-14T14:47:42Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+                        if (value.valueType() == ConfigValueType.STRING) {
+                            String strValue = (String) value.unwrapped();
+                            Matcher matcher = pattern.matcher(strValue);
+                            if (matcher.find()) {
+                                String placeholder = matcher.group(1);


how about move this part into PlaceholderUtils with new method?

I have moved this into PlaceholderUtils

Hisoka-X · 2025-09-14T14:48:24Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+                if (sinkObj.containsKey("sourceId")) {
+                    ConfigObject tmp = sinkObj;
+                    String sourceId = sinkObj.toConfig().getString("sourceId");
+                    JsonNode metalakeJson = metalakeClient.getMetaInfo(sourceId);
+                    for (Map.Entry<String, ConfigValue> entry : sinkObj.entrySet()) {
+                        String subKey = entry.getKey();
+                        ConfigValue value = entry.getValue();
+
+                        if (value.valueType() == ConfigValueType.STRING) {
+                            String strValue = (String) value.unwrapped();
+                            Matcher matcher = pattern.matcher(strValue);
+                            if (matcher.find()) {
+                                String placeholder = matcher.group(1);
+
+                                if (metalakeJson.has(placeholder)) {
+                                    String replaced = metalakeJson.get(placeholder).asText();


parse source and sink has same code. Let's refactor it again!

I have refactored these same code

Hisoka-X · 2025-09-14T14:49:46Z

tools/dependencies/known-dependencies.txt

+httpcore-4.4.13.jar
 httpcore-4.4.16.jar
+httpcore-4.4.4.jar


It's weired we introduce new two version of httpcore. Can you analyze the dependencies and use only httpcore-4.4.16.jar?

It is actually wired. I just introduce httpclient-4.5.13 which is already in the other part of the project, but the Dependency License test remind me to add other two version of httpcore

We should find the core reason before merge it. cc @liugddx

httpcore has been introduced in multiple versions. Can we unify the version?

Can we unify the version?

Sure.

I have unified the version of httpcore to 4.4.16 and removed the extra version in known-dependency.txt

Hisoka-X · 2025-09-19T02:37:05Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+@Slf4j
+public class MetalakeConfigUtils {
+
+    public static Config getMetalakeConfig(Config jobConfigTmp) {


We should support transform too. For example, OpenAI key in LLM/Embedding etc.

I have supported transform here.

Hisoka-X · 2025-09-19T02:42:47Z

...-core/src/main/java/org/apache/seatunnel/engine/core/parse/MultipleTableJobConfigParser.java

+        Config envConfig =
+                ConfigBuilder.of(Paths.get(jobDefineFilePath), variables).getConfig("env");
+        boolean metalakeEnabled =
+                envConfig.hasPath("metalake_enabled")
+                        ? envConfig.getBoolean("metalake_enabled")
+                        : Boolean.parseBoolean(
+                                System.getenv().getOrDefault("METALAKE_ENABLED", "false"));
+        if (metalakeEnabled) {
+            this.seaTunnelJobConfig =
+                    MetalakeConfigUtils.getMetalakeConfig(
+                            ConfigBuilder.of(Paths.get(jobDefineFilePath), variables));
+        } else {
+            this.seaTunnelJobConfig = ConfigBuilder.of(Paths.get(jobDefineFilePath), variables);
+        }


The part can be refactor too?

I have refactored this part

Hisoka-X

Thanks @wtybxqm . We almost fininsh this.

Hisoka-X · 2025-09-20T13:23:09Z

seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/PlaceholderUtils.java

        return result.toString();
    }
+
+    public static String replacePlaceholders(String input, JsonNode json) {


Suggested change

public static String replacePlaceholders(String input, JsonNode json) {

public static String replacePlaceholders(String input, JsonNode supportedValues) {

Hisoka-X · 2025-09-20T13:23:35Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+        Config update = jobConfigTmp;
+        try {
+            ConfigList sourceList = jobConfigTmp.getList("source");
+            update =
+                    update.withValue(
+                            "source",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sourceList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }
+
+        try {
+            ConfigList sinkList = jobConfigTmp.getList("sink");
+            update =
+                    update.withValue(
+                            "sink",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sinkList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }
+
+        try {
+            ConfigList sinkList = jobConfigTmp.getList("transform");
+            update =
+                    update.withValue(
+                            "transform",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sinkList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }


I found this part can be refactor too.

…etMetalakeConfig method

Hisoka-X · 2025-09-21T02:53:44Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/metalake/MetalakeConfigUtils.java

+        Config envConfig = updateConfig.getConfig("env");
+        String metalakeType =
+                envConfig.hasPath("metalake_type")
+                        ? envConfig.getString("metalake_type")
+                        : System.getenv("METALAKE_TYPE");
+        String metalakeUrl =
+                envConfig.hasPath("metalake_url")
+                        ? envConfig.getString("metalake_url")
+                        : System.getenv("METALAKE_URL");
+        MetalakeClient metalakeClient = MetalakeClientFactory.create(metalakeType, metalakeUrl);


we should move this part outer replaceConfigList. The metalakeClient should only created once.

wtybxqm added 4 commits July 20, 2025 18:57

feat:Add metalake configuration option in seatunnel-env.sh and seatun…

ca2d091

…nel-env.cmd files; modify EnvCommonOptions class to include metalake configuration option

Feat:Fetch metalake config from environment and merge into task's env

8ceffc9

feat:Fetch metalake config by sourceId and replace placeholders when …

b5b3cc0

…metalake_enable=true

feat:Add MetalakeClient interface & GravitinoClient implementation; I…

d4ec880

…ntroduce factory for type-based MetalakeClient creation; Fetch Metalake data by sourceId and replace Config placeholders

github-actions bot added Zeta api labels Aug 10, 2025

wtybxqm added 4 commits August 21, 2025 19:30

feat:apply spotless code style

d2a84c7

feat: replace wildcard imports with explicit imports for okhttp3

f32ac80

fix: delete useless EnvOption

d545047

feat: Integration Test for Metalake

84491da

github-actions bot added the e2e label Aug 25, 2025

wtybxqm added 7 commits August 27, 2025 19:08

feat: Integration Test

768815b

feat: apply spotless code style

a6d2327

fix: fix the error of config structure

a1148f7

feat: place okhttp3 with apache httpclient

c0b304e

fix: fix the bug of matainfo replace and remove log info

d66778e

feat: apply spotless code style and remove extra code

b741052

feat: Add docs of Metalake in zh and en

50d0158

github-actions bot added the document label Aug 29, 2025

wtybxqm added 5 commits September 1, 2025 12:39

Merge branch 'dev' into support_metalake_development

bd027c7

push an empty commit to trigger the workflow

84a2b0b

fix: add license header to conf file

1acfb14

feat: download gravitino in test container

cfeb84a

feat: apply spotless codestyle and remove useless code

ec25bb0

liugddx reviewed Sep 3, 2025

View reviewed changes

wtybxqm added 2 commits September 6, 2025 20:51

feat: support metalake for spark and flink engine; use assert connect…

1b42050

…or instead of console in the conf file for test

fix: fix the error of the assert connector

d67545f

wtybxqm added 3 commits September 7, 2025 23:26

fix: move metalakeIT to seatunnel-connector-v2-e2e

4013ed8

fix: apply spotless code style

d0b1ff2

fix: remove metalakeIT in seatunnel-engine-e2e

618fa28

github-actions bot added the core SeaTunnel core module label Sep 8, 2025

wtybxqm added 2 commits September 9, 2025 16:44

feat: add metalake integration test for spark and flink engine

dc45daa

fix: download gravitino in flink container

73a4d00

liugddx reviewed Sep 10, 2025

View reviewed changes

wtybxqm added 3 commits September 10, 2025 20:11

fix: move the docs to concept directory; remove extra test cases for …

b528a98

…spark and flink

Merge branch 'dev' into support_metalake_development

b26243a

fix: add httpcore dependency in known-dependencies.txt

4f3f006

github-actions bot added the dependencies Pull requests that update a dependency file label Sep 11, 2025

Hisoka-X reviewed Sep 12, 2025

View reviewed changes

wtybxqm added 3 commits September 13, 2025 14:14

fix: reuse PlaceholderUtils and refactor the getMetalakeConfig method…

bba310a

…; fix some errors in the docs

fix: add license header

72a7538

fix: add capture group in pattern matcher

46d19f7

Hisoka-X reviewed Sep 14, 2025

View reviewed changes

wtybxqm added 2 commits September 15, 2025 20:06

fix: refactor MetalakeConfigUtils and use PlaceholderUtils and JsonUtils

8c96119

fix: unify the version of httpcore

4df31f8

github-actions bot added the connectors-v2 label Sep 17, 2025

fix: remove extra version of httpcore in kwown-dependencies.txt

0f3363a

github-actions bot removed the dependencies Pull requests that update a dependency file label Sep 17, 2025

feat: get variables from env before from system

dffb3b6

Hisoka-X reviewed Sep 19, 2025

View reviewed changes

feat: refactor getMetalakeConfig method and support transform

5bb154f

Hisoka-X reviewed Sep 20, 2025

View reviewed changes

fix: modify the arg name of replacePlaceholders method and refactor g…

dde368c

…etMetalakeConfig method

Hisoka-X reviewed Sep 21, 2025

View reviewed changes

fix: create metalakeClient only once in getMetalakeConfig method

2362402

		@@ -0,0 +1,69 @@
		# METALAKE

		由于seatunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。


		由于seatunnel在执行任务时，需要将数据库用户名与密码等隐私信息明文写在脚本中，可能会导致信息泄露；并且维护较为困难，数据源信息发生变更时可能需要手动更改。

		因此引入了metalake，将数据源的信息存储于Apache Gravitino等metalake中，任务脚本采用`sourId`和占位符的方法来代替原本的用户名和密码等信息，运行时seatunnel-engine通过http请求从metalake获取信息，根据占位符进行替换。

		String metalakeType = System.getenv("METALAKE_TYPE");
		String metalakeUrl = System.getenv("METALAKE_URL");

	public static String replacePlaceholders(String input, JsonNode json) {
	public static String replacePlaceholders(String input, JsonNode supportedValues) {

[Feature][Engine]Metalake support for data source information storage and management #9688

Are you sure you want to change the base?

[Feature][Engine]Metalake support for data source information storage and management #9688

Uh oh!

Conversation

wtybxqm commented Aug 10, 2025 • edited by Hisoka-X Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Hisoka-X commented Aug 11, 2025

Uh oh!

liugddx commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

wtybxqm commented Aug 10, 2025 •

edited by Hisoka-X

Loading