feature: Raft cluster mode supports address translation #7069

PeppaO · 2024-12-23T02:44:15Z

I have registered the PR changes.

Ⅰ. Describe what this PR did

现有raft sdk如果在k8s环境下，客户端集群外访问，无法通过域名方式链接raft node，造成无法通信。
支持外部访问的方案：
server: 在元数据Node中添加一个metadata.external数组，在部署时，传入SEATA_REGISTRY_METADATA_EXTERNAL变量，创建节点时解析设置，同步至主节点。
client:
通过api获取集群元数据，然后解析并选择通信的IP及地址。如果客户端配置了preferredNetworks，那么将从元数据Node.metadata.external中正则匹配选择，否则就按照原来方案，使用Node.control及Node.transaction。
客户端配置：

registry {
  # file 、nacos 、eureka、redis、zk、consul、etcd3、sofa
  type = "raft"
  preferredNetworks = "192.168.*",
  raft {
    metadata-max-age-ms = 30000
    serverAddr = "10.10.92.71:30071,10.10.92.71:30072"

  }
 }

部署文件：

---
apiVersion: v1
kind: Service
metadata:
  name: s-0
  namespace: default
  labels:
    k8s-app: s-0
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30071
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30091
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-0
---
apiVersion: v1
kind: Service
metadata:
  name: s-1
  namespace: default
  labels:
    k8s-app: s-1
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30072
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30092
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-1
---
apiVersion: v1
kind: Service
metadata:
  name: s-2
  namespace: default
  annotations:
    my-custom-annotation: "value1"
    another-annotation: "value2"
  labels:
    k8s-app: s-2
    my-label: "30073"
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30073
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30093
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-2
---
apiVersion: v1
kind: Service
metadata:
  name: svc-l
  namespace: default
  labels:
    k8s-app: svc-l
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: false
  ports:
    - name: http
      port: 7091
      targetPort: 7091
    - name: tcp
      port: 8091
      targetPort: 8091
  selector:
    k8s-app: v

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: v
  namespace: default
  labels:
    k8s-app: v
spec:
  replicas: 3
  serviceName: svc-l
  selector:
    matchLabels:
      k8s-app: v
  template:
    metadata:
      labels:
        k8s-app: v
    spec:
      initContainers:
        - name: cluster-init
          image: harbor.local:80/seata/seata-server:4.0
          imagePullPolicy: Always
          command: [
            "sh", "-c", "index=$(hostname | awk -F '-' '{print $NF}'); \
            case $index in \
              0) echo 30071:30091 > /etc/nodeport/port ;; \
              1) echo 30072:30092 > /etc/nodeport/port ;; \
              2) echo 30073:30093 > /etc/nodeport/port ;; \
              *) echo 'Invalid index' > /etc/nodeport/port ;; \
            esac"
          ]
          volumeMounts:
            - name: nodeport-volume
              mountPath: /etc/nodeport
      containers:
        - name: server
          image: harbor.local:80/seata/seata-server:4.0
          imagePullPolicy: Always
          command: ["bash", "-c", "
            export SEATA_REGISTRY_METADATA_EXTERNAL=${NODE_IP}:$(cat /etc/nodeport/port) && \
            echo 'Starting server with SEATA_REGISTRY_METADATA_EXTERNAL='$SEATA_REGISTRY_METADATA_EXTERNAL && \
            /seata-server/bin/seata-server.sh && \
            echo 'Server started successfully!' && \
            tail -f /dev/null"
          ]
          env:
            - name: SEATA_HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SEATA_IP
              value: "$(SEATA_HOST_NAME).svc-l.default.svc.cluster.local"
            - name: server.raftPort
              value: "9091"
            - name: server.raft.serverAddr
              value: "v-0.svc-l.default.svc.cluster.local:9091,v-1.svc-l.default.svc.cluster.local:9091,v-2.svc-l.default.svc.cluster.local:9091"
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
          ports:
            - name: http
              containerPort: 8091
              protocol: TCP
          volumeMounts:
            - name: nodeport-volume
              mountPath: /etc/nodeport
      volumes:
        - name: nodeport-volume
          emptyDir: {}

这里之所以取名这么简单是测试的时候，发现service过长，会出现 #7082 的错误

元数据：

{
	"nodes": [{
		"control": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "LEADER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30071,
				"transactionPort": 30091
			}]
		}
	}, {
		"control": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "FOLLOWER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30073,
				"transactionPort": 30093
			}]
		}
	}, {
		"control": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "FOLLOWER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30072,
				"transactionPort": 30092
			}]
		}
	}],
	"storeMode": "raft",
	"term": 1
}

测试：

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

funky-eyes · 2024-12-23T02:55:55Z

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

+
+    private static InetSocketAddress selectEndpoint(String type, Node node) {
+        if (StringUtils.isBlank(PREFERRED_NETWORKS)) {
+            // 采取默认的方式，直接使用node.control  node.transaction


Suggested change

// 采取默认的方式，直接使用node.control node.transaction

// Use the default method, directly using node.control and node.transaction

funky-eyes · 2024-12-23T02:59:41Z

server/src/main/java/org/apache/seata/server/cluster/raft/RaftStateMachine.java

                        ((Environment)ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT))
                            .getProperty("server.port", String.valueOf(7091))),
-                    group, Collections.emptyMap());
+                    group, new HashMap<>());


Why change to new HashMap < > ()

因为Collections.emptyMap()返回的集合使用了final修饰，导致我进行put操作时，报错，unsupoort...

不要直接put,而是构建一个新的map替换它,copyonwrite

不要直接put,而是构建一个新的map替换它,copyonwrite

done

funky-eyes · 2024-12-23T03:01:07Z

server/src/main/java/org/apache/seata/server/cluster/raft/RaftStateMachine.java

                        ((Environment)ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT))
                            .getProperty("server.port", String.valueOf(7091))),
-                    group, Collections.emptyMap());
+                    group, new HashMap<>());


funky-eyes · 2024-12-23T03:04:24Z

server/src/main/java/org/apache/seata/server/cluster/raft/sync/msg/dto/RaftClusterMetadata.java

        node.setGroup(group);
        node.setVersion(Version.getCurrent());
        node.setInternal(node.createEndpoint(host, internalPort, "raft"));
+        String serverRegistryMetadataExternalValue = System.getProperty("SERVER_REGISTRY_METADATA_EXTERNAL_VALUE");


不应该这样读取，如果有用户直接使用properties配置文件去配置怎么办？应该通过下面这样的方式读取
It should not be read like this. What if a user directly uses the properties configuration file to configure it? It should be read in the following way

ConfigurableEnvironment environment=ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT); environment.resolvePlaceholders("${seata.registry.metadata.external:${registry.metadata.external:}}")

done

不要直接put,而是构建一个新的map替换它,copyonwrite

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

codecov · 2024-12-28T10:47:08Z

Codecov Report

Attention: Patch coverage is 25.83333% with 89 lines in your changes missing coverage. Please review.

Project coverage is 52.59%. Comparing base (c0a2e92) to head (ff5ff02).
Report is 1 commits behind head on 2.x.

Files with missing lines	Patch %	Lines
...in/java/org/apache/seata/common/metadata/Node.java	0.00%	41 Missing ⚠️
...scovery/registry/raft/RaftRegistryServiceImpl.java	47.05%	20 Missing and 7 partials ⚠️
...seata/common/exception/ParseEndpointException.java	0.00%	10 Missing ⚠️
...cluster/raft/sync/msg/dto/RaftClusterMetadata.java	37.50%	3 Missing and 2 partials ⚠️
...ain/java/org/apache/seata/common/util/NetUtil.java	40.00%	3 Missing ⚠️
...roperties/registry/RegistryMetadataProperties.java	25.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #7069      +/-   ##
============================================
- Coverage     52.65%   52.59%   -0.07%     
- Complexity     6666     6680      +14     
============================================
  Files          1129     1131       +2     
  Lines         40168    40277     +109     
  Branches       4708     4721      +13     
============================================
+ Hits          21152    21185      +33     
- Misses        16999    17067      +68     
- Partials       2017     2025       +8

Files with missing lines	Coverage Δ
...ava/org/apache/seata/common/ConfigurationKeys.java	`0.00% <ø> (ø)`
...toconfigure/SeataCoreEnvironmentPostProcessor.java	`100.00% <100.00%> (ø)`
...ta/spring/boot/autoconfigure/StarterConstants.java	`100.00% <ø> (ø)`
...ain/java/org/apache/seata/common/util/NetUtil.java	`52.98% <40.00%> (-0.51%)`	⬇️
...roperties/registry/RegistryMetadataProperties.java	`25.00% <25.00%> (ø)`
...cluster/raft/sync/msg/dto/RaftClusterMetadata.java	`77.14% <37.50%> (-12.15%)`	⬇️
...seata/common/exception/ParseEndpointException.java	`0.00% <0.00%> (ø)`
...scovery/registry/raft/RaftRegistryServiceImpl.java	`23.70% <47.05%> (+6.23%)`	⬆️
...in/java/org/apache/seata/common/metadata/Node.java	`38.09% <0.00%> (-24.41%)`	⬇️

... and 3 files with indirect coverage changes

…aO/seata into support_external_access_2.x

funky-eyes · 2024-12-30T01:36:34Z

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

+            throw new ParseEndpointException("Node metadata is empty.");
+        }
+
+        Object external = metadata.get("external");


这里要判空，高版本客户端对接低版本server是不会有这个字段的

低版本不支持这个功能，不会走到这个逻辑，会直接走这里

funky-eyes · 2024-12-30T01:37:39Z

changes/zh-cn/2.x.md

 ### feature:

- [[#PR_NO](https://github.com/seata/seata/pull/PR_NO)] 支持XXX
+- [[#7069](https://github.com/apache/incubator-seata/pull/7069)] Raft模式支持集群外访问


应该是 raft集群模式支持地址转换

funky-eyes

LGTM

集群外访问代码基本完成

5d6f777

funky-eyes added this to the 2.4.0 milestone Dec 23, 2024

funky-eyes added type: feature Category issues or prs related to feature request. module/discovery discovery module module/server server module store: raft labels Dec 23, 2024

funky-eyes reviewed Dec 23, 2024

View reviewed changes

PeppaO added 5 commits December 28, 2024 14:15

优化

96ac1ee

update

42da192

update

90683ea

update

82ca589

update

1d631e2

funky-eyes and others added 5 commits December 28, 2024 22:06

Merge branch '2.x' into support_external_access_2.x

263ddb4

update

3e03486

Merge branch '2.x' into support_external_access_2.x

9650fed

Merge branch 'support_external_access_2.x' of https://github.com/Pepp…

fe4ca17

…aO/seata into support_external_access_2.x

登记pr

5c4dd14

funky-eyes reviewed Dec 30, 2024

View reviewed changes

PeppaO changed the title ~~feature: Raft mode supports access outside the cluster~~ feature: Raft cluster mode supports address translation Dec 30, 2024

PeppaO added 2 commits December 30, 2024 10:22

pr修改

f9f9f32

add config

ff5ff02

funky-eyes approved these changes Dec 30, 2024

View reviewed changes

funky-eyes merged commit dc08160 into apache:2.x Dec 30, 2024
7 checks passed

This was referenced Nov 1, 2025

k8s部署seata-server的客户端连接问题 #7759

Closed

k8s中部署seata时客户端连接问题 #7373

Closed

YvCeung pushed a commit to YvCeung/incubator-seata that referenced this pull request Dec 25, 2025

feature: Raft cluster mode supports address translation (apache#7069)

2099e1e

	// 采取默认的方式，直接使用node.control node.transaction
	// Use the default method, directly using node.control and node.transaction

feature: Raft cluster mode supports address translation #7069

feature: Raft cluster mode supports address translation #7069

Uh oh!

Conversation

PeppaO commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Dec 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

funky-eyes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PeppaO commented Dec 23, 2024 •

edited

Loading

codecov bot commented Dec 28, 2024 •

edited

Loading