KeepAlive and TCP timeout set in destination rule does not seem to apply #56576

snyssen · 2025-06-11T10:00:47Z

snyssen
Jun 11, 2025

Hello,

We are trying to setup a connection to a remote service through an egress gateway, but we have an issue with our cloud gateway that closes the TCP connection silently after 4 min while both the egress gateway and the remote service keep it open; this means that the connection gets reused on subsequent calls, but packets never actually go through since the cloud gateway has already closed it.

Our proposed solution to this issue was to use the connection pool TCP settings of our destination rule to either (or both) send keepalive packets or close the TCP connection before the cloud gateway does it. However, it seems that none of the settings are applied properly. Please find below an example of our e2e configuration (merely a reproduction of our issue actually, targeting my own website).

---
# The gateway Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: snyssen-gateway
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: snyssen-gateway
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway
      labels:
        istio: snyssen-gateway
        sidecar.istio.io/inject: "true"
    spec:
      containers:
        - name: istio-proxy
          image: auto
          securityContext:
            capabilities:
              drop:
                - ALL
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
          ports:
            - containerPort: 15090
              protocol: TCP
              name: http-envoy-prom
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 128Mi
      securityContext:
        sysctls:
          - name: net.ipv4.ip_unprivileged_port_start
            value: "0"
---
# The gateway Service
apiVersion: v1
kind: Service
metadata:
  name: snyssen-gateway
  namespace: istio-system
spec:
  type: ClusterIP
  selector:
    istio: snyssen-gateway
  ports:
    - name: status-port
      port: 15021
      protocol: TCP
      targetPort: 15021
    - name: http2
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
---
# The Gateway itself
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: snyssen-gateway
  namespace: istio-system
spec:
  selector:
    istio: snyssen-gateway
  servers:
    - hosts:
        - snyssen.be
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: ISTIO_MUTUAL
---
# The ServiceEntry for the external service
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: snyssen
  namespace: istio-system
spec:
  hosts:
    - snyssen.be
  location: MESH_EXTERNAL
  ports:
    - name: https-port
      number: 443
      protocol: HTTPS
  resolution: DNS
---
# The VirtualService that handles traffic from the client pod to the gateway and then from the gateway to the external service
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: snyssen
  namespace: istio-system
spec:
  gateways:
    - mesh
    - istio-system/snyssen-gateway
  hosts:
    - snyssen.be
  http:
    - match:
        - gateways:
            - mesh
          port: 80
      route:
        - destination:
            host: snyssen-gateway.istio-system.svc.cluster.local
            port:
              number: 443
            subset: snyssen
          weight: 100
    - match:
        - gateways:
            - istio-system/snyssen-gateway
          port: 443
      route:
        - destination:
            host: snyssen.be
            port:
              number: 443
          weight: 100
---
# The DestinationRule between the client pod and the gateway
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: snyssen-sidecar-to-egress
  namespace: istio-system
spec:
  host: snyssen-gateway.istio-system.svc.cluster.local
  subsets:
    - name: snyssen
      trafficPolicy:
        connectionPool:
          http:
            idleTimeout: 30s
          tcp:
            maxConnections: 100
        loadBalancer:
          simple: ROUND_ROBIN
        portLevelSettings:
          - port:
              number: 443
            tls:
              mode: ISTIO_MUTUAL
              sni: snyssen.be
---
# The DestinationRule between the gateway and the external service
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: snyssen-443
  namespace: istio-system
spec:
  host: snyssen.be
  trafficPolicy:
    connectionPool:
      tcp:
        idleTimeout: 120s
        tcpKeepalive:
          interval: 10s
          probes: 3
          time: 15s
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: SIMPLE
          sni: snyssen.be

In this example, the external service (https://snyssen.be) is actually closing the connection after 3 minutes, but I think it still serves as a good example since we have set the destination rule to start sending keepalive packets after 15s and to close the connection after 120s. As you can see in the following TCP dump, no keepalive is ever sent to the external service, and the connection is not closed after 120s.

(Alas the screenshot would not upload, I will try again later. In it you could see TCP packets exchange between the gateway and the remote service, then nothing for 3 minutes, and then the remote service closing the connection.)

The TCP dump is captured at the gateway using ksniff with the following command:

kubectl sniff -n istio-system snyssen-gateway-754c9bc788-g4vs2 -c istio-proxy -p -f "net 81.240.174.125"

What we tried:

Running on different clusters and with different Istio versions. The issue first became apparent on our Openshift cluster, where Istio version 1.24.1 (though istioctl version reports control plane version: 1.24.1_ossm_tp.2 is installed; but data plane version: 1.24.1) it was replicated on my dev machine in Minikube with istio version 1.24.3.
Bypassing the Gateway by updating the VirtualService to the following (and thus reusing the same DestinationRule):

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: snyssen
  namespace: istio-system
spec:
  gateways:
    - mesh
    - istio-system/snyssen-gateway
  hosts:
    - snyssen.be
  http:
    - match:
        - gateways:
            - mesh
          port: 80
      route:
        - destination:
            host: snyssen.be
            port:
              number: 443
          weight: 100

Then sniffing packets directly on the sidecar:

kubectl sniff -n investigation-fscloud-2551 network-multitool-84f4768858-9w9qn -c istio-proxy -p -f "net 81.240.174.125"

Which unfortunately highlights the same issue.

snyssen · 2025-06-12T06:59:35Z

snyssen
Jun 12, 2025
Author

We have also tried setting an EnvoyFilter directly targeting the gateways, but this also has no effect

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-tcp-timeout
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.tcp_proxy
      patch:
        operation: MERGE
        value:
          name: envoy.filters.network.tcp_proxy
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
            idle_timeout: 30s

5 replies

snyssen Jun 13, 2025
Author

Seeing how my issue seems very similar to #29115 (only mine is on outbound), I have tried setting the ISTIO_META_IDLE_TIMEOUT variable on the IstioOperator config, like so:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator
  namespace: istio-system
spec:
  components:
    cni:
      enabled: true
      namespace: kube-system
  meshConfig:
    accessLogFile: /dev/stdout
    defaultConfig:
      proxyMetadata:
        ISTIO_META_IDLE_TIMEOUT: 30s # <---------
  values:
    global:
      platform: openshift

But I noticed no change. I am unsure how such variable should be applied though.

zirain Jun 13, 2025
Collaborator

ISTIO_META_IDLE_TIMEOUT required rollout/restart.

snyssen Jun 13, 2025
Author

I have restarted istiod and the gateway and I still don't see the effect unfortunately.

snyssen Jun 16, 2025
Author

As a workaround, we have now applied a retry on reset (which apparently covers cases where the server does not respond at all) on the VirtualService

retries:
  attempts: 3
  perTryTimeout: 2s
  # https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-retry-on
  retryOn: reset,connect-failure

But this is of course only hiding the actual issue, which we would prefer to fix.

ThomasGaliotto Jun 17, 2025

Hello,

We are running in the same situation. I believe the tcp proxy filter applies on tcp gateway & tcp virtual service only.
Also,

                            "commonHttpProtocolOptions": {
                                "idleTimeout": "30s"
                            },

does not seem to apply on mTLS connection. Maybe someone can confirm this...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KeepAlive and TCP timeout set in destination rule does not seem to apply #56576

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

KeepAlive and TCP timeout set in destination rule does not seem to apply #56576

Uh oh!

Uh oh!

snyssen Jun 11, 2025

Replies: 1 comment · 5 replies

Uh oh!

snyssen Jun 12, 2025 Author

Uh oh!

snyssen Jun 13, 2025 Author

Uh oh!

zirain Jun 13, 2025 Collaborator

Uh oh!

snyssen Jun 13, 2025 Author

Uh oh!

snyssen Jun 16, 2025 Author

Uh oh!

ThomasGaliotto Jun 17, 2025

snyssen
Jun 11, 2025

Replies: 1 comment 5 replies

snyssen
Jun 12, 2025
Author

snyssen Jun 13, 2025
Author

zirain Jun 13, 2025
Collaborator

snyssen Jun 13, 2025
Author

snyssen Jun 16, 2025
Author