Release 1.7.1

What's New

This release updates the build to use Go 1.25.+. This is the only change in the release.

Release 1.7.0

Backwards Compatibility

Important note: The 1.7.0 router requires a 1.7.0 controller. When updating an OpenZiti network, the controller should be updated to 1.7.0 first, followed by the routers.

What's New

• proxy.v1 config type
• Alert Events (Beta)
• Azure Service Bus Event Sink (contributed by @ffaraone)

New proxy.v1 Config Type

Added support for dynamic service proxies with configurable binding and protocol options.
This allows Edge Routers and Tunnelers to create proxy endpoints that can forward traffic for Ziti services.

This differs from intercept.v1 in that intercept.v1 will intercept traffic on specified
IP ip addresses or DNS entries to forward to a service using tproxy or tun interface,
depending on implementation.

A proxy on the other hand will just start a regular TCP/UDP listener on the configured port,
so traffic will have to be configured for that destination.

Example proxy.v1 Configuration:

  {
    "port": 8080,
    "protocols": ["tcp"],
    "binding": "0.0.0.0"
  }

Configuration Properties:

• port (required): Port number to listen on (1-65535)
• protocols (required): Array of supported protocols (tcp, udp)
• binding (optional): Interface to bind to. For the ER/T defaults to the configured lanIF config property.

This config type is currently supported by the ER/T when running in either proxy or tproxy mode.

Alert Events

A new alert event type has been added to allow Ziti components to emit alerts for issues that network operators can address.
Alert events are generated when components encounter problems such as service configuration errors or resource
availability issues.

Alert events include:

• Alert source type and ID (currently supports routers, with controller and SDK support planned for future releases)
• Severity level (currently supports error, with info and warning planned for future releases)
• Alert message and supporting details
• Related entities (router, identity, service, etc.) associated with the alert

Example alert event when a router cannot bind a configured network interface:

  {
    "namespace": "alert",
    "event_src_id": "ctrl1",
    "timestamp": "2021-11-08T14:45:45.785561479-05:00",
    "alert_source_type": "router",
    "alert_source_id": "DJFljCCoLs",
    "severity": "error",
    "message": "error starting proxy listener for service 'test'",
    "details": [
      "unable to bind eth0, no address"
    ],
    "related_entities": {
      "router": "DJFljCCoLs",
      "identity": "DJFljCCoLs",
      "service": "3DPjxybDvXlo878CB0X2Zs"
    }
  }

Alert events can be consumed through the standard event system and logged to configured event handlers for monitoring and alerting purposes.

These events are currently in Beta, as the format is still subject to change. Once they've been in use in production for a while
and proven useful, they will marked as stable.

Azure Service Bus Event Sink

GitHub user @ffaraone contributed this feature, which adds support for streaming controller events to Azure Service Bus.
The new logger enables real-time event streaming from the OpenZiti controller to Azure Service Bus
queues or topics, providing integration with Azure-based monitoring and analytics systems.

To enable the Azure Service Bus event logger, add configuration to the controller config file under the events section:

  events:
    serviceBusLogger:
      subscriptions:
        - type: circuit
        - type: session
        - type: metrics
          sourceFilter: .*
          metricFilter: .*
        # Add other event types as needed
      handler:
        type: servicebus
        format: json
        connectionString: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-key"
        topic: "ziti-events"          # Use 'topic' for Service Bus topic
        # queue: "ziti-events-queue"  # Or use 'queue' for Service Bus queue
        bufferSize: 100                # Optional, defaults to 50

• Required configuration:

  • format: Event format, currently supports only json
  • connectionString: Azure Service Bus connection string
  • Either topic or queue: Destination name (mutually exclusive)

• Optional configuration:

  • bufferSize: Internal message buffer size (default: 50)

Component Updates and Bug Fixes

• github.com/openziti/agent: v1.0.31 -> v1.0.33

• github.com/openziti/channel/v4: v4.2.28 -> v4.2.41

• github.com/openziti/edge-api: v0.26.47 -> v0.26.50

• github.com/openziti/foundation/v2: v2.0.72 -> v2.0.79

  • Issue #455 - Correctly close goroutine pool when external close is signaled
  • Issue #452 - Goroutine pool with a min worker count of 1 can drop to 0 workers due to race condition

• github.com/openziti/identity: v1.0.111 -> v1.0.118

  • Issue #68 - Shutdown file watcher when stopping identity watcher

• github.com/openziti/runzmd: v1.0.80 -> v1.0.84

• github.com/openziti/sdk-golang: v1.2.3 -> v1.2.10

  • Issue #818 - Full re-auth should not clear services list, as that breaks the on-change logic
  • Issue #817 - goroutines can get stuck when iterating over randomized HA controller list
  • Issue #736 - Migrate from github.com/mailru/easyjson
  • Issue #813 - SDK doesn't stop close listener when it detects that a service being hosted gets deleted
  • Issue #811 - Credentials are lost when explicitly set
  • Issue #807 - Don't send close from rxer to avoid blocking
  • Issue #800 - Tidy create service session logging

• github.com/openziti/secretstream: v0.1.39 -> v0.1.41

• github.com/openziti/storage: v0.4.26 -> v0.4.31

• github.com/openziti/transport/v2: v2.0.188 -> v2.0.198

• github.com/openziti/go-term-markdown: v1.0.1 (new)

• github.com/openziti/ziti: v1.6.8 -> v1.7.0

  • Issue #3264 - Add support for streaming events to Azure Service Bus
  • Issue #3321 - Health Check API missing base path on discovery endpoint
  • Issue #3323 - router/tunnel static services fail to bind unless new param protocol is defined
  • Issue #3309 - Detect link connections meant for another router
  • Issue #3286 - edge-api binding doesn't have the correct path on discovery endpoints
  • Issue #3297 - stop promoting hotfixes downstream
  • Issue #3295 - make ziti tunnel service:port pairs optional
  • Issue #3291 - replace decommissioned bitnami/kubectl
  • Issue #3277 - Router can deadlock on closing a connection if the incoming data channel is full
  • Issue #3269 - Add host-interfaces config type
  • Issue #3258 - Add config type proxy.v1 so proxies can be defined dynamically for the ER/T
  • Issue #3259 - Interfaces config type not added due to wrong name
  • Issue #3265 - Forwarding errors should log at debug, since they are usual part of circuit teardown
  • Issue #3261 - ER/T dialed xgress connections may only half-close when peer is fully closed
  • Issue #3207 - Allow router embedders to customize config before start

Release 1.5.7

What's New

• Additional library updates to resolve CVEs in dependencies
• GitHub Actions release workflow fix

Component Updates and Bug Fixes

• github.com/openziti/foundation/v2: v2.0.59 -> v2.0.77

  • Issue #455 - Correctly close goroutine pool when external close is signaled
  • Issue #452 - Goroutine pool with a min worker count of 1 can drop to 0 workers due to race condition
  • Issue #443 - Allow injecting custom method into go-routine pools, to allow identifying them in stack dumps

• github.com/openziti/runzmd: v1.0.67 -> v1.0.83

• github.com/openziti/go-term-markdown: v1.0.1 (new)

• github.com/openziti/ziti: v1.5.6 -> v1.5.7

  • Issue #3291 - replace decommissioned bitnami/kubectl

Release 1.5.6

What's New

• Update several dependencies that had CVEs

Release 1.5.5

What's New

The build has been updated so this release will be created with the latest Go 1.24 release.
No other changes have been made

Release 1.6.9

What's New

This release contains a fix for the goroutine pooling functionality, which prevents a
race-condition where the pool can drop to 0 workers, when configured with 1 minimum worker. Unlike
when the pool is configured for 0 minimum workers, the pool does not recover from this state.

It also contains a fix for ER/T connections which may not fully close, causing circuits to build up.

Component Updates and Bug Fixes

• github.com/openziti/agent: v1.0.31 -> v1.0.32

• github.com/openziti/channel/v4: v4.2.28 -> v4.2.35

• github.com/openziti/foundation/v2: v2.0.72 -> v2.0.77

  • Issue #455 - Correctly close goroutine pool when external close is signaled
  • Issue #452 - Goroutine pool with a min worker count of 1 can drop to 0 workers due to race condition

• github.com/openziti/identity: v1.0.111 -> v1.0.116

  • Issue #68 - Shutdown file watcher when stopping identity watcher

• github.com/openziti/runzmd: v1.0.80 -> v1.0.82

• github.com/openziti/sdk-golang: v1.2.3 -> v1.2.4

  • Issue #800 - Tidy create service session logging

• github.com/openziti/storage: v0.4.26 -> v0.4.28

• github.com/openziti/transport/v2: v2.0.188 -> v2.0.193

• github.com/openziti/ziti: v1.6.8 -> v1.6.9

  • Issue #3261 - ER/T dialed xgress connections may only half-close when peer is fully closed

Release 1.6.8

What's New

• Bug fixes and library updates
• Session Events for JWT Sessions
• OIDC Fix when using a separate certificate for the API

Session Events for JWT sessions

When using JWT sessions, instead of legacy sessions, session events will now be created.
There is a new provider field in session events, whose value will either be legacy or jwt.

OIDC Fix

There was an issue where OIDC authentication would fail if the API was configured with a different
certificate than the controller's root identity certificate.

The v1.2.3 release of the Go SDK made OIDC the default, if the controller supported it. Since the
quickstart uses separate certs certs, this was quickly noticed. If using the v1.2.3 release of
the Go SDK, and affected by this issue, updating to OpenZiti controller v1.6.8 should resolve the
problem.

Component Updates and Bug Fixes

• github.com/openziti/agent: v1.0.30 -> v1.0.31

• github.com/openziti/channel/v4: v4.2.21 -> v4.2.28

• github.com/openziti/foundation/v2: v2.0.70 -> v2.0.72

• github.com/openziti/identity: v1.0.109 -> v1.0.111

• github.com/openziti/runzmd: v1.0.77 -> v1.0.80

• github.com/openziti/sdk-golang: v1.2.2 -> v1.2.3

  • Issue #779 - Remove need to EnableHA flag in Go SDK

• github.com/openziti/secretstream: v0.1.38 -> v0.1.39

• github.com/openziti/storage: v0.4.22 -> v0.4.26

• github.com/openziti/transport/v2: v2.0.183 -> v2.0.188

• github.com/openziti/ziti: v1.6.7 -> v1.6.8

  • Issue #3207 - Allow router embedders to customize config before start
  • Issue #3241 - Disconnecting Routers May Have Nil Fingerprint, causes panic
  • Issue #3248 - let cluster agent also support unix domain sockets
  • Issue #3219 - AuthenticatorManager ReadByFingerprint/Username should use indexes
  • Issue #3225 - JWT edge sessions should generate events
  • Issue #3245 - Revocation time check is checking wrong entity
  • Issue #3231 - OIDC authentication fails if the client api has a separate cert chain
  • Issue #3239 - Router JWTs use Identity expiration configuration value
  • Issue #3226 - Only report router network interfaces if controller supports receiving those events
  • Issue #3164 - Router data model doesn't work correctly if the edge listener isn't enabled

Release 1.6.7

What's New

• Bug fixes and library updates

Component Updates and Bug Fixes

• github.com/openziti/channel/v4: v4.2.18 -> v4.2.21

  • Issue #203 - Track last dial time in UnderlayConstraints

• github.com/openziti/edge-api: v0.26.46 -> v0.26.47

• github.com/openziti/sdk-golang: v1.2.1 -> v1.2.2

  • Issue #786 - Slow down dials to an ER if they happen too quickly
  • Issue #784 - Drop retransmit error to debug

• github.com/openziti/secretstream: v0.1.37 -> v0.1.38

• github.com/openziti/transport/v2: v2.0.182 -> v2.0.183

• github.com/openziti/ziti: v1.6.6 -> v1.6.7

  • Issue #3199 - Other routers don't react to link listener address changes
  • Issue #3178 - Controller List Edge APIs Missing
  • Issue #3193 - Add flag to TOTP auth query with enrollment state
  • Issue #3162 - Update go tunnel implementation to support multiple interfaces

Release 1.6.6

What's New

• SDK Flow Control Updates
• Multi-underlay links
• Nested Identity App Data

SDK Flow Control Updates

The circuit testing for sdk flow control is complete. Many fixes were made. The SDK flow control
code is still considered experimental, in that the features or API may change. However, it should
now be feature complete and relatively stable.. Once it has been in production use for a reasonable
period and no further changes are anticipated, it will be marked stable.

Multi-underlay Link

In previous releases, routers would attempt to set up two connections per link, one for payloads and one for acks.
If either one failed, the whole link would be torn down. With this release, links can be made up of a
user-configurable number of connections.

Link Connection Types

Link connections are of two types:

• default - These may carry payloads and acks. As long as there is at least one default connection, the link will stay up.
• ack - These may carry only acks. They act as a prioritization mechanism for acks. There may be zero ack connections.

The desired number of default and ack channels can be configured in the router configuration.

link:
  dialers:
    - binding: transport

      # Target number of default connections. Allowed range 1-100. Defaults to 3.
      maxDefaultConnections: 3
  
      # Target number of ack connections. Allowed range 1-100. Defaults to 1.
      maxAckConnections: 1

      # Time to delay making additional connections after the initial connection. Defaults to 3s
      # Reduces connection churn when routers are dialing each other at the same time.
      startupDelay: 3s

It's recommended to configure at least two connections per link.

Why Multiple Connections?

1. They allow for link continuity even if one of the connections goes down.
2. They can keep traffic moving if one of the connections stalls for some reason.
3. Using multiple links also multiples the number of OS buffers in use, although the amount of per-connection buffers can also be bumped up at the OS level.

Why a ACK Priority Connection?

If a payload gets dropped, it will need to be retransmitted. If an ack gets dropped, a payload
that's already been received will be retransmitted. Acks are also generally much smaller than
payloads. The faster we can deliver them, the faster the flow control logic can react.

How Many Connections?

At least two. However, having more connections doesn't increase the physical bandwidth available between routers. Some
additional connections provide additional resilience and perhaps more performance due to increased OS resources. However,
the benefits diminish quickly. More than the default of three is unlikely to provide much benefit.

How is traffic load-balanced?

There is a queue for payloads and other for acks. Default connections pull from both queues, ack connections only pull from
the ack queue. Because connections pull from the queues, if one connection is slower it will naturally pull fewer messages
than other connections.

Backwards Compatibility

When creating links to a router older than 1.6.6, routers will fallback to the old logic and dial one payload and one
ack channel.

Link Events

Links will now report their connections to the controller. They are now reported when listing links using ziti fabric list links.

Here is an example from a test setup.

$ ziti fabric list links 'skip 3 limit 2'
╭────────────────────────┬───────────────────────┬────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────┬──────────────────────────────────────────────────────────────╮
│ ID                     │ DIALER                │ ACCEPTOR               │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │ CONNECTIONS                                                  │
├────────────────────────┼───────────────────────┼────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┼──────────────────────────────────────────────────────────────┤
│ 101OzJLiMrrFSpwT0LnYOY │ router-eu-central-3.7 │ router-eu-central-2.11 │           1 │       2.7ms │       2.7ms │ Connected │     up │         5 │ link.default: tcp:10.0.0.230:40028 -> tcp:54.93.210.111:6011 │
│                        │                       │                        │             │             │             │           │        │           │ link.default: tcp:10.0.0.230:40032 -> tcp:54.93.210.111:6011 │
│                        │                       │                        │             │             │             │           │        │           │ link.ack: tcp:10.0.0.230:46092 -> tcp:54.93.210.111:6011     │
│                        │                       │                        │             │             │             │           │        │           │ link.default: tcp:10.0.0.230:46096 -> tcp:54.93.210.111:6011 │
│ 101YAe327nSngeRIXeKR0T │ router-eu-central-3.5 │ router-us-east-4.17    │           1 │      91.5ms │      91.4ms │ Connected │     up │       183 │ ack: tcp:10.0.0.230:57574 -> tcp:13.220.214.103:6017         │
│                        │                       │                        │             │             │             │           │        │           │ payload: tcp:10.0.0.230:57568 -> tcp:13.220.214.103:6017     │
╰────────────────────────┴───────────────────────┴────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────┴──────────────────────────────────────────────────────────────╯
results: 4-5 of 79803

A link is considered created once it has an initial default connection. The link will then attempt to reach the desired count of default
and ack connections. Whenever a new underlay connection is established or closes, the controller will be notified and an event will
be generated.

Link event example:

{
  "namespace": "link",
  "event_src_id": "ctrl_client",
  "timestamp": "2025-07-11T10:35:01.614896435-04:00",
  "event_type": "connectionsChanged",
  "link_id": "7mCYLrQAiO93du7SLGDeXf",
  "connections": [
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33682",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33686",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.ack",
      "local_addr": "tcp:127.0.0.1:33696",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33702",
      "remote_addr": "tcp:127.0.0.1:4024"
    }
  ]
}

NOTES

1. Link events show the full set of connections for the current state instead of the change.
2. New routers dialing older routers will still report link connections. See the second link in the list above.
3. Old routers will not report connections.

Nested Identity App Data

Identity app data may now be a full JSON document, rather than just a single layer map. There
are also some additional CLI methods to work with the data:

$ ziti edge create identity test --app-data foo=bar
$ ziti edge create identity test --app-data-json '{ "foo" : "bar", "test" : { "nested" : true, "number" : 234 } }'
$ ziti edge create identity test --app-data-json-file test-app-data.json 

$ ziti edge update identity test --app-data foo=bar
$ ziti edge update identity test --app-data-json '{ "foo" : "bar", "test" : { "nested" : true, "number" : 234 } }'
$ ziti edge update identity test --app-data-json-file test-app-data.json 

Component Updates and Bug Fixes

• github.com/openziti/agent: v1.0.29 -> v1.0.30

  • Issue #27 - Add support for generating heap dumps

• github.com/openziti/channel/v4: v4.2.13 -> v4.2.18

  • Issue #201 - SendAndWait methods should return an error if the channel closes instead of blocking
  • Issue #199 - Reject multi-underlay connections that are the first connection for a channel, but aren't marked as such.
  • Issue #197 - Break out of dial loop if channel is closed

• github.com/openziti/foundation/v2: v2.0.69 -> v2.0.70

• github.com/openziti/identity: v1.0.108 -> v1.0.109

• github.com/openziti/runzmd: v1.0.76 -> v1.0.77

• github.com/openziti/sdk-golang: v1.1.2 -> v1.2.1

  • Issue #777 - OIDC auth token refresh doesn't fall back to re-auth if token has expired
  • Issue #772 - xgress close tweaks
  • Issue #769 - Require sdk flow control when using more than one default connection
  • Issue #765 - Allow independent close of xgress send and receive
  • Issue #763 - Use a go-routine pool for payload ingest
  • Issue #761 - Use cmap.ConcurrentMap for message multiplexer
  • Issue #754 - panic: unaligned 64-bit atomic operation when running on 32-bit raspberry pi
  • Issue #757 - Not authenticated check fails on session create when using OIDC

• github.com/openziti/secretstream: [v0.1.36 -> v0.1.37](https://github.co...

Release 1.6.5

What's New

Bugfixes and dependency updates.

Component Updates and Bug Fixes

• github.com/openziti/channel/v4: v4.2.8 -> v4.2.13

  • Issue #194 - Add GetUnderlays and GetUnderlayCountsByType to Channel

• github.com/openziti/foundation/v2: v2.0.66 -> v2.0.69

  • Issue #443 - Allow injecting custom method into go-routine pools, to allow identifying them in stack dumps

• github.com/openziti/identity: v1.0.105 -> v1.0.108

• github.com/openziti/metrics: v1.4.1 -> v1.4.2

• github.com/openziti/runzmd: v1.0.73 -> v1.0.76

• github.com/openziti/storage: v0.4.17 -> v0.4.20

• github.com/openziti/transport/v2: v2.0.177 -> v2.0.180

• github.com/openziti/xweb/v2: v2.3.3 -> v2.3.4

• github.com/openziti/ziti: v1.6.3 -> v1.6.5

  • Issue #3149 - add dial/bind type column to sp list