Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix 66568: Prevent unclosed TCP channels and avoid additional errors #68277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: 3007.x
Choose a base branch
from
Open

Fix 66568: Prevent unclosed TCP channels and avoid additional errors #68277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
90f070b
Fix 66568: Prevent unclosed TCP channels and avoid additional errors
Aug 20, 2025
5c74aaf
Fix formatting errors
Aug 26, 2025
d8ed156
Add tests to ensure that tasks are closed and transport is cleaned up…
Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/66568.fixed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix closing of TCP transport channels and avoid additional errors
5 changes: 4 additions & 1 deletion salt/channel/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,10 @@ def connect_callback(self, result):
# may have been restarted
yield self.send_id(self.token, self._reconnected)
self.connected = True
self.event.fire_event({"master": self.opts["master"]}, "__master_connected")
if self.event:
Copy link
Contributor

@dwoz dwoz Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When won't there be an event? Is this something that came up in the test suite only?

Copy link
Contributor Author

@bluesliverx bluesliverx Sep 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dwoz, I'm not sure exactly what triggered it, but we would see an error about self.event being None when running salt-call commands pretty reliably. This prevents the error, but I'm not sure how to even reproduce this in a test to be honest without having it be very contrived.

Copy link
Contributor

@bdrx312 bdrx312 Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There won't be an event if the channel is already closed.

    def close(self):
        """
        Close the channel
        """
        self.transport.close()
        if self.event is not None:
            self.event.destroy()
            self.event = None

Is it possible this could get called after the channel is already closed?

Copy link
Contributor Author

@bluesliverx bluesliverx Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bdrx312, not sure if you're asking me or @dwoz. For my part, I'm not sure why the behavior is happening, only that it is happening. And sadly, I lack the deeper knowledge of salt's message bus handling to know exactly what's going on here.

Copy link
Contributor

@bdrx312 bdrx312 Oct 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was asking @dwoz, trying to address his question of a way that there might not be an event. It seems appropriate to handle the case of there not being event here in case reconnect is called on when it was already closed.

self.event.fire_event(
{"master": self.opts["master"]}, "__master_connected"
)
if self._reconnected:
# On reconnects, fire a master event to notify that the minion is
# available.
Expand Down
17 changes: 15 additions & 2 deletions salt/transport/tcp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,6 @@ def __init__(self, opts, io_loop, **kwargs): # pylint: disable=W0231
self.connected = False
self._closing = False
self._stream = None
self._closing = False
self._closed = False
self.backoff = opts.get("tcp_reconnect_backoff", 1)
self.resolver = kwargs.get("resolver")
Expand Down Expand Up @@ -1763,6 +1762,7 @@ def __init__(self, opts, io_loop, **kwargs): # pylint: disable=W0231
self._closed = False
self._stream_return_running = False
self._stream = None
self.task = None
self.disconnect_callback = _null_callback
self.connect_callback = _null_callback
self.backoff = opts.get("tcp_reconnect_backoff", 1)
Expand Down Expand Up @@ -1831,7 +1831,7 @@ async def _stream_return(self):
message_id,
)
except tornado.iostream.StreamClosedError as e:
log.error(
log.debug(
"tcp stream to %s:%s closed, unable to recv",
self.host,
self.port,
Expand Down Expand Up @@ -1873,6 +1873,8 @@ async def _stream_return(self):
stream.close()
unpacker = salt.utils.msgpack.Unpacker()
await self.connect()
except asyncio.CancelledError:
log.debug("Stream return cancelled")
self._stream_return_running = False

def _message_id(self):
Expand Down Expand Up @@ -1921,9 +1923,20 @@ async def _do_send():
def close(self):
if self._closing:
return
self._closing = True
if self._stream is not None:
self._stream.close()
self._stream = None
if self.task is not None:
self.task.cancel()
# Wait for the task to finish via asyncio
group = asyncio.gather(self.task)
try:
self.task.get_loop().run_until_complete(group)
except RuntimeError:
# Ignore event loop was already running message
pass
self.task = None

def __enter__(self):
return self
Expand Down
6 changes: 6 additions & 0 deletions tests/pytests/functional/transport/server/test_request_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,16 @@ async def handler(message):
try:

ret = await req_client.send({"req": "test"})
if transport == "tcp":
assert req_client.task is not None
assert [reqmsg] == requests
assert repmsg == ret
finally:
req_client.close()
if transport == "tcp":
# Ensure that issue 68277 is fixed
assert req_client.task is None or req_client.task.done()
assert req_client._closing
req_server.close()

# Yield to loop in order to allow background close methods to finish.
Expand Down
Loading