[BUG] 3007 master 4505 publish port failed to respond to salt commands unless restart #66282
Description
Description
3007 master not responding salt commands unless restarting the master. But it does answer salt-call ran from minion
The timeout issue does not have log in /var/log/salt/master file with log file level set to debug.
Currently trying the trace level.
log_level_logfile: trace
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- [ x] on-prem machine
- [ x] VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- [ x] onedir packaging
- [ x] used bootstrap to install
Steps to Reproduce the behavior
$sudo salt minion-on-saltmaster test.ping
Expected behavior
salt-master should answer the test.ping command quickly. In stead,it timeout.
Screenshots
- cmd line
[me@salt01 ~]$ sudo salt salt01 test.ping
salt01:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20240328074306181473
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 0
# of minions that did not return: 1
# of minions with errors: 0
-------------------------------------------
ERROR: Minions returned with non-zero exit code
[me@salt01 ~]$
- and this is trace log in /etc/salt/master when a minion not responding.
2024-03-28 02:36:29,940 [salt.loader.lazy ][DEBUG ] The functions from module 'no_return' are being loaded by dir() on the loaded module
2024-03-28 02:36:29,940 [salt.utils.lazy ][DEBUG ] LazyLoaded no_return.output
2024-03-28 02:36:29,941 [salt.output ][TRACE ] data = {'salt01': 'Minion did not return. [No response]\nThe minions may not have all finished running and any remaining minions will return upon completion.
To look up the return data for this job later, run the following command:\n\nsalt-run jobs.lookup_jid 20240328073609814061'}
- salt-master status
Mar 27 13:09:15 salt01 salt-master[2162920]: [ERROR ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
Mar 27 14:00:04 salt01 salt-master[2163029]: [ERROR ] Requested method not exposed: _ext_nodes
Mar 27 15:00:04 isalt01 salt-master[2163274]: [ERROR ] Requested method not exposed: _ext_nodes
<snipped>
Mar 28 03:00:04 salt01 salt-master[2163147]: [ERROR ] Requested method not exposed: _ext_nodes
Mar 28 04:00:07 salt01 salt-master[2163282]: [ERROR ] Requested method not exposed: _ext_nodes
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.18.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green ObsidianAdditional context
Add any other context about the problem here.