Thanks to visit codestin.com
Credit goes to github.com

Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host #7823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gangaswamy opened this issue Aug 6, 2023 · 19 comments

Comments

@gangaswamy
Copy link

Similar bugs were reported for previous versions like #6716, but tagged as fixed however it is not working in 4.18

ISSUE TYPE
  • Bug Report

UI,

CLOUDSTACK VERSION

4.18

CONFIGURATION

Basic Networking

OS / ENVIRONMENT

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

SUMMARY

Error 530
Could not add host at [http://192.168.122.1] with zone [2], pod [2] and cluster [2] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.122.1 - Failed to setup keystore on the KVM host: 192.168.122.1].

STEPS TO REPRODUCE

Fresh install on Ubuntu 20.04
Cloudstack Management server installed successfully and I can add Zone, Pod, Cluster, Network etc but not able to add host on KVM host.
I am using the sudo user and password

EXPECTED RESULTS

Suppose to add host easily

ACTUAL RESULTS

Not able to add KVM host

@boring-cyborg
Copy link

boring-cyborg bot commented Aug 6, 2023

Thanks for opening your first issue here! Be sure to follow the issue template!

@rohityadavcloud
Copy link
Member

Hi @gangaswamy can you share your management server and KVM agent logs in DEBUG more (set DEBUG instead of INFO in the log4j xml configuration). Does it happen only on 20.04, have you tried 22.04?

@gangaswamy
Copy link
Author

I am trying on Ubuntu 20.04,

$ sudo systemctl status cloudstack-agent

  • cloudstack-agent.service - CloudStack Agent
    Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
    Active: activating (auto-restart) since Sun 2023-08-06 16:45:59 CDT; 3s ago
    Docs: http://www.cloudstack.org/
    Process: 7191 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUC>
    Main PID: 7191 (code=exited, status=0/SUCCESS)

$ sudo systemctl status cloudstack-management

  • cloudstack-management.service - CloudStack Management Server
    Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
    Active: active (running) since Sun 2023-08-06 16:22:10 CDT; 24min ago
    Main PID: 1269 (java)
    Tasks: 189 (limit: 38371)
    Memory: 1.1G
    CGroup: /system.slice/cloudstack-management.service
    `-1269 /usr/bin/java -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers -Djava.a>

Log Management-server.log
tail -100 management-server.log
2023-08-06 16:46:52,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5334a76b) (logid:409817d1) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:46:52,715 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-fdcea4b5) (logid:9801384b) HA health check task is running...
2023-08-06 16:46:53,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:46:53,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Trying to connect to 69.30.206.114
2023-08-06 16:46:53,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:46:54,036 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-b0695b7a) (logid:30f15c20) AutoScaling Monitor is running...
2023-08-06 16:46:54,037 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-b0695b7a) (logid:30f15c20) Skipping AutoScaling Monitor
2023-08-06 16:46:54,040 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-0cac04e8) (logid:ee24c5bf) HostStatsCollector is running...
2023-08-06 16:46:54,074 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-dd217e8b) (logid:6b972f22) StorageCollector is running...
2023-08-06 16:46:55,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:46:55,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Trying to connect to 69.30.206.114
2023-08-06 16:46:55,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:46:56,717 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-cf2e0db4) (logid:709a1cfe) HA health check task is running...
2023-08-06 16:46:56,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:46:56,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Trying to connect to 69.30.206.114
2023-08-06 16:46:56,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:46:58,270 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:46:58,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Trying to connect to 69.30.206.114
2023-08-06 16:46:58,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:46:59,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:46:59,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Trying to connect to 69.30.206.114
2023-08-06 16:46:59,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:00,719 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-05d2fb9b) (logid:60d3c258) HA health check task is running...
2023-08-06 16:47:00,785 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-0273aabb) (logid:5b5fb62c) Enabled non-edge zones available for scan:
2023-08-06 16:47:00,806 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-94682ae5) (logid:5084e107) Skip capacity scan as there is no Primary Storage in 'Up' state
2023-08-06 16:47:01,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:01,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Trying to connect to 69.30.206.114
2023-08-06 16:47:01,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:02,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:02,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Trying to connect to 69.30.206.114
2023-08-06 16:47:02,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:04,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:04,268 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Trying to connect to 69.30.206.114
2023-08-06 16:47:04,268 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:04,721 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-c1821802) (logid:755d25d1) HA health check task is running...
2023-08-06 16:47:05,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:05,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Trying to connect to 69.30.206.114
2023-08-06 16:47:05,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:07,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:07,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Trying to connect to 69.30.206.114
2023-08-06 16:47:07,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:08,723 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-6afb8f78) (logid:3850adc9) HA health check task is running...
2023-08-06 16:47:08,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:08,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Trying to connect to 69.30.206.114
2023-08-06 16:47:08,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:09,029 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 routers to update status.
2023-08-06 16:47:09,031 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 VPC's to update Redundant State.
2023-08-06 16:47:09,032 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 networks to update RvR status.
2023-08-06 16:47:10,270 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:10,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Trying to connect to 69.30.206.114
2023-08-06 16:47:10,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:11,769 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:11,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Trying to connect to 69.30.206.114
2023-08-06 16:47:11,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:12,725 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-66d5a4dd) (logid:332442fe) HA health check task is running...
2023-08-06 16:47:13,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:13,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Trying to connect to 69.30.206.114
2023-08-06 16:47:13,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:14,769 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:14,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Trying to connect to 69.30.206.114
2023-08-06 16:47:14,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:16,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:16,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Trying to connect to 69.30.206.114
2023-08-06 16:47:16,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:16,726 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-7e7be91a) (logid:0798b84f) HA health check task is running...
2023-08-06 16:47:17,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:17,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Trying to connect to 69.30.206.114
2023-08-06 16:47:17,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:19,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:19,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Trying to connect to 69.30.206.114
2023-08-06 16:47:19,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:20,728 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-29a0a8d8) (logid:fb74a13e) HA health check task is running...
2023-08-06 16:47:20,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:20,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Trying to connect to 69.30.206.114
2023-08-06 16:47:20,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:22,277 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:22,277 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Trying to connect to 69.30.206.114
2023-08-06 16:47:22,277 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:23,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:23,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Trying to connect to 69.30.206.114
2023-08-06 16:47:23,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:24,730 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-ed0a4a4c) (logid:43ada46b) HA health check task is running...
2023-08-06 16:47:25,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:25,274 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Trying to connect to 69.30.206.114
2023-08-06 16:47:25,274 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:26,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:26,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Trying to connect to 69.30.206.114
2023-08-06 16:47:26,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:28,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:28,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Trying to connect to 69.30.206.114
2023-08-06 16:47:28,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:28,732 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-b17d4c43) (logid:678c3cee) HA health check task is running...
2023-08-06 16:47:29,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:29,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Trying to connect to 69.30.206.114
2023-08-06 16:47:29,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Management node 4 is detected inactive by timestamp but is pingable
2023-08-06 16:47:30,785 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-a6a43229) (logid:d983f3ba) Enabled non-edge zones available for scan:
2023-08-06 16:47:30,806 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a1a52827) (logid:d3350bcf) Skip capacity scan as there is no Primary Storage in 'Up' state
2023-08-06 16:47:31,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Detected management node left, id:4, nodeIP:69.30.206.114
2023-08-06 16:47:31,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Trying to connect to 69.30.206.114
2023-08-06 16:47:31,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Management node 4 is detected inactive by timestamp but is pingabl

Agent log
$ tail -100 agent.log
2023-08-06 16:43:19,029 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:43:29,767 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:43:29,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:43:29,773 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:43:40,469 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:43:40,474 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:43:40,476 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:43:51,034 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:43:51,039 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:43:51,041 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:01,675 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:01,680 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:01,681 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:12,273 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:12,278 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:12,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:23,016 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:23,021 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:23,023 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:33,770 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:33,775 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:33,777 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:44,528 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:44,533 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:44,535 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:44:55,272 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:44:55,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:44:55,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:06,019 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:06,024 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:06,025 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:16,768 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:16,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:16,774 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:27,521 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:27,526 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:27,527 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:38,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:38,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:38,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:49,019 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:49,024 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:49,026 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:45:59,760 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:45:59,764 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:45:59,766 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:46:10,524 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:46:10,529 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:46:10,530 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:46:21,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:46:21,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:46:21,276 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:46:32,022 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:46:32,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:46:32,028 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:46:42,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:46:42,777 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:46:42,778 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:46:53,511 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:46:53,516 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:46:53,517 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:04,280 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:04,285 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:04,287 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:15,021 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:15,025 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:15,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:25,757 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:25,763 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:25,765 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:36,518 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:36,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:36,525 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:47,270 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:47,275 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:47,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:47:58,022 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:47:58,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:47:58,029 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:48:08,781 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:48:08,785 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:48:08,787 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:48:19,517 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:48:19,521 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:48:19,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:48:30,273 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:48:30,278 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:48:30,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:48:41,008 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:48:41,013 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:48:41,015 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:48:51,773 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:48:51,778 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:48:51,780 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:49:02,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:49:02,528 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:49:02,530 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-08-06 16:49:13,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-08-06 16:49:13,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0
2023-08-06 16:49:13,276 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties

@rajujith
Copy link

rajujith commented Aug 7, 2023

@gangaswamy could you check whether package tzdata-java is installed on the KVM host if not retry the agent setup after installing this package?

@gangaswamy
Copy link
Author

$ dpkg -l | grep tzdata
ii tzdata 2023c-0ubuntu0.20.04.2 all time zone and daylight-saving time data

it is installed and its latest version. tzdata-java package is not available for Ubuntu

@xuanyuanaosheng
Copy link

@gangaswamy Was this server added to CloudStack during zone initialization? If so, please modify the server's IP address to another unused IP address in the same VLAN, and re-add the server.

@weizhouapache
Copy link
Member

weizhouapache commented Aug 10, 2023

@gangaswamy
it looks cloudstack-agent service is stuck at activating

can you run commands systemctl status cloudstack-agent and journalctl -xeu cloudstack-agent ?
they might give more information

please also check the status and log of libvirtd service.

@axeldaiber
Copy link

Any news regarding this issue? I do have the same issue with Ubuntu 22.04 jammy and Cloudstack 4.18.1.0: Failed to setup keystore on the KVM host:

@weizhouapache
Copy link
Member

Any news regarding this issue? I do have the same issue with Ubuntu 22.04 jammy and Cloudstack 4.18.1.0: Failed to setup keystore on the KVM host:

@axeldaiber
although the log says "Failed to setup keystore", the issue could be caused by something else, for example the network config.

do you use netplan ? have you configured cloudbr0 and/or cloudbr1 ?

@hoogenm
Copy link

hoogenm commented Jan 28, 2024

I got this error message when the automated ssh-login (from setup to the host) without a tty was unsuccessful. For diagnostics, I tried again with root, and that worked. Hope this may help others. Ubuntu Jammy & Cloudstack 4.18

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Apr 30, 2024

I'm not sure if using the sudo user and password is supported. The docs suggest to use root user with password, or with ssh-public key auth of the management server (cloud user's ssh key can ssh the KVM host via root user)

@rudolfjs
Copy link

I'm not sure if using the sudo user and password is supported. The docs suggest to use root user with password, or with ssh-public key auth of the management server (cloud user's ssh key can ssh the KVM host via root user)

I temporarily allowed root password login via SSH (revered after deployment) and it worked. Setting root as the user for the target host fixed the keystore problem.

@tampler
Copy link

tampler commented Sep 9, 2024

I see this error on ACS 4.20.0:

can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.1.1 - Failed to setup keystore on the K
VM host: 192.168.1.1 

This happens due to the following:

2024-09-10T11:22:15,279 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-370:[ctx-23c36759, ctx-2d609a42]) (logid:136b20d9) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/                                      
SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper            
sudo: a password is required

So there's an invalid ssh auth on the ACS mgmt side when adding a host with the user/pass.
Should be done with the "-S" option as per error above

@tampler
Copy link

tampler commented Sep 10, 2024

Actually, keystore-setup script doesn't work:

> # /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying

@weizhouapache
Copy link
Member

Actually, keystore-setup script doesn't work:

> # /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying

this is not the full command , if you copied from management server log.

@tampler
Copy link

tampler commented Sep 10, 2024

this is not the full command , if you copied from management server log.

Yes, the command above was lifted from the log. I executed it in the command line and pasted the output

2024-09-10T11:52:05,946 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-33:[ctx-b9629dea, ctx-60272547]) (logid:c352422a) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/

SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

@weizhouapache
Copy link
Member

this is not the full command , if you copied from management server log.

Yes, the command above was lifted from the log. I executed it in the command line and pasted the output

2024-09-10T11:52:05,946 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-33:[ctx-b9629dea, ctx-60272547]) (logid:c352422a) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/

SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

Can you enable NOPASSWD in sudoers for the user ?

@tampler
Copy link

tampler commented Sep 10, 2024

Can you enable NOPASSWD in sudoers for the user ?

@weizhouapache Thanks, that fixed my problem! I've been using ACS since 2019 and each time I reinstall it I figure out yet another issue/trouble 😆

@almostobvious
Copy link

Spent an hour battling this one. For me error wasn't associated with access. I was only able to resolve by installing agent manually and then making sure the java 17 is default.

On the host you are adding:

  1. Add the cloudstack yum repo
  2. yum install cloudstack-agent
  3. alternatives --config java (and then select java 17)

The installation guide needs a revamp. It didn't say anything about how to prepare a host. I only found some (outdated) instructions in quickstart which is focused on much simpler setup. Also, the need to create bridges is only in troubleshooting but not in the installation guide.

@apache apache locked and limited conversation to collaborators Apr 15, 2025
@DaanHoogland DaanHoogland converted this issue into discussion #10716 Apr 15, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Projects
Development

No branches or pull requests