Thanks to visit codestin.com
Credit goes to github.com

Skip to content

New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) #10514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
May 2, 2025
Merged

New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand) #10514

merged 13 commits into from
May 2, 2025

Conversation

weizhouapache
Copy link
Member

@weizhouapache weizhouapache commented Mar 6, 2025 •
edited
Loading

Description

This PR aims to improve the process on some agent commands and answers.

Current process

Many cloudstack operations require the communication between management server and cloudstack agent.
The normal process is

management server --> send commands to agents --> agents process the commands ->
agents send the answers to management server --> management server process the answers

Each operation might have one or more processes above.

Issues in some scenarios

Normally the process works fine. However, there are some issues in some scenarios

  • agent lost connectivity to management server
  • agent crashes
  • agent is stopped or restarted
  • management server is restarted (not in scope of FR doc)

Consider the following examples

  • agent has processed the command completely, and sent the answer to management server but the answer is not received on management server (connectivity issue or worker threads are all in use), therefore timed out on management server
  • agent has asked 3rd party (for example libvirt or scaleio) to process the command, but agent crashes, which leads to some inconsistent state of resources. For example vm has been migrated, volume has been copied to destination pool, but management server does not get the results.
  • management is restarted while vm/volume is being migrated, which leads vm/volume to be stuck at Migrating state.

Operations to address

This FR focuses on the following operations

  • migrate vm
  • migrate vm with volumes
  • migrate volume to another pool

The backend processes can be found at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337678693#AsyncAgentCommandReconciliation-4.1BackendcommandsofVMandvolumemigrations

Main changes

Design doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation

Global settings

  • reconcile.commands.enabled (default: false)
  • reconcile.commands.interval (default: 60)
  • reconcile.commands.max.attempts (default: 30)
  • reconcile.commands.workers (default: 100)

New terminology: Reconcile commands

  • Add command state: CommandInfo.State
    • Management server: CREATED -> COMPLETED/FAILED or INTERRUPTED/TIMED_OUT -> RECONCILING -> RECONCILED/RECONCILE_RETRY/RECONCILE_FAILED
    • Agent: STARTED -> PROCESSING -> COMPLETED/FAILED/INTERRUPTED
  • Add property to Command: isReconcile (false by default). true for 3 commands
    • CopyCommand
    • MigrateCommand
    • MigrateVolumeCommand

How it works

  • management server creates a record in reconcile_commands table when create the command
  • agent updates the command/answer in a JSON file while process the command
  • agent syncs with management server every 60 seconds (ping.interval)
  • when management server receives the update from agent, it updates the database
  • when agent receives the PingAnswer form management server, it removes the JSON file with state=COMPLETED/FAILED

For reconcile commands, during stop/start of mgmt server and agent

  • when agent is stopped/started, it updates the state (by agent) to INTERUPTED in JSON files and sync with mgmt server
  • when mgmt server is stopped/started, it updates the state (by management) to INTERUPTED in database
  • Every minute, management server loads the reconcile commands and reconcile the command in INTERUPTED or TIMED_OUT state

Improvement on management server when wait for the answer of reconcile commands

  • meanwhile, every 60 seconds, it reads the state and answer from database
  • if state is INTERUPTED or FAILED, it terminates the process
  • if state is COMPLETED and answer is null, it processes the answer as normal
  • This fixes the intermittent connection failure between agent and management server
  • when times out, it updates the state (by mgmt) to TIMED_OUT in database for further reconciliation

Improvement on VM migration w/wo volumes

  • If the operation fails with operation timeout exception, it might because of the connection failure between soource host and management server
  • check if the vm is Running on destination host, if yes, consider the migration succceed
  • if succeed, consider migration as success, and commit the network and volume changes.
  • if not, destroy the vm on destination host and rollback the network/volume changes

Fixes after Volume migration

  • on NFS, fix the new volumes at Creating state after failed migration
  • on Powerflex, fix the read-only VM issue by checking if volume has been migrated by checking via ScaleIO gateway

Improvement on Agent

  • Add MigrationCancelHook to terminate vm migration jobs when agent is disconnected or restarted
  • Add VolumeMigrationCancelHook to terminate block copy jobs when agent is disconnected or restarted

Test results

It has been tested by dev on NFS and Powerflex

Refer to https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation#AsyncAgentCommandReconciliation-4.3Summaryoftestresults

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@codecov Codecov
Copy link

codecov bot commented Mar 6, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 11.11686% with 1719 lines in your changes missing coverage. Please review.

Project coverage is 16.28%. Comparing base (4c31f9d) to head (b4bc020).
Report is 68 commits behind head on main.

Files with missing lines Patch % Lines
...loudstack/command/ReconcileCommandServiceImpl.java 0.00% 751 Missing ⚠️
...source/wrapper/LibvirtReconcileCommandWrapper.java 0.62% 160 Missing ⚠️
.../apache/cloudstack/command/ReconcileCommandVO.java 0.00% 96 Missing ⚠️
...ache/cloudstack/command/ReconcileCommandUtils.java 37.30% 73 Missing and 6 partials ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java 26.16% 76 Missing and 3 partials ⚠️
...loudstack/command/dao/ReconcileCommandDaoImpl.java 0.00% 70 Missing ⚠️
...ain/java/com/cloud/agent/manager/AgentAttache.java 3.70% 52 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java 3.03% 32 Missing ⚠️
...torage/motion/StorageSystemDataMotionStrategy.java 0.00% 32 Missing ⚠️
...atastore/driver/ScaleIOPrimaryDataStoreDriver.java 21.62% 28 Missing and 1 partial ⚠️
... and 42 more
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #10514      +/-   ##
============================================
- Coverage     16.30%   16.28%   -0.03%     
- Complexity    13449    13483      +34     
============================================
  Files          5676     5695      +19     
  Lines        499208   501081    +1873     
  Branches      60374    60657     +283     
============================================
+ Hits          81414    81612     +198     
- Misses       408722   410376    +1654     
- Partials       9072     9093      +21
Flag Coverage Δ
uitests 3.99% <ø> (ø)
unittests 17.14% <11.11%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@weizhouapache weizhouapache force-pushed the 4.21-reconcile-commands branch from 5dcdcb8 to f9bba6e Compare March 6, 2025 09:21
@weizhouapache weizhouapache added this to the 4.21.0 milestone Mar 6, 2025
@weizhouapache
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12678

@weizhouapache
Copy link
Member Author

@blueorangutan test matrix

@blueorangutan
Copy link

@weizhouapache a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-12593)

@JoaoJandre JoaoJandre self-requested a review March 6, 2025 12:01
@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-12594)

@blueorangutan
Copy link

[SF] Trillian test result (tid-12592)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58377 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12592-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1521.32 test_network.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-12595)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 77369 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12595-xcpng82.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py
test_01_prepare_and_cancel_maintenance Error 0.11 test_ms_maintenance_and_safe_shutdown.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 10.65 test_network.py
test_01_non_strict_host_anti_affinity Error 241.59 test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity Error 121.60 test_nonstrict_affinity_group.py
test_02_create_volume Error 5.30 test_resource_names.py
test_05_scale_vm_dont_allow_disk_offering_change Failure 69.74 test_scale_vm.py

@weizhouapache
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12766

@rohityadavcloud
Copy link
Member

@blueorangutan test matrix

@blueorangutan
Copy link

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-12928)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 115341 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12928-kvm-ol8.zip
Smoke tests completed. 104 look OK, 37 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_events_resource Error 7.85 test_events_resource.py
test_01_events_resource Error 7.85 test_events_resource.py
ContextSuite context=TestAccounts>:setup Error 0.00 test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup Error 0.00 test_accounts.py
test_DeleteDomain Error 14.51 test_accounts.py
test_forceDeleteDomain Failure 14.22 test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup Error 16.10 test_accounts.py
ContextSuite context=TestTemplateHierarchy>:setup Error 1536.22 test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup Error 0.00 test_affinity_groups_projects.py
test_replace_acl_of_network Error 4.63 test_global_acls.py
ContextSuite context=TestAnnotations>:setup Error 0.00 test_annotations.py
ContextSuite context=TestAsyncJob>:setup Error 0.00 test_async_job.py
test_00_deploy_vm_root_resize Error 1.52 test_deploy_vm_root_resize.py
ContextSuite context=TestMultipleVolumeAttach>:setup Error 0.00 test_attach_multiple_volumes.py
test_01_condensed_drs_algorithm Error 1.27 test_cluster_drs.py
test_02_balanced_drs_algorithm Error 0.11 test_cluster_drs.py
ContextSuite context=TestClusterDRS>:teardown Error 1.20 test_cluster_drs.py
ContextSuite context=TestDeployVmWithUserData>:setup Error 0.00 test_deploy_vm_with_userdata.py
ContextSuite context=TestConsoleEndpoint>:setup Error 0.00 test_console_endpoint.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup Error 0.00 test_deploy_virtio_scsi_vm.py
ContextSuite context=TestInternalLb>:setup Error 0.00 test_internal_lb.py
test_02_deploy_vm_with_extraconfig_kvm Error 1.39 test_deploy_vm_extra_config_data.py
test_03_update_vm_with_extraconfig_kvm Error 1.37 test_deploy_vm_extra_config_data.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup Error 0.00 test_deploy_vms_with_varied_deploymentplanners.py
ContextSuite context=TestDeployVMFromISO>:setup Error 0.00 test_deploy_vm_iso.py
ContextSuite context=TestDeployVMsInParallel>:setup Error 0.00 test_deploy_vms_in_parallel.py
ContextSuite context=TestIpv4Routing>:setup Error 0.00 test_ipv4_routing.py
ContextSuite context=TestDeployVMFromISOWithUefi>:setup Error 0.00 test_deploy_vm_iso_uefi.py
ContextSuite context=TestRemoteDiagnostics>:setup Error 0.00 test_diagnostics.py
test_01_deploy_vm_from_direct_download_template_nfs_storage Error 1.38 test_direct_download.py
ContextSuite context=TestDirectDownloadTemplates>:teardown Error 1.10 test_direct_download.py
ContextSuite context=TestDomainsServiceOfferings>:setup Error 1519.80 test_domain_service_offerings.py
test_03_create_vpc_domain_vpc_offering Error 17.55 test_domain_vpc_offerings.py
test_updating_nics_on_two_shared_networks Error 1.57 test_gateway_on_shared_networks.py
ContextSuite context=TestGatewayOnSharedNetwork>:teardown Error 3.73 test_gateway_on_shared_networks.py
ContextSuite context=TestHostControlState>:setup Error 53.35 test_host_control_state.py
test_01_browser_migrate_template Error 65.76 test_image_store_object_migration.py
ContextSuite context=TestImportAndUnmanageVolumes>:setup Error 0.00 test_import_unmanage_volumes.py
test_01_1_create_iso_with_checksum_sha1_negative Error 66.57 test_iso.py
test_01_create_iso_with_checksum_sha1 Error 66.51 test_iso.py
test_01_create_iso_with_checksum_sha1 Error 66.52 test_iso.py
test_02_1_create_iso_with_checksum_sha256_negative Error 66.57 test_iso.py
test_02_create_iso_with_checksum_sha256 Error 66.54 test_iso.py
test_02_create_iso_with_checksum_sha256 Error 66.54 test_iso.py
test_03_1_create_iso_with_checksum_md5_negative Error 66.55 test_iso.py
test_03_create_iso_with_checksum_md5 Error 66.55 test_iso.py
test_03_create_iso_with_checksum_md5 Error 66.55 test_iso.py
test_04_create_iso_with_no_checksum Error 66.53 test_iso.py
test_04_create_iso_with_no_checksum Error 66.54 test_iso.py
test_01_create_iso Failure 1519.01 test_iso.py
ContextSuite context=TestISO>:setup Error 3036.83 test_iso.py
test_01_invalid_upgrade_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster Failure 0.01 test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle Error 0.01 test_kubernetes_clusters.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
test_01_create_template Error 1.16 test_templates.py
test_CreateTemplateWithDuplicateName Error 1.15 test_templates.py
test_02_create_template_with_checksum_sha1 Error 65.64 test_templates.py
ContextSuite context=TestTemplates>:setup Error 295.78 test_templates.py
test_01_snapshot_usage Error 31.01 test_usage.py
test_01_template_usage Error 1.22 test_usage.py
test_01_volume_usage Error 129.77 test_usage.py
test_01_scale_up_verify Error 45.12 test_vm_autoscaling.py
test_02_update_vmprofile_and_vmgroup Failure 245.72 test_vm_autoscaling.py
test_03_scale_down_verify Failure 304.60 test_vm_autoscaling.py
test_04_stop_remove_vm_in_vmgroup Failure 0.03 test_vm_autoscaling.py
test_06_autoscaling_vmgroup_on_project_network Error 52.71 test_vm_autoscaling.py
test_06_autoscaling_vmgroup_on_project_network Error 52.72 test_vm_autoscaling.py
test_07_autoscaling_vmgroup_on_vpc_network Error 7.41 test_vm_autoscaling.py
ContextSuite context=TestVmAutoScaling>:teardown Error 19.54 test_vm_autoscaling.py
test_01_deploy_vm_on_specific_host Error 1.45 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 1.34 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 1.37 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 1.42 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 1.41 test_vm_deployment_planner.py
ContextSuite context=TestDeployVM>:setup Error 0.00 test_vm_life_cycle.py
test_01_migrate_VM_and_root_volume Error 1.45 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 1.42 test_vm_life_cycle.py
test_01_secure_vm_migration Error 78.62 test_vm_life_cycle.py
test_02_unsecure_vm_migration Error 223.99 test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration Error 149.31 test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration Error 150.44 test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup Error 6.70 test_vm_life_cycle.py
ContextSuite context=TestVMSchedule>:setup Error 0.00 test_vm_schedule.py

@harikrishna-patnala
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13132

@weizhouapache
Copy link
Member Author

@blueorangutan test matrix

@blueorangutan
Copy link

@weizhouapache a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13087)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57963 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13087-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_restore_vm_strict_tags_failure Failure 55.52 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 58.73 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 5.72 test_vm_strict_host_tags.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13088)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 87892 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13088-kvm-ubuntu22.zip
Smoke tests completed. 129 look OK, 12 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_nic_secondaryip_add_remove Error 1521.81 test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup Error 0.00 test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup Error 0.00 test_network_acl.py
ContextSuite context=TestIpv6Network>:setup Error 0.00 test_network_ipv6.py
test_delete_account Error 1519.46 test_network.py
test_delete_network_while_vm_on_it Error 1.25 test_network.py
test_deploy_vm_l2network Error 1.24 test_network.py
test_l2network_restart Error 2.43 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.74 test_network.py
ContextSuite context=TestPublicIP>:setup Error 13.74 test_network.py
test_reboot_router Failure 0.10 test_network.py
test_releaseIP Error 6.81 test_network.py
test_releaseIP_using_IP Error 7.17 test_network.py
ContextSuite context=TestRouterRules>:setup Error 7.27 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1522.11 test_network.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
ContextSuite context=TestAdapterTypeForNic>:setup Error 0.00 test_nic_adapter_type.py
test_oobm_multiple_mgmt_server_ownership Failure 31.93 test_outofbandmanagement.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 1531.33 test_projects.py
test_02_restore_vm_strict_tags_failure Failure 54.75 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 59.20 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 3.90 test_vm_strict_host_tags.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13089)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 126197 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13089-vmware-70u3.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_events_resource Error 365.42 test_events_resource.py
test_01_prepare_and_cancel_maintenance Error 0.15 test_ms_maintenance_and_safe_shutdown.py
test_04_deploy_vm_for_other_user_and_test_vm_operations Error 133.80 test_network_permissions.py
test_01_deployVMInSharedNetwork Error 191.51 test_network.py
test_02_restore_vm_with_disk_offering Error 65.79 test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size Error 66.64 test_restore_vm.py
test_01_deploy_vm_on_specific_host Error 14.78 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 3602.22 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 4.60 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 2.50 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 12.86 test_vm_deployment_planner.py
test_01_migrate_vm_strict_tags_success Error 3604.29 test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure Error 8.13 test_vm_strict_host_tags.py
ContextSuite context=TestMigrateVMStrictTags>:teardown Error 10.52 test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success Error 18.15 test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure Error 3603.69 test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success Error 19.29 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Error 3606.32 test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags Error 25.44 test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags Error 3605.99 test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success Error 3.97 test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success Error 7224.27 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 2.77 test_vm_strict_host_tags.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13090)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 128476 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13090-xcpng82.zip
Smoke tests completed. 121 look OK, 20 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_condensed_drs_algorithm Failure 166.36 test_cluster_drs.py
test_02_balanced_drs_algorithm Failure 180.18 test_cluster_drs.py
test_01_prepare_and_cancel_maintenance Error 0.19 test_ms_maintenance_and_safe_shutdown.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 8.76 test_network.py
test_01_non_strict_host_anti_affinity Error 249.44 test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity Error 124.86 test_nonstrict_affinity_group.py
test_CRUD_operations_userdata Error 1523.36 test_register_userdata.py
test_deploy_vm_with_registered_userdata Error 7.74 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_allow Error 7.78 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_append Error 8.06 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_deny Error 7.72 test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_params Error 7.92 test_register_userdata.py
test_link_and_unlink_userdata_to_template Error 8.18 test_register_userdata.py
test_user_userdata_crud Error 7.97 test_register_userdata.py
ContextSuite context=TestResetVmOnReboot>:setup Error 0.00 test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup Error 0.00 test_resource_accounting.py
ContextSuite context=TestResourceNames>:setup Error 0.00 test_resource_names.py
ContextSuite context=TestRestoreVM>:setup Error 0.00 test_restore_vm.py
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestScaleVm>:setup Error 0.00 test_scale_vm.py
ContextSuite context=TestServiceOfferings>:setup Error 1517.85 test_service_offerings.py
ContextSuite context=TestSetSourceNatIp>:setup Error 0.00 test_set_sourcenat.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup Error 0.00 test_snapshots.py
test_02_restore_vm_strict_tags_failure Failure 78.05 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 74.17 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 7.77 test_vm_strict_host_tags.py

@weizhouapache
Copy link
Member Author

@blueorangutan test ol8 vmware-70u3

@blueorangutan
Copy link

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member

@blueorangutan test

@blueorangutan
Copy link

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13125)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 61359 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13125-kvm-ubuntu22.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_oobm_multiple_mgmt_server_ownership Failure 31.91 test_outofbandmanagement.py
test_02_restore_vm_strict_tags_failure Failure 62.99 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 61.13 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 6.04 test_vm_strict_host_tags.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13124)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 78957 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13124-vmware-70u3.zip
Smoke tests completed. 132 look OK, 9 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_events_resource Error 365.82 test_events_resource.py
test_01_events_resource Error 365.83 test_events_resource.py
test_01_prepare_and_cancel_maintenance Error 0.19 test_ms_maintenance_and_safe_shutdown.py
test_04_deploy_vm_for_other_user_and_test_vm_operations Error 124.44 test_network_permissions.py
test_01_deployVMInSharedNetwork Error 161.00 test_network.py
test_02_restore_vm_with_disk_offering Error 69.83 test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size Error 65.66 test_restore_vm.py
test_01_deploy_vm_on_specific_host Error 19.90 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 3602.86 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 4.76 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 1.47 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 4.44 test_vm_deployment_planner.py
test_01_migrate_vm_strict_tags_success Error 9.04 test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure Error 9.00 test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success Error 9.01 test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure Error 9.00 test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success Error 8.95 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Error 5.90 test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags Error 8.90 test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags Error 12.04 test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success Error 8.95 test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success Error 8.96 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 5.83 test_vm_strict_host_tags.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 454.25 test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 454.29 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Error 593.95 test_vpc_redundant.py
test_01_cancel_host_maintenace_with_no_migration_jobs Error 29.75 test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs Error 196.58 test_host_maintenance.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13126)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 85501 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13126-xcpng82.zip
Smoke tests completed. 132 look OK, 9 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_condensed_drs_algorithm Failure 174.03 test_cluster_drs.py
test_02_balanced_drs_algorithm Failure 180.11 test_cluster_drs.py
test_05_ping_in_cpvm_success Failure 15.53 test_diagnostics.py
test_01_prepare_and_cancel_maintenance Error 0.19 test_ms_maintenance_and_safe_shutdown.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 10.60 test_network.py
test_01_non_strict_host_anti_affinity Error 239.32 test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity Error 126.14 test_nonstrict_affinity_group.py
test_02_create_volume Error 2.25 test_resource_names.py
test_05_scale_vm_dont_allow_disk_offering_change Failure 74.16 test_scale_vm.py
test_02_restore_vm_strict_tags_failure Failure 90.42 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 78.23 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 7.82 test_vm_strict_host_tags.py
test_06_download_detached_volume Failure 334.63 test_volumes.py

@blueorangutan
Copy link

[SF] Trillian test result (tid-13134)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54919 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13134-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_restore_vm_strict_tags_failure Failure 54.44 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 58.90 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 3.66 test_vm_strict_host_tags.py

@rohityadavcloud
Copy link
Member

@blueorangutan test

@blueorangutan
Copy link

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13183)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 85278 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13183-kvm-ol8.zip
Smoke tests completed. 129 look OK, 12 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestClusterDRS>:setup Error 0.00 test_cluster_drs.py
test_nic_secondaryip_add_remove Error 1518.51 test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup Error 0.00 test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup Error 0.00 test_network_acl.py
ContextSuite context=TestIpv6Network>:setup Error 0.00 test_network_ipv6.py
test_delete_account Error 1517.68 test_network.py
test_delete_network_while_vm_on_it Error 1.22 test_network.py
test_deploy_vm_l2network Error 1.21 test_network.py
test_l2network_restart Error 2.46 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.74 test_network.py
ContextSuite context=TestPublicIP>:setup Error 11.69 test_network.py
test_reboot_router Failure 0.08 test_network.py
test_releaseIP Error 5.63 test_network.py
test_releaseIP_using_IP Error 6.11 test_network.py
ContextSuite context=TestRouterRules>:setup Error 6.20 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1521.97 test_network.py
ContextSuite context=TestAdapterTypeForNic>:setup Error 0.00 test_nic_adapter_type.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 1529.00 test_projects.py
test_02_restore_vm_strict_tags_failure Failure 56.33 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 59.56 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 5.63 test_vm_strict_host_tags.py

@weizhouapache
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13233

@weizhouapache
Copy link
Member Author

@blueorangutan test

@blueorangutan
Copy link

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13190)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 56893 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t13190-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_restore_vm_strict_tags_failure Failure 55.40 test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure Failure 58.68 test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure Failure 5.67 test_vm_strict_host_tags.py

@DaanHoogland DaanHoogland merged commit fd74895 into apache:main May 2, 2025
24 of 26 checks passed
@DaanHoogland DaanHoogland deleted the 4.21-reconcile-commands branch May 2, 2025 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rohityadavcloud rohityadavcloud rohityadavcloud approved these changes

@harikrishna-patnala harikrishna-patnala harikrishna-patnala approved these changes

@vladimirpetrov vladimirpetrov vladimirpetrov approved these changes

@JoaoJandre JoaoJandre Awaiting requested review from JoaoJandre

@shwstppr shwstppr Awaiting requested review from shwstppr

@sureshanaparti sureshanaparti Awaiting requested review from sureshanaparti

Assignees

@harikrishna-patnala harikrishna-patnala

@vladimirpetrov vladimirpetrov

Labels
component:agent component:api component:orchestration component:storage
Projects
None yet
Milestone
4.21.0
Development

Successfully merging this pull request may close these issues.
6 participants
@weizhouapache @blueorangutan @rohityadavcloud @harikrishna-patnala @vladimirpetrov @DaanHoogland