Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Migrate public templates that have URLs on data migration across secondary storages #10364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 15, 2025
Merged

Migrate public templates that have URLs on data migration across secondary storages #10364

merged 3 commits into from
Apr 15, 2025

Conversation

winterhazel
Copy link
Member

Description

Currently, data migration across secondary storages does not migrate public templates that have URLs (only private templates and public templates that do not have URLs are migrated) because, when this feature was implemented, it was assumed that these templates would automatically be downloaded when to newly added secondary storages. However, that's not always the case, as URLs can become inaccessible (unable to download the template), and their templates can be updated (unable to install due to checksum mismatch). This forces operators to manually move the templates inside the storage.

This PR solves this issue by allowing the migration of public templates that have URLs.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

How Has This Been Tested?

In an environment with the following templates:

  • Public with URL: a, b, c
  • Public without URL: t
  • Private with URL: System VM Template (KVM), z

And two secondary storages:

  • nfs-1: has all the templates
  • nfs-2: has b, c (all public templates except a)

I did the following tests:

  1. I restarted the SSVM. Then, I verified that it began to download templates a and System VM Template (KVM) to nfs-2

  2. While the templates were downloading, I called migrateSecondaryStorageData to migrate the data from nfs-1 to nfs-2

  3. After the migration ended, I verified that all templates in nfs-1 were moved to nfs-2. I also verified that there were no duplicate template entries, and that the install_path of the new entries matched the template's path in nfs-2

@winterhazel
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@codecov Codecov
Copy link

codecov bot commented Feb 11, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 64.17910% with 24 lines in your changes missing coverage. Please review.

Project coverage is 16.02%. Comparing base (16b7b71) to head (5ff9e48).
Report is 26 commits behind head on 4.20.

Files with missing lines Patch % Lines
...ack/storage/image/SecondaryStorageServiceImpl.java 56.86% 20 Missing and 2 partials ⚠️
...ack/engine/orchestration/DataMigrationUtility.java 92.85% 1 Missing ⚠️
...om/cloud/storage/download/DownloadActiveState.java 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff            @@
##               4.20   #10364   +/-   ##
=========================================
  Coverage     16.01%   16.02%           
- Complexity    13115    13125   +10     
=========================================
  Files          5651     5651           
  Lines        495863   495898   +35     
  Branches      60049    60051    +2     
=========================================
+ Hits          79405    79448   +43     
+ Misses       407595   407586    -9     
- Partials       8863     8864    +1
Flag Coverage Δ
uitests 4.00% <ø> (ø)
unittests 16.86% <64.17%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12402

@winterhazel winterhazel force-pushed the copy-public-templates-on-data-migration branch from dd9a443 to 55eaba0 Compare February 26, 2025 17:57
@winterhazel winterhazel changed the base branch from main to 4.20 February 26, 2025 17:57
@winterhazel
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12587

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12597

@winterhazel
Copy link
Member Author

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12885

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12893

@DaanHoogland
Copy link
Contributor

@winterhazel , it seems something in https://github.com/apache/cloudstack/pull/10364/files#diff-70d3c567cc7f97b4ac1a9e41f86723ba74291feacc95c0a2c0a6623458eaa92d causes timeouts .

17:59:14 [INFO] Running org.apache.cloudstack.storage.image.SecondaryStorageServiceImplTest
18:29:14 Build timed out (after 30 minutes). Marking the build as failed.
18:29:14 New run name is '#14896-10364'
18:29:14 Build was aborted

@winterhazel
Copy link
Member Author

@winterhazel , it seems something in https://github.com/apache/cloudstack/pull/10364/files#diff-70d3c567cc7f97b4ac1a9e41f86723ba74291feacc95c0a2c0a6623458eaa92d causes timeouts .

17:59:14 [INFO] Running org.apache.cloudstack.storage.image.SecondaryStorageServiceImplTest
18:29:14 Build timed out (after 30 minutes). Marking the build as failed.
18:29:14 New run name is '#14896-10364'
18:29:14 Build was aborted

Will take a look.

@winterhazel
Copy link
Member Author

@winterhazel , it seems something in https://github.com/apache/cloudstack/pull/10364/files#diff-70d3c567cc7f97b4ac1a9e41f86723ba74291feacc95c0a2c0a6623458eaa92d causes timeouts .

17:59:14 [INFO] Running org.apache.cloudstack.storage.image.SecondaryStorageServiceImplTest
18:29:14 Build timed out (after 30 minutes). Marking the build as failed.
18:29:14 New run name is '#14896-10364'
18:29:14 Build was aborted

Should be fixed now :)

@blueorangutan package

@blueorangutan
Copy link

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12902

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-12834)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53309 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10364-t12834-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

DaanHoogland
DaanHoogland approved these changes Mar 28, 2025
View reviewed changes
Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@DaanHoogland DaanHoogland added this to the 4.20.1 milestone Mar 28, 2025
@borisstoyanov borisstoyanov self-assigned this Apr 15, 2025
borisstoyanov
borisstoyanov approved these changes Apr 15, 2025
View reviewed changes
Copy link
Contributor

@borisstoyanov borisstoyanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, manually checked it

@borisstoyanov borisstoyanov removed their assignment Apr 15, 2025
@DaanHoogland DaanHoogland merged commit ac6b1b3 into apache:4.20 Apr 15, 2025
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DaanHoogland DaanHoogland DaanHoogland approved these changes

@borisstoyanov borisstoyanov borisstoyanov approved these changes

Assignees
No one assigned
Labels
component:api component:orchestration component:storage
Projects
None yet
Milestone
4.20.1
Development

Successfully merging this pull request may close these issues.
4 participants
@winterhazel @blueorangutan @DaanHoogland @borisstoyanov