[WIP] Remove allocated snapshots / vm snapshots on start #8452

sureshanaparti · 2024-01-05T07:39:15Z

Description

Some latest active snapshots / vm snapshots are stuck in allocated state when MS is stopped, these are listed / shown in UI as well (not allowed to delete). Remove them on MS start itself.

This PR removes allocated snapshots / vm snapshots on start.

Fixes #8424

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested: Take VM Snapshot -> Stop MS -> Start MS

VM Snapshot record after MS stopped =>

                 id: 1
               uuid: 07542708-27b7-47c4-959a-f7f92828f43a
               name: i-2-3-VM_VS_20240108075812
       display_name: testvm01-snap
        description: NULL
              vm_id: 3
         account_id: 2
          domain_id: 1
service_offering_id: 1
   vm_snapshot_type: DiskAndMemory
              state: Allocated
             parent: NULL
            current: NULL
       update_count: 0
            updated: NULL
            created: 2024-01-08 07:58:12
            removed: NULL

VM Snapshot record after MS started (Not listed / shown in the UI) =>

                 id: 1
               uuid: 07542708-27b7-47c4-959a-f7f92828f43a
               name: i-2-3-VM_VS_20240108075812
       display_name: testvm01-snap
        description: NULL
              vm_id: 3
         account_id: 2
          domain_id: 1
service_offering_id: 1
   vm_snapshot_type: DiskAndMemory
              state: Allocated
             parent: NULL
            current: NULL
       update_count: 0
            updated: NULL
            created: 2024-01-08 07:58:12
            removed: 2024-01-08 07:59:26

How did you try to break this feature and the system with this change?

sureshanaparti · 2024-01-05T07:40:10Z

@blueorangutan package

blueorangutan · 2024-01-05T07:48:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-01-05T07:48:16Z

Codecov Report

Attention: Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.

Project coverage is 22.99%. Comparing base (6d916ca) to head (a00734b).
Report is 627 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	25.00%	2 Missing and 1 partial ⚠️
...a/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java	25.00%	2 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (6d916ca) and HEAD (a00734b). Click for more details.

HEAD has 2 uploads less than BASE

Flag BASE (6d916ca) HEAD (a00734b)

simulator-marvin-tests 17 16

unit-tests 1 0

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8452      +/-   ##
============================================
- Coverage     30.85%   22.99%   -7.86%     
+ Complexity    34048    23743   -10305     
============================================
  Files          5341     5209     -132     
  Lines        374861   352833   -22028     
  Branches      54518    50531    -3987     
============================================
- Hits         115659    81140   -34519     
- Misses       243973   260029   +16056     
+ Partials      15229    11664    -3565

Flag	Coverage Δ
simulator-marvin-tests	`24.63% <45.45%> (-0.12%)`	⬇️
uitests	`4.39% <ø> (+<0.01%)`	⬆️
unit-tests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

weizhouapache · 2024-01-05T08:10:59Z

@sureshanaparti
would it be better to clean the snapshots by storage garbage collector which run periodically ?

blueorangutan · 2024-01-05T08:51:40Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8207

sureshanaparti · 2024-01-08T05:35:56Z

@sureshanaparti would it be better to clean the snapshots by storage garbage collector which run periodically ?

@weizhouapache issue is some active snapshots / vm snapshots are left in allocated when MS is stopped, no point in keeping them and wait until storage garbage collector cleans up (what if storage cleanup is disabled?) as these are listed / shown in UI as well (which are not allowed to delete). So, I think, it's better remove them on start itself. Any other thoughts / suggestions?

weizhouapache · 2024-01-08T08:07:08Z

@sureshanaparti would it be better to clean the snapshots by storage garbage collector which run periodically ?

@weizhouapache issue is some active snapshots / vm snapshots are left in allocated when MS is stopped, no point in keeping them and wait until storage garbage collector cleans up (what if storage cleanup is disabled?) as these are listed / shown in UI as well (which are not allowed to delete). So, I think, it's better remove them on start itself. Any other thoughts / suggestions?

@sureshanaparti
the storage garbage collector cleans up the primary and secondary storage, including the snapshot/volumes/templates

                    // remove snapshots in Error state
                    List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Error);

We could add the cleanup of Allocated snapshot and Allocated/Error vm snapshots in the process

IMHO, it is not the right place to cleanup during mgmt server start, as mgmt server might be running for several days.

sureshanaparti · 2024-01-08T08:24:24Z

@sureshanaparti would it be better to clean the snapshots by storage garbage collector which run periodically ?

@weizhouapache issue is some active snapshots / vm snapshots are left in allocated when MS is stopped, no point in keeping them and wait until storage garbage collector cleans up (what if storage cleanup is disabled?) as these are listed / shown in UI as well (which are not allowed to delete). So, I think, it's better remove them on start itself. Any other thoughts / suggestions?

@sureshanaparti the storage garbage collector cleans up the primary and secondary storage, including the snapshot/volumes/templates
                    // remove snapshots in Error state
                    List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Error);
We could add the cleanup of Allocated snapshot and Allocated/Error vm snapshots in the process

IMHO, it is not the right place to cleanup during mgmt server start, as mgmt server might be running for several days.

@weizhouapache Agreed for the resources in error state (as there might be things to reset / cleanup) and MS is running. snapshot / vm snapshots are in Allocated, mostly for the very recent active snapshot / vm snapshots before MS is stopped, so no need for any cleanup, better to set them as removed (Any event through state transition would still keep it in Allocated). Otherwise, at least have to not list them on UI, or allow explicit deletion from the UI.

Snapshots in destroying state are deleted here. So, I thought, it's better to remove there on MS start.

cloudstack/server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

Lines 1519 to 1523 in 3f9dd4d

    
           //destroy snapshots in destroying state 
        
           List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Destroying); 
        
           for (SnapshotVO snapshotVO : snapshots) { 
        
               try { 
        
                   if (!deleteSnapshot(snapshotVO.getId(), null)) {

weizhouapache · 2024-01-08T08:54:45Z

@weizhouapache issue is some active snapshots / vm snapshots are left in allocated when MS is stopped, no point in keeping them and wait until storage garbage collector cleans up (what if storage cleanup is disabled?) as these are listed / shown in UI as well (which are not allowed to delete). So, I think, it's better remove them on start itself. Any other thoughts / suggestions?

@sureshanaparti the storage garbage collector cleans up the primary and secondary storage, including the snapshot/volumes/templates
                    // remove snapshots in Error state
                    List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Error);
We could add the cleanup of Allocated snapshot and Allocated/Error vm snapshots in the process
IMHO, it is not the right place to cleanup during mgmt server start, as mgmt server might be running for several days.
@weizhouapache Agreed for the resources in error state (as there might be things to reset / cleanup) and MS is running. snapshot / vm snapshots are in Allocated, mostly for the very recent active snapshot / vm snapshots before MS is stopped, so no need for any cleanup, better to set them as removed (Any event through state transition would still keep it in Allocated). Otherwise, at least have to not list them on UI, or allow explicit deletion from the UI.

Snapshots in destroying state are deleted here. So, I thought, it's better to remove there on MS start.

cloudstack/server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

Lines 1519 to 1523 in 3f9dd4d

//destroy snapshots in destroying state

List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Destroying);

for (SnapshotVO snapshotVO : snapshots) {

try {

if (!deleteSnapshot(snapshotVO.getId(), null)) {

@sureshanaparti
the mgmt service is a key service for some users, especially on production.
the service is running for several days without stopping.
IMHO it is not good to cleanup resource during start ...

(OK with cleaning up resource in both start process and garbage collector)

sureshanaparti · 2024-01-08T10:15:37Z

@weizhouapache issue is some active snapshots / vm snapshots are left in allocated when MS is stopped, no point in keeping them and wait until storage garbage collector cleans up (what if storage cleanup is disabled?) as these are listed / shown in UI as well (which are not allowed to delete). So, I think, it's better remove them on start itself. Any other thoughts / suggestions?

@sureshanaparti the storage garbage collector cleans up the primary and secondary storage, including the snapshot/volumes/templates
                    // remove snapshots in Error state
                    List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Error);
We could add the cleanup of Allocated snapshot and Allocated/Error vm snapshots in the process
IMHO, it is not the right place to cleanup during mgmt server start, as mgmt server might be running for several days.
@weizhouapache Agreed for the resources in error state (as there might be things to reset / cleanup) and MS is running. snapshot / vm snapshots are in Allocated, mostly for the very recent active snapshot / vm snapshots before MS is stopped, so no need for any cleanup, better to set them as removed (Any event through state transition would still keep it in Allocated). Otherwise, at least have to not list them on UI, or allow explicit deletion from the UI.
Snapshots in destroying state are deleted here. So, I thought, it's better to remove there on MS start.

cloudstack/server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

Lines 1519 to 1523 in 3f9dd4d

//destroy snapshots in destroying state

List<SnapshotVO> snapshots = _snapshotDao.listAllByStatus(Snapshot.State.Destroying);

for (SnapshotVO snapshotVO : snapshots) {

try {

if (!deleteSnapshot(snapshotVO.getId(), null)) {
@sureshanaparti the mgmt service is a key service for some users, especially on production. the service is running for several days without stopping. IMHO it is not good to cleanup resource during start ...

(OK with cleaning up resource in both start process and garbage collector)

ok, will check & update. thanks @weizhouapache

github-actions · 2024-02-08T13:19:08Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-06-18T08:42:06Z

@sureshanaparti , does this need rebasing?

DaanHoogland · 2024-06-18T08:45:27Z

on another note, is it maybe safe to first mark it as State == Error and then delete? (I think this is the same @weizhouapache suggests)

DaanHoogland · 2025-02-03T14:55:53Z

@sureshanaparti , any progress on this?

DaanHoogland · 2025-02-04T14:13:56Z

@blueorangutan package

blueorangutan · 2025-02-04T14:16:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-04T14:52:47Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 12329

rohityadavcloud · 2025-02-28T04:47:19Z

@sureshanaparti are you working on this?

shwstppr

@sureshanaparti what is the status for this? Do we need more changes or it needs testing?

shwstppr · 2025-04-15T06:24:13Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

+        //remove snapshots in allocated state
+        List<SnapshotVO> allocatedSnapshots = _snapshotDao.listAllByStatus(Snapshot.State.Allocated);
+        for (SnapshotVO snapshot : allocatedSnapshots) {
+            _snapshotDao.remove(snapshot.getId());
+        }


this can be a single call to DB - removeAllByStatus?

shwstppr · 2025-04-15T06:24:20Z

server/src/main/java/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java

+        //Remove VM Snapshots in allocated state
+        List<VMSnapshotVO> allocatedVMSnapshots = _vmSnapshotDao.listAllByStatus(VMSnapshot.State.Allocated);
+        for (VMSnapshotVO vmSnapshot : allocatedVMSnapshots) {
+            _vmSnapshotDao.remove(vmSnapshot.getId());
+        }


this can be a single call to DB - removeAllByStatus?

Remove allocated snapshots / vm snapshots on start

a00734b

boring-cyborg bot added the component:storage label Jan 5, 2024

sureshanaparti marked this pull request as ready for review January 8, 2024 08:08

sureshanaparti marked this pull request as draft January 8, 2024 10:16

DaanHoogland added this to the 4.19.1.0 milestone Jan 22, 2024

github-actions bot added the status:has-conflicts label Feb 8, 2024

sureshanaparti self-assigned this Mar 12, 2024

sureshanaparti added the status:waiting-for-author label Jun 23, 2024

sureshanaparti changed the title ~~Remove allocated snapshots / vm snapshots on start~~ [WIP] Remove allocated snapshots / vm snapshots on start Jun 25, 2024

sureshanaparti modified the milestones: 4.19.1.0, 4.19.2.0 Jun 25, 2024

DaanHoogland changed the base branch from main to 4.19 February 4, 2025 14:12

github-actions bot removed the status:has-conflicts label Feb 5, 2025

DaanHoogland modified the milestones: 4.19.2, 4.19.3 Feb 7, 2025

DaanHoogland added the Severity:Major label Mar 5, 2025

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Mar 17, 2025

shwstppr reviewed Apr 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Remove allocated snapshots / vm snapshots on start #8452

[WIP] Remove allocated snapshots / vm snapshots on start #8452

sureshanaparti commented Jan 5, 2024 •

edited

Loading

sureshanaparti commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

codecov bot commented Jan 5, 2024 •

edited

Loading

weizhouapache commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

sureshanaparti commented Jan 8, 2024

weizhouapache commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024 •

edited

Loading

weizhouapache commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024

github-actions bot commented Feb 8, 2024

DaanHoogland commented Jun 18, 2024

DaanHoogland commented Jun 18, 2024

DaanHoogland commented Feb 3, 2025

DaanHoogland commented Feb 4, 2025

blueorangutan commented Feb 4, 2025

blueorangutan commented Feb 4, 2025

rohityadavcloud commented Feb 28, 2025

shwstppr left a comment

shwstppr Apr 15, 2025

shwstppr Apr 15, 2025

[WIP] Remove allocated snapshots / vm snapshots on start #8452

Are you sure you want to change the base?

[WIP] Remove allocated snapshots / vm snapshots on start #8452

Conversation

sureshanaparti commented Jan 5, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sureshanaparti commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

codecov bot commented Jan 5, 2024 • edited Loading

Codecov Report

weizhouapache commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

sureshanaparti commented Jan 8, 2024

weizhouapache commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024 • edited Loading

weizhouapache commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024

github-actions bot commented Feb 8, 2024

DaanHoogland commented Jun 18, 2024

DaanHoogland commented Jun 18, 2024

DaanHoogland commented Feb 3, 2025

DaanHoogland commented Feb 4, 2025

blueorangutan commented Feb 4, 2025

blueorangutan commented Feb 4, 2025

rohityadavcloud commented Feb 28, 2025

shwstppr left a comment

Choose a reason for hiding this comment

shwstppr Apr 15, 2025

Choose a reason for hiding this comment

shwstppr Apr 15, 2025

Choose a reason for hiding this comment

sureshanaparti commented Jan 5, 2024 •

edited

Loading

codecov bot commented Jan 5, 2024 •

edited

Loading

sureshanaparti commented Jan 8, 2024 •

edited

Loading