ssvm: delete temp directory while deleting entity download url#12562
ssvm: delete temp directory while deleting entity download url#12562shwstppr wants to merge 1 commit intoapache:4.20from
Conversation
url When expired entity URLs are cleaned up, orphan directories are left behind. This change tries to delete the base temporary directory while cleanup. Signed-off-by: Abhishek Kumar <[email protected]>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.20 #12562 +/- ##
=========================================
Coverage 16.26% 16.26%
- Complexity 13426 13432 +6
=========================================
Files 5660 5660
Lines 499949 499982 +33
Branches 60704 60712 +8
=========================================
+ Hits 81296 81332 +36
+ Misses 409581 409572 -9
- Partials 9072 9078 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@blueorangutan package |
|
@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16663 |
|
@blueorangutan test |
|
@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-15346)
|
RosiKyu
left a comment
There was a problem hiding this comment.
I have identified an issue in the PR:
Test Steps
- Configure short expiration intervals:
extract.url.expiration.interval= 120 secondsextract.url.cleanup.interval= 60 seconds
- Restart management server for config changes to take effect
- Create and attach a DATA volume to a VM
- Stop the VM
- Extract the volume to generate a download URL
- Verify UUID directory is created in
/var/www/html/userdata/ - Wait for URL expiration and cleanup (~3 minutes)
- Check if symlink and UUID directory are deleted
Expected Result:
- The symlink
/var/www/html/userdata/<UUID>/test-vol.qcow2should be deleted - The UUID directory
/var/www/html/userdata/<UUID>/should be deleted - SSVM logs should show: "Deleting symlink root directory: for "
Actual Result:
- FAIL - The unlink command failed due to incorrect path construction
- The unlink tried
/var/www/html/userdata/test-vol.qcow2instead of/var/www/html/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2 - Since unlink failed, the
deleteEntitySymlinkRootPathIfNeededmethod never executed - Both the symlink AND UUID directory remain orphaned
Root Cause:
There seems to be an issue in the path extraction logic:
String linkPath = extractUrl.substring(extractUrl.lastIndexOf(File.separator) + 1);This extracts only the filename (test-vol.qcow2) instead of the full relative path (7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2).
Test Evidence:
- Configuration changes:
(localcloud) 🐱 > update configuration name=extract.url.expiration.interval value=120
{
"configuration": {
"category": "Advanced",
"component": "management-server",
"defaultvalue": "14400",
"description": "The life of an extract URL after which it is deleted ",
"displaytext": "Extract url expiration interval",
"group": "Management Server",
"isdynamic": false,
"name": "extract.url.expiration.interval",
"subgroup": "Limits",
"type": "Number",
"value": "120"
}
}
(localcloud) 🐱 > update configuration name=extract.url.cleanup.interval value=60
{
"configuration": {
"category": "Advanced",
"component": "management-server",
"defaultvalue": "7200",
"description": "The interval (in seconds) to wait before cleaning up the extract URL's ",
"displaytext": "Extract url cleanup interval",
"group": "Management Server",
"isdynamic": false,
"name": "extract.url.cleanup.interval",
"subgroup": "Limits",
"type": "Number",
"value": "60"
}
}
- Volume creation and attachment:
(localcloud) 🐱 > attach volume id=baff5910-2d21-4542-b2ee-6ee34cc6be73 virtualmachineid=90e7a447-1df2-4e75-8afc-ba807548ca42
{
"volume": {
"account": "admin",
"attached": "2026-02-04T11:45:14+0000",
"clusterid": "6f535ef8-be7f-4a5a-b87c-c589d427341b",
"clustername": "p1-c1",
"created": "2026-02-04T11:44:36+0000",
"deleteprotection": false,
"destroyed": false,
"deviceid": 1,
"diskioread": 0,
"diskiowrite": 0,
"diskkbsread": 0,
"diskkbswrite": 0,
"diskofferingdisplaytext": "Small Disk, 5 GB",
"diskofferingid": "4391aa82-7d9b-4e3c-9f16-9a4506f58c68",
"diskofferingname": "Small",
"displayvolume": true,
"domain": "ROOT",
"domainid": "ae560d02-01b6-11f1-9894-1e003100046f",
"domainpath": "/",
"hasannotations": false,
"hypervisor": "KVM",
"id": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"isextractable": true,
"jobid": "1c3936a5-d499-45de-a2d7-0495df953d8e",
"jobstatus": 0,
"name": "test-vol",
"path": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"podid": "09b0e9f6-5957-4f76-95ed-00764e23ed02",
"podname": "Pod1",
"provisioningtype": "thin",
"quiescevm": false,
"size": 5368709120,
"state": "Ready",
"storage": "ref-trl-10829-k-Mol9-rositsa-kyuchukova-kvm-pri2",
"storageid": "234e7aff-ac28-3150-8891-497563810b52",
"storagetype": "shared",
"supportsstoragesnapshot": false,
"tags": [],
"type": "DATADISK",
"virtualmachineid": "90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmdisplayname": "VM-90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmname": "VM-90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmstate": "Running",
"vmtype": "User",
"zoneid": "7bc0e169-5133-42ba-8e5f-ea6f5db39ee8",
"zonename": "ref-trl-10829-k-Mol9-rositsa-kyuchukova"
}
}
- Volume extraction:
(localcloud) 🐱 > extract volume id=baff5910-2d21-4542-b2ee-6ee34cc6be73 mode=HTTP_DOWNLOAD zoneid=7bc0e169-5133-42ba-8e5f-ea6f5db39ee8
{
"volume": {
"accountid": "ec2bc4c5-01b6-11f1-9894-1e003100046f",
"extractMode": "HTTP_DOWNLOAD",
"id": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"name": "test-vol",
"state": "DOWNLOAD_URL_CREATED",
"url": "http://10.0.55.122/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2",
"zoneid": "7bc0e169-5133-42ba-8e5f-ea6f5db39ee8",
"zonename": "ref-trl-10829-k-Mol9-rositsa-kyuchukova"
}
}
- UUID directory and symlink created:
root@s-3-VM:~# ls -la /var/www/html/userdata/
total 16
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 5 www-data www-data 4096 Feb 4 11:05 ..
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 7357f278-a8be-4c93-a43a-4679f6f220ab
-rwxr-xr-x 1 www-data www-data 17 Jan 31 10:47 .htaccess
root@s-3-VM:~# ls -la /var/www/html/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/
total 12
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 ..
lrwxrwxrwx 1 root root 107 Feb 4 11:52 test-vol.qcow2 -> /mnt/SecStorage/af60251b-b7ae-3132-a37a-9ba1f92e3118/volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2
- Management Server log showing correct extractUrl sent:
[root@ref-trl-10829-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# grep -i "DeleteEntityDownloadURL" /var/log/cloudstack/management/management-server.log | tail -20
2026-02-04 11:55:06,720 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StorageManager-Scavenger-1:[]) (logid:) getCommandHostDelegation: class com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand
2026-02-04 11:55:06,721 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StorageManager-Scavenger-1:[]) (logid:) Wait time setting on com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand is 1800 seconds
2026-02-04 11:55:06,722 DEBUG [c.c.a.t.Request] (StorageManager-Scavenger-1:[]) (logid:) Seq 4-4037477065937649683: Sending { Cmd , MgmtId: 32986170917999, via: 4(s-3-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand":{"path":"volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2","extractUrl":"http://10.0.55.122/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2","type":"VOLUME","parentPath":"af60251b-b7ae-3132-a37a-9ba1f92e3118","accountId":"0","wait":"0","bypassHostMaintenance":"false"}}] }
- SSVM log showing unlink failure due to incorrect path:
root@s-3-VM:~# grep -A5 -B5 "handleDeleteEntityDownloadURLCommand" /var/log/cloud.log
2026-02-04T11:55:06,734 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) handleDeleteEntityDownloadURLCommand Path:volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2 Type:VOLUME
2026-02-04T11:55:06,756 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Execution of process [4146] for command [/bin/bash -c unlink /var/www/html/userdata/test-vol.qcow2 ] failed.
2026-02-04T11:55:06,757 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Process [4146] for command [/bin/bash -c unlink /var/www/html/userdata/test-vol.qcow2 ] encountered the error: [unlink: cannot unlink '/var/www/html/userdata/test-vol.qcow2': No such file or directory].
2026-02-04T11:55:06,757 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Error in deleting symlink :unlink: cannot unlink '/var/www/html/userdata/test-vol.qcow2': No such file or directory
- After cleanup interval - UUID directory still exists (orphaned):
root@s-3-VM:~# ls -la /var/www/html/userdata/
total 16
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 5 www-data www-data 4096 Feb 4 11:05 ..
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 7357f278-a8be-4c93-a43a-4679f6f220ab
-rwxr-xr-x 1 www-data www-data 17 Jan 31 10:47 .htaccess
- Verified PR code is deployed (method exists in JAR):
root@s-3-VM:/usr/local/cloud/systemvm# unzip -p cloud-secondary-storage-4.20.3.0-SNAPSHOT.jar | grep -a "deleteEntitySymlinkRootPathIfNeeded"
extractUrlkl#deleteEntitySymlinkRootPathIfNeededQ(Lcom/cloud/agent/api/storage/DeleteEntityDownloadURLCommand;Ljava/lang/String;)V...
root@s-3-VM:/usr/local/cloud/systemvm# unzip -p cloud-secondary-storage-4.20.3.0-SNAPSHOT.jar | grep -a "Deleting symlink root directory"
...Deleting symlink root directory: {} for {}...
The path extraction logic needs to be fixed. The code should extract everything after /userdata/ from the extractUrl, not just the filename after the last /.
|
Thanks @RosiKyu . I was maybe working with main branch env for testing and didn't face the problem. |
There was a problem hiding this comment.
Pull request overview
This pull request aims to fix a bug where orphan temporary directories are left behind when expired entity download URLs are cleaned up in the SSVM (Secondary Storage VM). The fix adds logic to delete the UUID-named root directory that contained the symlink after the symlink itself is removed.
Changes:
- Added a new
deleteRecursivelymethod toFileUtil.javafor recursive directory deletion using the modern NIO.2 Path API - Modified
UploadManagerImpl.javato call a new cleanup method after symlink deletion - Added unit tests to verify the UUID directory cleanup logic works correctly
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| utils/src/main/java/com/cloud/utils/FileUtil.java | Adds new deleteRecursively method to support recursive deletion of directories and their contents |
| services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/template/UploadManagerImpl.java | Introduces BASE_EXTRACT_DIR constant and deleteEntitySymlinkRootPathIfNeeded method to clean up orphan UUID directories after symlink deletion |
| services/secondary-storage/server/src/test/java/org/apache/cloudstack/storage/template/UploadManagerImplTest.java | Adds unit tests for the new cleanup logic covering empty paths, invalid UUIDs, and valid UUID scenarios |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| String linkPath = extractUrl.substring(extractUrl.lastIndexOf(File.separator) + 1); | ||
| command.add("unlink " + BASE_EXTRACT_DIR + linkPath); | ||
| result = command.execute(); | ||
| if (result != null) { | ||
| // FIXME - Ideally should bail out if you can't delete symlink. Not doing it right now. | ||
| // This is because the ssvm might already be destroyed and the symlinks do not exist. | ||
| logger.warn("Error in deleting symlink :" + result); | ||
| } else { | ||
| deleteEntitySymlinkRootPathIfNeeded(cmd, linkPath); |
There was a problem hiding this comment.
The linkPath extraction logic is incorrect. Given extractUrl format http://host/userdata/UUID/filename, using lastIndexOf(File.separator) will find the last "/" before the filename, resulting in linkPath being just "filename" instead of "UUID/filename". This breaks the deleteEntitySymlinkRootPathIfNeeded method which expects linkPath to contain the UUID directory name as the first path component. The code should extract everything after "/userdata/" using extractUrl.substring(extractUrl.indexOf("/userdata/") + "/userdata/".length()) instead, with proper validation that "/userdata/" exists in the URL.
| String linkPath = extractUrl.substring(extractUrl.lastIndexOf(File.separator) + 1); | |
| command.add("unlink " + BASE_EXTRACT_DIR + linkPath); | |
| result = command.execute(); | |
| if (result != null) { | |
| // FIXME - Ideally should bail out if you can't delete symlink. Not doing it right now. | |
| // This is because the ssvm might already be destroyed and the symlinks do not exist. | |
| logger.warn("Error in deleting symlink :" + result); | |
| } else { | |
| deleteEntitySymlinkRootPathIfNeeded(cmd, linkPath); | |
| final String userDataSegment = "/userdata/"; | |
| int userDataIndex = extractUrl.indexOf(userDataSegment); | |
| if (userDataIndex == -1) { | |
| logger.warn("Extract URL does not contain expected segment '/userdata/': " + extractUrl); | |
| } else { | |
| String linkPath = extractUrl.substring(userDataIndex + userDataSegment.length()); | |
| if (StringUtils.isBlank(linkPath)) { | |
| logger.warn("Extract URL does not contain a valid link path after '/userdata/': " + extractUrl); | |
| } else { | |
| command.add("unlink " + BASE_EXTRACT_DIR + linkPath); | |
| result = command.execute(); | |
| if (result != null) { | |
| // FIXME - Ideally should bail out if you can't delete symlink. Not doing it right now. | |
| // This is because the ssvm might already be destroyed and the symlinks do not exist. | |
| logger.warn("Error in deleting symlink :" + result); | |
| } else { | |
| deleteEntitySymlinkRootPathIfNeeded(cmd, linkPath); | |
| } | |
| } |
|
|
||
| public static boolean deleteRecursively(Path path) throws IOException { | ||
| LOGGER.debug("Deleting path: {}", path); | ||
| if (Files.isDirectory(path)) { |
There was a problem hiding this comment.
The deleteRecursively method may behave unexpectedly when encountering symbolic links to directories. Files.isDirectory(path) follows symlinks by default, which means if a symlink points to a directory, this method will recursively delete the contents of the target directory instead of just removing the symlink. Consider using Files.isDirectory(path, LinkOption.NOFOLLOW_LINKS) to treat symlinks as files and delete them directly without following them.
| @Test | ||
| public void deletesSymlinkRootDirectoryWhenValidUuid() { | ||
| String validLinkPath = "123e4567-e89b-12d3-a456-426614174000/file"; | ||
| uploadManager.deleteEntitySymlinkRootPathIfNeeded(mock(DeleteEntityDownloadURLCommand.class), validLinkPath); | ||
| fileUtilMock.verify(() -> FileUtil.deleteRecursively(any(Path.class)), times(1)); | ||
| } |
There was a problem hiding this comment.
Consider adding a test case for linkPath that contains only a UUID without a filename (e.g., "123e4567-e89b-12d3-a456-426614174000"). While this may not occur in normal operation given the URL format, it's a valid edge case that should be handled correctly.
| return; | ||
| } | ||
| logger.info("Deleting symlink root directory: {} for {}", rootDir, cmd.getExtractUrl()); | ||
| Path rootDirPath = Path.of(BASE_EXTRACT_DIR + rootDir); |
There was a problem hiding this comment.
Consider using Path.of(BASE_EXTRACT_DIR, rootDir) instead of string concatenation for path construction. This is more robust and handles path separators correctly across different operating systems, even if BASE_EXTRACT_DIR is modified in the future to not include a trailing separator.
| Path rootDirPath = Path.of(BASE_EXTRACT_DIR + rootDir); | |
| Path rootDirPath = Path.of(BASE_EXTRACT_DIR, rootDir); |
Description
When expired entity URLs are cleaned up, orphan directories are left behind. This change tries to delete the base temporary directory while cleanup.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Download link for a volume expires and server calls DeleteEntityDownloadURLCommand
Before
After
How did you try to break this feature and the system with this change?