fix: clean up empty namespace folders on storage after drop - #1858
fix: clean up empty namespace folders on storage after drop#1858dadavidtseng wants to merge 15 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds best-effort deletion of empty namespace storage folders when a namespace is dropped with ChangesNamespace storage cleanup on purge drop
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Hi @c-thiel @twuebi — this is my first contribution to lakekeeper. I've implemented storage cleanup for namespace folders on drop, as described in #1064. The folder is only removed if it's empty (checked via is_empty()), and cleanup is best-effort so it never fails the drop. Would appreciate a review when you have time! |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/lakekeeper-storage-postgres/src/namespace.rs`:
- Around line 833-843: The namespace_locations parsing in the drop_namespace
function currently uses the ? operator to propagate parse errors, which causes
the entire drop_namespace operation to fail if cleanup metadata parsing fails.
Since namespace_locations is only for best-effort storage cleanup, parse
failures should not abort the namespace drop. Modify the zip and map logic that
processes dropped_ns_ids and dropped_ns_locations to handle Location::from_str
parse errors gracefully by logging warnings for failures instead of propagating
errors with ?, allowing the operation to continue and return successfully even
when some cleanup metadata cannot be parsed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8a8f8695-c37f-4900-988d-80ff275a392e
⛔ Files ignored due to path filters (1)
.sqlx/query-5ff9d5b588ca8f9e76728f27f62254bad468626ff30dd67fea3ab972ae60a0cb.jsonis excluded by!.sqlx/**
📒 Files selected for processing (4)
crates/lakekeeper-integration-tests/tests/namespace_storage_cleanup.rscrates/lakekeeper-storage-postgres/src/namespace.rscrates/lakekeeper/src/server/namespace.rscrates/lakekeeper/src/service/catalog_store/namespace.rs
e075893 to
2f439a3
Compare
|
Fixed — replaced ?-propagating collect with filter_map that logs and skips unparseable locations. Cleanup metadata parse failures no longer abort the drop operation. Updated in commit 2f439a3. |
|
Thank you for your contribution @dadavidtseng! Nontheless this is a useful feature for folks using custom |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/lakekeeper-storage-postgres/src/namespace.rs (1)
837-845: ⚡ Quick winInclude namespace ID in the warning message for better debugging.
The warning correctly logs parse failures and skips invalid locations, but omitting the namespace ID makes it harder to identify which namespace's location is malformed when multiple failures occur during a recursive drop.
🔍 Suggested observability improvement
.filter_map(|(ns_id, loc)| match Location::from_str(&loc) { Ok(location) => Some((NamespaceId::from(ns_id), location)), Err(e) => { tracing::warn!( - "Failed to parse namespace location '{loc}' for cleanup, skipping: {e}" + namespace_id = %ns_id, + location = %loc, + error = %e, + "Failed to parse namespace location for cleanup, skipping" ); None }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/lakekeeper-storage-postgres/src/namespace.rs` around lines 837 - 845, The warning message logged when Location::from_str fails in the filter_map closure does not include the namespace ID, making it difficult to identify which namespace has a malformed location when multiple failures occur. Modify the tracing::warn! call to include the ns_id variable in the warning message alongside the location string and error details so that the warning provides sufficient context for debugging namespace location parse failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/lakekeeper-storage-postgres/src/namespace.rs`:
- Around line 837-845: The warning message logged when Location::from_str fails
in the filter_map closure does not include the namespace ID, making it difficult
to identify which namespace has a malformed location when multiple failures
occur. Modify the tracing::warn! call to include the ns_id variable in the
warning message alongside the location string and error details so that the
warning provides sufficient context for debugging namespace location parse
failures.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4238aa33-1c1b-412a-afad-521a2926c3ee
📒 Files selected for processing (1)
crates/lakekeeper-storage-postgres/src/namespace.rs
|
Thanks for picking this up — empty namespace folders on HDFS/ADLS have been a real, long-standing gap. The shape is good: Let's keep this best-effort and inline (a dedicated task queue would be overkill here) — but two things need fixing, and the limitations should be stated loudly so nobody expects more than it delivers.
It only removes a folder that is already empty at the moment of the drop. Table-data purge is asynchronous, so dropping a namespace soon after its tables (and any recursive drop that still contains live tables) will find the folder non-empty and skip it. In practice it cleans up namespaces whose data was purged earlier. It's a single attempt with no retry — a skipped folder isn't revisited. That way it does what it can, never does harm, and sets the right expectation. The core idea is solid — mostly about scoping the location it touches and being upfront about when it fires. |
|
@c-thiel Thank you for the thorough review! All three points make sense — I'll address them:
I'll also add doc comments documenting the limitations (single attempt, no retry, depends on prior data purge). Will push the updates shortly. |
|
@c-thiel Thanks for the detailed review! I've addressed all three points:
All tests pass locally, clippy and fmt are clean. |
c-thiel
left a comment
There was a problem hiding this comment.
@dadavidtseng are you sure this works currently? I believe the detection is_empty -> remove_all pretty much cancels out as remove_all uses the same list internally. We definitly need a test for the happy path.
| // Guard: never delete the warehouse base itself or locations outside it. | ||
| // On flat/default layouts the persisted location equals the base, and on | ||
| // layout switches the snapshot may no longer match the current layout. | ||
| if *location == base || !location.as_str().starts_with(base.as_str()) { |
There was a problem hiding this comment.
starts_with isn't trailing-slash safe; S3's base_location() has no trailing slash, so s3://bucket/wh matches sibling s3://bucket/wh-other/..... We should use the existing helper:
if *location == base || !location.is_sublocation_of(&base) {
There was a problem hiding this comment.
Fixed, now using location.is_sublocation_of(&base) which handles the trailing-slash edge case correctly.
| }, | ||
| NamespaceParameters { | ||
| prefix: Some(Prefix(prefix.clone())), | ||
| namespace: NamespaceIdent::new("nonempty-ns".to_string()), | ||
| }, | ||
| ) | ||
| .await | ||
| .unwrap(); | ||
|
|
||
| // The file should still exist — cleanup skipped because folder was non-empty. | ||
| let content = storage.read(&file_path).await; | ||
| assert!( | ||
| content.is_ok(), | ||
| "File should still exist because the namespace folder was not empty" | ||
| ); |
There was a problem hiding this comment.
the marker is deleted at :108-113, so the prefix is already empty before drop, and MemoryStorage has no empty-dir entity → the assertion passes even if cleanup never runs. Assert remove_all was called, or rename to reflect it only covers the guard path.
There was a problem hiding this comment.
You're right, MemoryStorage has no directory entities, so the old assertion was vacuous. With the is_hierarchical() gate (see comment 4), cleanup is correctly skipped for non-hierarchical backends. Renamed the tests and updated comments to honestly reflect they only cover the catalog-side contract.
| try_cleanup_namespace_locations( | ||
| &warehouse, | ||
| &state.v1_state.secrets, | ||
| &drop_info.namespace_locations, | ||
| ) | ||
| .await; |
There was a problem hiding this comment.
We should probably not run this if flags.purge is explicitly set to false.
There was a problem hiding this comment.
Agreed, added if flags.purge { ... } around both call sites (non-recursive and recursive paths).
|
|
||
| match crate::service::storage::is_empty(&file_io, location).await { | ||
| Ok(true) => { | ||
| if let Err(e) = super::io::remove_all(&file_io, location).await { |
There was a problem hiding this comment.
I don't think we should remove recursive. I also not sure if it works because it lists recursive and deletes then.
We should gate the whole functionality to Storages that are hierarchical - should probably expose that via the Trait.
There was a problem hiding this comment.
Added StorageProfile::is_hierarchical() (returns true for ADLS/OneLake, false for S3/GCS/Memory) and gated the entire cleanup behind it, so object stores never hit this path.
On the remove_all concern, I agree a recursive delete is a heavier hammer than needed for removing an empty directory. The is_empty guard prevents data loss, but the abstraction isn't ideal. Since adding a trait method (e.g. delete_directory) to LakekeeperStorage is a public API change, I'd appreciate your guidance on how you'd like it shaped. Happy to implement whichever direction you prefer.
|
@c-thiel Thanks for the follow-up review! Addressed all four points:
The one open design question is whether to replace |
…ification\n\nReturns true for ADLS/OneLake (real directory entities), false for\nS3/GCS/Memory (key-prefix based). Used to gate namespace folder\ncleanup to backends where empty directories actually persist.
…n- Use Location::is_sublocation_of() instead of starts_with() for\n trailing-slash safe base-location guard\n- Only run cleanup when flags.purge is true (both drop paths)\n- Skip cleanup entirely for non-hierarchical backends (S3/GCS/Memory)\n where empty directories don't persist\n- Update doc comments to reflect hierarchical gating and limitations
…nMemoryStorage is non-hierarchical, so cleanup is correctly skipped.\nTests now honestly cover the catalog-side contract (namespace removal,\npurge flag behavior) rather than making vacuous storage assertions.\nReplaced test_drop_namespace_with_nonempty_folder_keeps_folder with\ntest_drop_namespace_without_purge_skips_cleanup.
✅ PR Title Formatted CorrectlyThe title of this PR match the correct format. Thank you! |
|
@c-thiel Friendly ping, I've addressed all four review points from your last round (is_sublocation_of, purge gate, hierarchical gate, honest tests). Would you have a chance to re-review? Also still wondering about your preference on the |
What does this PR do?
Adds best-effort storage cleanup for namespace folders when a namespace is dropped.
On filesystem-based storage backends (HDFS, ADLS), dropping a namespace previously
left empty folders behind. This PR wires the namespace drop path to check whether the
namespace's storage folder is empty after the database transaction commits, and removes
it if so.
Why was this PR needed?
Issue #1064 reported that on storage backends with filesystem semantics (HDFS, ADLS),
dropping a namespace leaves remnant empty folders. While object stores like S3 handle
this implicitly (prefixes disappear), true filesystems retain the empty directory.
Investigation confirmed that the
drop_namespacecode path inserver/namespace.rsonly handled database deletion and authorizer cleanup — it never interacted with the
storage backend. The table drop path already had storage cleanup via
TabularPurgeTask,but this pattern was never applied to namespace folders.
What are the relevant issue numbers?
Closes #1064
Implementation details
Extended
NamespaceDropInfowithnamespace_locations: Vec<(NamespaceId, Location)>to carry each dropped namespace's storage location through the drop pipeline.
Updated the PostgreSQL
drop_namespaceSQL query to fetchnamespace_properties->>'location'for the target namespace and all child namespaces(for recursive drops).
Added
try_cleanup_namespace_locations()helper that performs best-effort cleanup:is_empty()remove_all()Wired cleanup into both drop paths: non-recursive and recursive.
Added 3 integration tests covering:
Does this PR meet the acceptance criteria?
Summary by CodeRabbit
New Features
Tests