Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: clean up empty namespace folders on storage after drop - #1858

Open
dadavidtseng wants to merge 15 commits into
lakekeeper:mainfrom
dadavidtseng:fix-issue-1064
Open

fix: clean up empty namespace folders on storage after drop#1858
dadavidtseng wants to merge 15 commits into
lakekeeper:mainfrom
dadavidtseng:fix-issue-1064

Conversation

@dadavidtseng

@dadavidtseng dadavidtseng commented Jun 17, 2026

Copy link
Copy Markdown

What does this PR do?

Adds best-effort storage cleanup for namespace folders when a namespace is dropped.
On filesystem-based storage backends (HDFS, ADLS), dropping a namespace previously
left empty folders behind. This PR wires the namespace drop path to check whether the
namespace's storage folder is empty after the database transaction commits, and removes
it if so.

Why was this PR needed?

Issue #1064 reported that on storage backends with filesystem semantics (HDFS, ADLS),
dropping a namespace leaves remnant empty folders. While object stores like S3 handle
this implicitly (prefixes disappear), true filesystems retain the empty directory.

Investigation confirmed that the drop_namespace code path in server/namespace.rs
only handled database deletion and authorizer cleanup — it never interacted with the
storage backend. The table drop path already had storage cleanup via TabularPurgeTask,
but this pattern was never applied to namespace folders.

What are the relevant issue numbers?

Closes #1064

Implementation details

  1. Extended NamespaceDropInfo with namespace_locations: Vec<(NamespaceId, Location)>
    to carry each dropped namespace's storage location through the drop pipeline.

  2. Updated the PostgreSQL drop_namespace SQL query to fetch
    namespace_properties->>'location' for the target namespace and all child namespaces
    (for recursive drops).

  3. Added try_cleanup_namespace_locations() helper that performs best-effort cleanup:

    • Gets storage IO via the warehouse's storage profile and secret
    • For each namespace location, checks if the folder is empty using is_empty()
    • If empty, removes the folder using remove_all()
    • Errors are logged and swallowed — cleanup must not fail the drop operation
  4. Wired cleanup into both drop paths: non-recursive and recursive.

  5. Added 3 integration tests covering:

    • Empty namespace folder is cleaned up after drop
    • Non-empty namespace folder is preserved
    • Recursive drop correctly removes namespace from catalog

Does this PR meet the acceptance criteria?

  • Tests added for new/changed behavior
  • All tests passing
  • Follows project style guide (conventional commits, clippy clean)
  • No breaking changes introduced
  • Documentation updated (N/A — no user-facing docs needed)

Summary by CodeRabbit

  • New Features

    • Namespace deletion with purge now performs best-effort removal of empty namespace storage directories after catalog and authorization updates.
    • Cleanup runs for both non-recursive and recursive purge flows.
    • Cleanup is resilient: invalid location entries are skipped with a warning, and cleanup failures are logged without blocking deletion.
  • Tests

    • Added integration tests for: empty-directory cleanup on non-recursive purge, retention when storage isn’t empty, and recursive purge where only catalog removal is asserted (storage cleanup not validated).

@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds best-effort deletion of empty namespace storage folders when a namespace is dropped with purge: true. NamespaceDropInfo gains a namespace_locations field; the PostgreSQL drop_namespace query is extended to return those locations; the server-side drop handler wires them into a new try_cleanup_namespace_locations helper; three integration tests validate the behavior.

Changes

Namespace storage cleanup on purge drop

Layer / File(s) Summary
NamespaceDropInfo contract
crates/lakekeeper/src/service/catalog_store/namespace.rs
Adds namespace_locations: Vec<(NamespaceId, Location)> to NamespaceDropInfo to carry namespace-id/location pairs for post-drop storage cleanup.
PostgreSQL query: collect dropped locations
crates/lakekeeper-storage-postgres/src/namespace.rs
Adds use lakekeeper_io::Location and str::FromStr to imports, extends the drop_namespace SQL CTE to return dropped_ns_ids and dropped_ns_locations (from namespace_properties->>'location') ordered by namespace_id, and populates NamespaceDropInfo.namespace_locations by zipping and parsing those values with graceful error handling (invalid locations are logged and skipped).
Server drop handler wiring & cleanup helper
crates/lakekeeper/src/server/namespace.rs
Adds #[allow(clippy::too_many_lines)], threads secret_store into try_recursive_drop, calls try_cleanup_namespace_locations after both the non-recursive and recursive drop paths, and adds the new helper that fetches the warehouse storage secret, initializes storage IO, checks folder emptiness, and removes it if empty—logging all errors without failing the drop.
Integration tests
crates/lakekeeper-integration-tests/tests/namespace_storage_cleanup.rs
Adds CleanupSetup, setup_cleanup_test (in-memory IO, hard delete profile), and three sqlx::test cases covering empty-folder cleanup on non-recursive purge drop, non-empty folder preservation (expected not to be cleaned), and recursive purge drop removing the namespace from the catalog.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐇 Hop, hop, I sweep the floor,
Empty folders? Gone—no more!
If something lingers, I take a peek,
Non-empty dens? I dare not seek.
The catalog shines, the storage clean,
The neatest warren you've ever seen! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: implementing cleanup of empty namespace folders on storage after namespace drop.
Linked Issues check ✅ Passed All coding requirements from #1064 are met: the implementation checks if namespace folders are empty before deletion via is_empty(), prevents blanket deletion, and maintains best-effort cleanup without aborting the drop operation.
Out of Scope Changes check ✅ Passed All changes are directly scoped to namespace storage cleanup: adding namespace_locations to NamespaceDropInfo, extending SQL to fetch locations, implementing cleanup logic, and adding integration tests to validate the behavior.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cla-assistant

cla-assistant Bot commented Jun 17, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@dadavidtseng

Copy link
Copy Markdown
Author

Hi @c-thiel @twuebi — this is my first contribution to lakekeeper. I've implemented storage cleanup for namespace folders on drop, as described in #1064. The folder is only removed if it's empty (checked via is_empty()), and cleanup is best-effort so it never fails the drop. Would appreciate a review when you have time!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/lakekeeper-storage-postgres/src/namespace.rs`:
- Around line 833-843: The namespace_locations parsing in the drop_namespace
function currently uses the ? operator to propagate parse errors, which causes
the entire drop_namespace operation to fail if cleanup metadata parsing fails.
Since namespace_locations is only for best-effort storage cleanup, parse
failures should not abort the namespace drop. Modify the zip and map logic that
processes dropped_ns_ids and dropped_ns_locations to handle Location::from_str
parse errors gracefully by logging warnings for failures instead of propagating
errors with ?, allowing the operation to continue and return successfully even
when some cleanup metadata cannot be parsed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8a8f8695-c37f-4900-988d-80ff275a392e

📥 Commits

Reviewing files that changed from the base of the PR and between 01e62ef and 98294d9.

⛔ Files ignored due to path filters (1)
  • .sqlx/query-5ff9d5b588ca8f9e76728f27f62254bad468626ff30dd67fea3ab972ae60a0cb.json is excluded by !.sqlx/**
📒 Files selected for processing (4)
  • crates/lakekeeper-integration-tests/tests/namespace_storage_cleanup.rs
  • crates/lakekeeper-storage-postgres/src/namespace.rs
  • crates/lakekeeper/src/server/namespace.rs
  • crates/lakekeeper/src/service/catalog_store/namespace.rs

Comment thread crates/lakekeeper-storage-postgres/src/namespace.rs Outdated
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 18, 2026
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 18, 2026
@dadavidtseng

Copy link
Copy Markdown
Author

Fixed — replaced ?-propagating collect with filter_map that logs and skips unparseable locations. Cleanup metadata parse failures no longer abort the drop operation. Updated in commit 2f439a3.

@c-thiel

c-thiel commented Jun 18, 2026

Copy link
Copy Markdown
Member

Thank you for your contribution @dadavidtseng!
Please note that we recently changed our default storage layout to not have namespace folders anymore (#1853). This will be helpful as we are adding a "move" Operation soon.

Nontheless this is a useful feature for folks using custom StorageLayouts! We'll try to get a review in shortly.

@dadavidtseng

Copy link
Copy Markdown
Author

@c-thiel Thank you for the context on #1853 and the upcoming move operation! I'm glad this will still be useful for custom StorageLayouts. Looking forward to the review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/lakekeeper-storage-postgres/src/namespace.rs (1)

837-845: ⚡ Quick win

Include namespace ID in the warning message for better debugging.

The warning correctly logs parse failures and skips invalid locations, but omitting the namespace ID makes it harder to identify which namespace's location is malformed when multiple failures occur during a recursive drop.

🔍 Suggested observability improvement
             .filter_map(|(ns_id, loc)| match Location::from_str(&loc) {
                 Ok(location) => Some((NamespaceId::from(ns_id), location)),
                 Err(e) => {
                     tracing::warn!(
-                        "Failed to parse namespace location '{loc}' for cleanup, skipping: {e}"
+                        namespace_id = %ns_id,
+                        location = %loc,
+                        error = %e,
+                        "Failed to parse namespace location for cleanup, skipping"
                     );
                     None
                 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/lakekeeper-storage-postgres/src/namespace.rs` around lines 837 - 845,
The warning message logged when Location::from_str fails in the filter_map
closure does not include the namespace ID, making it difficult to identify which
namespace has a malformed location when multiple failures occur. Modify the
tracing::warn! call to include the ns_id variable in the warning message
alongside the location string and error details so that the warning provides
sufficient context for debugging namespace location parse failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/lakekeeper-storage-postgres/src/namespace.rs`:
- Around line 837-845: The warning message logged when Location::from_str fails
in the filter_map closure does not include the namespace ID, making it difficult
to identify which namespace has a malformed location when multiple failures
occur. Modify the tracing::warn! call to include the ns_id variable in the
warning message alongside the location string and error details so that the
warning provides sufficient context for debugging namespace location parse
failures.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4238aa33-1c1b-412a-afad-521a2926c3ee

📥 Commits

Reviewing files that changed from the base of the PR and between 2f439a3 and 915a0e6.

📒 Files selected for processing (1)
  • crates/lakekeeper-storage-postgres/src/namespace.rs

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 18, 2026
@c-thiel

c-thiel commented Jun 22, 2026

Copy link
Copy Markdown
Member

Thanks for picking this up — empty namespace folders on HDFS/ADLS have been a real, long-standing gap. The shape is good: is_empty before delete, errors logged-and-swallowed so cleanup never fails the drop, filter_map skipping unparseable locations, and going through the LakekeeperStorage trait so it's storage-profile-agnostic.

Let's keep this best-effort and inline (a dedicated task queue would be overkill here) — but two things need fixing, and the limitations should be stated loudly so nobody expects more than it delivers.

  1. Don't let it delete the warehouse base.
    On the default tabular-only/flat layout, render_namespace_path returns vec![] (storage_layout.rs:273), so the persisted namespace location equals the warehouse base. That makes is_empty(base)remove_all(base) capable of wiping the warehouse root when it happens to be empty. Please gate on the persisted location, not the current layout (the layout can be switched, and the location is a create-time snapshot — pre-0.13 namespaces even carry a {uuid} dir): only act when loc != base && loc.starts_with(base) (strictly below the base). Fails closed, layout-switch-safe.

  2. Deepest-first ordering for recursive drops.
    The SQL orders by namespace_id (arbitrary), but nested child folders sit under their parents, so a parent is often checked before its children → is_empty false → skipped. Order by location depth/length descending so children go first. Cheap, and it's the difference between recursive cleanup working or not on full-hierarchy.

  3. Make the tests honest.
    They build a separate MemoryStorage::new() rather than the warehouse's IO handle, so test_drop_empty_namespace_cleans_up_folder only asserts a catalog 404 — not that the folder was removed; the recursive test asserts no storage state. Pulling IO from warehouse.storage_profile.file_io(...) and asserting on the location after drop would actually cover Github Actions #1/Maybe migrate to Stub generation for OpenAPI spec #2. (The tokio::time::sleep "give async cleanup time" calls are no-ops, since cleanup is inline — fine to drop them.)

  4. Document the limitations — in a doc comment on try_cleanup_namespace_locations and a line in the PR description from which we generate the release notes:

It only removes a folder that is already empty at the moment of the drop. Table-data purge is asynchronous, so dropping a namespace soon after its tables (and any recursive drop that still contains live tables) will find the folder non-empty and skip it. In practice it cleans up namespaces whose data was purged earlier. It's a single attempt with no retry — a skipped folder isn't revisited.

That way it does what it can, never does harm, and sets the right expectation. The core idea is solid — mostly about scoping the location it touches and being upfront about when it fires.

@dadavidtseng

dadavidtseng commented Jun 24, 2026

Copy link
Copy Markdown
Author

@c-thiel Thank you for the thorough review! All three points make sense — I'll address them:

  1. Guard against deleting the warehouse base by checking loc != base && loc.starts_with(base)
  2. Order locations by path length descending for deepest-first cleanup
  3. Fix tests to use the warehouse's actual IO handle and assert on real storage state

I'll also add doc comments documenting the limitations (single attempt, no retry, depends on prior data purge). Will push the updates shortly.

@dadavidtseng

Copy link
Copy Markdown
Author

@c-thiel Thanks for the detailed review! I've addressed all three points:

  1. Base-location guard: Added a check that skips cleanup when location == base || !location.starts_with(base) — strictly below only. Also added doc comments covering the limitations (single attempt, depends on prior purge, errors swallowed).
  2. Deepest-first ordering: Changed both ORDER BY clauses to length(n.namespace_properties->>'location') DESC, n.namespace_id so children are cleaned before parents in recursive drops. Updated the .sqlx offline cache accordingly.
  3. Honest tests: Tests now use the shared thread-local MemoryStorage::new() (same backing store as file_io()) and assert actual storage state after drop — not just catalog 404. Removed all sleep calls.

All tests pass locally, clippy and fmt are clean.

@c-thiel c-thiel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dadavidtseng are you sure this works currently? I believe the detection is_empty -> remove_all pretty much cancels out as remove_all uses the same list internally. We definitly need a test for the happy path.

// Guard: never delete the warehouse base itself or locations outside it.
// On flat/default layouts the persisted location equals the base, and on
// layout switches the snapshot may no longer match the current layout.
if *location == base || !location.as_str().starts_with(base.as_str()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

starts_with isn't trailing-slash safe; S3's base_location() has no trailing slash, so s3://bucket/wh matches sibling s3://bucket/wh-other/..... We should use the existing helper:

if *location == base || !location.is_sublocation_of(&base) {

@dadavidtseng dadavidtseng Jun 27, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, now using location.is_sublocation_of(&base) which handles the trailing-slash edge case correctly.

Comment on lines +156 to +170
},
NamespaceParameters {
prefix: Some(Prefix(prefix.clone())),
namespace: NamespaceIdent::new("nonempty-ns".to_string()),
},
)
.await
.unwrap();

// The file should still exist — cleanup skipped because folder was non-empty.
let content = storage.read(&file_path).await;
assert!(
content.is_ok(),
"File should still exist because the namespace folder was not empty"
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the marker is deleted at :108-113, so the prefix is already empty before drop, and MemoryStorage has no empty-dir entity → the assertion passes even if cleanup never runs. Assert remove_all was called, or rename to reflect it only covers the guard path.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, MemoryStorage has no directory entities, so the old assertion was vacuous. With the is_hierarchical() gate (see comment 4), cleanup is correctly skipped for non-hierarchical backends. Renamed the tests and updated comments to honestly reflect they only cover the catalog-side contract.

Comment on lines +586 to +591
try_cleanup_namespace_locations(
&warehouse,
&state.v1_state.secrets,
&drop_info.namespace_locations,
)
.await;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably not run this if flags.purge is explicitly set to false.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, added if flags.purge { ... } around both call sites (non-recursive and recursive paths).


match crate::service::storage::is_empty(&file_io, location).await {
Ok(true) => {
if let Err(e) = super::io::remove_all(&file_io, location).await {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should remove recursive. I also not sure if it works because it lists recursive and deletes then.

We should gate the whole functionality to Storages that are hierarchical - should probably expose that via the Trait.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added StorageProfile::is_hierarchical() (returns true for ADLS/OneLake, false for S3/GCS/Memory) and gated the entire cleanup behind it, so object stores never hit this path.
On the remove_all concern, I agree a recursive delete is a heavier hammer than needed for removing an empty directory. The is_empty guard prevents data loss, but the abstraction isn't ideal. Since adding a trait method (e.g. delete_directory) to LakekeeperStorage is a public API change, I'd appreciate your guidance on how you'd like it shaped. Happy to implement whichever direction you prefer.

@dadavidtseng

Copy link
Copy Markdown
Author

@c-thiel Thanks for the follow-up review! Addressed all four points:

  1. is_sublocation_of for trailing-slash safety
  2. Honest tests reflecting MemoryStorage limitations
  3. flags.purge gate on both drop paths
  4. StorageProfile::is_hierarchical() gate — S3/GCS/Memory skip cleanup entirely

The one open design question is whether to replace remove_all with a dedicated trait method for directory removal. I've left it as-is for now since ADLS's remove_all override uses native DELETE ?recursive=true which does remove the directory entry correctly, but happy to add a narrower delete_directory method if you'd prefer that.

dadavidtseng and others added 4 commits June 27, 2026 12:36
…ification\n\nReturns true for ADLS/OneLake (real directory entities), false for\nS3/GCS/Memory (key-prefix based). Used to gate namespace folder\ncleanup to backends where empty directories actually persist.
…n- Use Location::is_sublocation_of() instead of starts_with() for\n trailing-slash safe base-location guard\n- Only run cleanup when flags.purge is true (both drop paths)\n- Skip cleanup entirely for non-hierarchical backends (S3/GCS/Memory)\n where empty directories don't persist\n- Update doc comments to reflect hierarchical gating and limitations
…nMemoryStorage is non-hierarchical, so cleanup is correctly skipped.\nTests now honestly cover the catalog-side contract (namespace removal,\npurge flag behavior) rather than making vacuous storage assertions.\nReplaced test_drop_namespace_with_nonempty_folder_keeps_folder with\ntest_drop_namespace_without_purge_skips_cleanup.
@github-actions

Copy link
Copy Markdown
Contributor

✅ PR Title Formatted Correctly

The title of this PR match the correct format. Thank you!

@dadavidtseng

dadavidtseng commented Jul 18, 2026

Copy link
Copy Markdown
Author

@c-thiel Friendly ping, I've addressed all four review points from your last round (is_sublocation_of, purge gate, hierarchical gate, honest tests). Would you have a chance to re-review? Also still wondering about your preference on the delete_directory trait method vs remove_all question. Happy to split that out into a follow-up PR if you'd prefer.

@c-thiel c-thiel added this to the Release 0.14.0 milestone Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deleted namespaces may leave traces on filesystems

2 participants