feat: add support for multipart namespaces #2629
Draft
+229
−25
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
52d7283 to
c8cf40a
Compare
Enables catalogs to support hierarchical organization of tables through multi-part namespaces. Introduces changes to the Catalog class and its implementations (DynamoDB, REST, SQL) to handle multi-part namespaces. A new document is added to explain the usage of the multi-part namespace feature. enable multipart namespace Multipart namespace efers to a namespace that is structured with multiple levels or parts and default separated by a delmiiter dot (`.`)
improved type handling and added utility methods
215bd58 to
cc97382
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This PR introduces support for multi-part namespaces, enabling more flexible and organized data layouts. This feature allows users to structure their tables within nested hierarchies (e.g.,
('database', 'schema', 'table')), which is a common pattern in many data warehouse systems.The level of support varies depending on the underlying catalog implementation. Here's a summary of the changes:
closes #2240
Are these changes tested?
Yes, this PR includes comprehensive test coverage:
test_list_multipart_namespaces_200()intests/catalog/test_rest.pyto verify multi-part namespace listing via REST APImkdocs/docs/multi-part-namespace.mdwith usage examples and catalog compatibility matrixAre there any user-facing changes?
Yes, this introduces a new user-facing feature:
Breaking changes: None. This is a backward-compatible addition.
New capabilities:
catalog.create_table("database.schema.table", ...)).) is used consistently across catalog implementationsAPI changes:
Catalogclass now exposes_support_namespacesattribute to indicate multi-part namespace supportnamespace_level()for determining namespace depthidentifier_to_database()and related methods to handle multi-part identifiersDocumentation:
multi-part-namespace.md) explains usage patterns and catalog-specific support