feat: add icon for partition keys #977
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ad9fc58 to
cdb1b64
Compare
cdb1b64 to
2296029
Compare
czgu
reviewed
Aug 23, 2022
| total_size = int(total_size) if total_size is not None else None | ||
|
|
||
| raw_description = ujson.pdumps(description, default=lambda o: o.__dict__) | ||
| partition_keys = get_partition_keys(raw_description) |
There was a problem hiding this comment.
why not operate on description directly? so you don't need to do json dump/load
| raw_description: str = None | ||
|
|
||
| # Arrays of partition keys | ||
| partition_keys: List = [] |
querybook/server/models/metastore.py
Outdated
| earliest_partitions = sql.Column(sql.String(description_length)) | ||
| description = sql.Column(sql.Text(length=mediumtext_length)) | ||
| hive_metastore_description = sql.Column(sql.Text(length=mediumtext_length)) | ||
| partition_keys = sql.Column(sql.Text(length=mediumtext_length)) |
There was a problem hiding this comment.
can you use sql.JSON instead? I am thinking we can use a generic column to store these kind of information.
something like
column_info = sql.Column(sql.JSON)
that way, we can include something like
{
partition_keys: ['colA', 'colB']
}
and in future it is extensible for primary keys, foreign keys for tables that's applicable
| hollow?: boolean; | ||
| } | ||
|
|
||
| export const ColumnIcon: React.FunctionComponent<IColumnIconProps> = ({ |
There was a problem hiding this comment.
it is a bit of overkill to have a helper component in ui/, can you put it under DataTableViewMini/?
| export interface IColumnIconProps { | ||
| name?: AllLucideIconNames; | ||
| tooltip?: string; | ||
| hollow?: boolean; |
There was a problem hiding this comment.
can you use fill instead? or just pass props from this component to the Icon component? so there is no flip of variables
1b5f374 to
c5da338
Compare
|
@czgu Great suggestion, updated! |
czgu
reviewed
Aug 24, 2022
| partition_key.name | ||
| for partition_key in hive_metastore_description.partitionKeys | ||
| ] | ||
| except ValueError: |
There was a problem hiding this comment.
i don't think any of the action here would raise value error? if anything it would be AttributeError right
querybook/server/logic/metastore.py
Outdated
| ) | ||
|
|
||
| partition_keys_string = "\n".join(partition_keys) | ||
| column_infomation = {"partition_keys": partition_keys_string} |
There was a problem hiding this comment.
{"partition_keys": partition_keys}
c5da338 to
f485fa9
Compare
Updated! |
czgu
reviewed
Aug 26, 2022
querybook/server/logic/metastore.py
Outdated
| data_table_id, session=session | ||
| ) | ||
|
|
||
| column_infomation = {"partition_keys": partition_keys} |
There was a problem hiding this comment.
column_infomation = None
if partition_keys:
column_infomation = {"partition_keys": partition_keys}
| const { table, schema, tableColumns, getTable } = useDataTable(tableId); | ||
|
|
||
| const renderPanelView = () => { | ||
| const partitionKeyList = table.column_info |
There was a problem hiding this comment.
const partitionKeyList = table.column_info?.partition_keys ?? [];
querybook/webapp/const/metastore.ts
Outdated
| earliest_partitions: string; | ||
| description: string | ContentState; | ||
| hive_metastore_description: string; | ||
| column_info: Record<string, string[]>; |
There was a problem hiding this comment.
column_info?: {
partition_keys?: string[];
}
f485fa9 to
b1f5665
Compare
b1f5665 to
8e209a0
Compare
|
@czgu Thanks for the advice! |
czgu
approved these changes
Aug 26, 2022
lilyli9
pushed a commit
to lilyli9/querybook
that referenced
this pull request
Jan 26, 2023
## [0.20.0](https://github.expedia.biz/eg-analytics-platform/querybook/compare/0.19.1...0.20.0) (2022-09-08) ### Features * (experimental) add Query transpilation to Querybook (pinterest#988) ([cbf650a](https://github.expedia.biz/eg-analytics-platform/querybook/commit/cbf650a7f0a5dbae16419b0c05f50adf043d3c0d)) * Add a new API of syncing a table from metastore (pinterest#982) ([c0e1dec](https://github.expedia.biz/eg-analytics-platform/querybook/commit/c0e1decfe69cd5919779d7b5504e6c9d516ad76e)) * add icon for partition keys (pinterest#977) ([ca62466](https://github.expedia.biz/eg-analytics-platform/querybook/commit/ca62466b853eb1b3d2523bcabb44d49eafc000e6)) * add query validation to querybook (pinterest#984) ([7069dac](https://github.expedia.biz/eg-analytics-platform/querybook/commit/7069dac81169dacea27f884e770dce079e38d469)) * add support of syncing table/column description from metastore (pinterest#980) ([b8a3eb5](https://github.expedia.biz/eg-analytics-platform/querybook/commit/b8a3eb576a0039b66e8838f20bd1b3ceeae0bd94)) * improve table upload with managed/external (pinterest#979) ([156d020](https://github.expedia.biz/eg-analytics-platform/querybook/commit/156d02086d7363757a52b1b34db8199c6fe40c50)) * Merge remote-tracking branch 'upstream/master' ([394f0da](https://github.expedia.biz/eg-analytics-platform/querybook/commit/394f0daf3e9805358026d591066e63b636e56041)) ### Bug Fixes * column icon css for partition keys (pinterest#987) ([6b08a88](https://github.expedia.biz/eg-analytics-platform/querybook/commit/6b08a88d97fd515c2d74fde10d5b13f6a2a92bbf)) * **DataDoc:** Use Previous Query Engine (pinterest#983) ([1ef498b](https://github.expedia.biz/eg-analytics-platform/querybook/commit/1ef498be2c1e4924c00bce652b0c21f2cea1058d)) * docs website failed to build (pinterest#981) ([970628d](https://github.expedia.biz/eg-analytics-platform/querybook/commit/970628df6ec51b0f907b8ce00ab9e073ff0222f9)) * overflow in raw metastore info (pinterest#976) ([339b2ce](https://github.expedia.biz/eg-analytics-platform/querybook/commit/339b2cee4c7f9d1bc0ca2cafb414a9be40130325)) * scatter/bubble charts were losing their labels after save (pinterest#985) ([0ae4659](https://github.expedia.biz/eg-analytics-platform/querybook/commit/0ae46596480e158c4466e0fee531c4c290207fe4))
rohan-sh1
pushed a commit
to CAI-TECHNOLOGIES/cai-ext-db-explorer
that referenced
this pull request
Feb 28, 2023
* feat: add icon for partition keys * feat: add column info for tables Co-authored-by: shuczhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We add an icon for artition keys in hive tables. The partition_keys are parsed by the hive loader. We left place for other loaders to put partition keys in the base_metastore_loader as well. The icon for partition keys should be like:
