Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Remove trailing slash when building metadata location in Iceberg #13984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 5, 2022
Merged

Remove trailing slash when building metadata location in Iceberg #13984

merged 1 commit into from
Sep 5, 2022

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Sep 5, 2022 •
edited
Loading

Description

stripTrailingSlash is already used in IcebergUtil#getLocationProvider > locationsFor for data path. We may want additional handling for double slash in the middle of path, but removing trailing slash anyway as data path.
Fixes #13759

Documentation

(x) No documentation is needed.

Release notes

(x) Release notes entries required with the following suggested text:

# Iceberg
* Fix creating metadata and manifest files with url encoded name when the metadata location has trailing slashes on S3. ({issue}`13759`)

@cla-bot cla-bot bot added the cla-signed label Sep 5, 2022
@findepi
Copy link
Member

findepi commented Sep 5, 2022

Remove trailing spaces when building metadata location in Iceberg

"spaces" or "slash"?

also, what's the end-user effect of the change?

@ebyhr
Copy link
Member Author

ebyhr commented Sep 5, 2022 •
edited
Loading

My bad, it's "slash".

Before this change, the connector created metadata files with fragments on S3 when the metadata location has trailing slashes likes CREATE TABLE ... WITH (location = 's3://.../'). Probably, #13530 introduced this bug.

aws s3 ls s3://.../metadata/
2022-09-05 12:21:54       1593 00000-aeacb940-209f-4b60-806a-b0d27866d8a1.metadata.json#%2Febyhr%2Ftest_13759_4%2F%2Fmetadata%2F00000-aeacb940-209f-4b60-806a-b0d27866d8a1.metadata.json
...
2022-09-05 12:57:42       6447 2dcd8027-d149-4a37-bd55-36675487a88c-m0.avro#%2Febyhr%2Ftest_13759_4%2F%2Fmetadata%2F2dcd8027-d149-4a37-bd55-36675487a88c-m0.avro

@ebyhr ebyhr changed the title Remove trailing spaces when building metadata location in Iceberg Remove trailing slash when building metadata location in Iceberg Sep 5, 2022
@ebyhr ebyhr force-pushed the ebi/iceberg-s3-path-regression branch from 61230f4 to 9194f0f Compare September 5, 2022 11:01
@ebyhr ebyhr merged commit c155a3e into trinodb:master Sep 5, 2022
@ebyhr ebyhr deleted the ebi/iceberg-s3-path-regression branch September 5, 2022 22:54
@ebyhr ebyhr mentioned this pull request Sep 5, 2022
Closed
@github-actions github-actions bot added this to the 395 milestone Sep 5, 2022
@colebow colebow mentioned this pull request Sep 6, 2022
Merged
@alexjo2144 alexjo2144 mentioned this pull request Sep 27, 2022
Closed
@ebyhr ebyhr mentioned this pull request Sep 28, 2022
Closed
@findepi findepi mentioned this pull request Oct 11, 2022
Closed
@ebyhr ebyhr mentioned this pull request Oct 11, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@homar homar homar approved these changes

@findepi findepi findepi approved these changes

@electrum electrum Awaiting requested review from electrum

@findinpath findinpath Awaiting requested review from findinpath

@alexjo2144 alexjo2144 Awaiting requested review from alexjo2144

Assignees
No one assigned
Labels
cla-signed
Milestone
395
Development

Successfully merging this pull request may close these issues.

Regression on v393 -- Iceberg can't find key on S3
3 participants
@ebyhr @findepi @homar