Provide workaround for prefetching parquet files with incorrect page offsets#13697
Merged
Mytherin merged 6 commits intoduckdb:featurefrom Sep 26, 2024
Conversation
…nism when detecting them
This forces users to become aware that this is happening and preventing quiet performance regressions
carlopi
reviewed
Sep 2, 2024
test/parquet/prefetching.test
Outdated
Comment on lines
21
to
22
| # This file messes with DuckDB's prefetching mechanism, however the prefetching mechanism automatically disables | ||
| # when it detects this situation meaning that the query still succeeds (possibly at a performance penalty) |
Contributor
There was a problem hiding this comment.
Outdated? Also the title of the PR I think
Collaborator
Author
|
I think this is good to go @Mytherin? |
Collaborator
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the following bug:
Leading to
why this happens?
The reason is that in the respective parquet file the index_page_offset fields are set to 0. DuckDB uses this value to compute the byte range that needs to be prefetched for this column chunk. This prefetching mechanism now no longer works properly because DuckDB can not easily determine the byte ranges to prefetch
The fix
The workaround is to add an option that disables the Prefetching mechanism and hinting the users to use that option. This will disable the prefetching mechanism for these files allowing them to be scanned.
To be able to test this easily I also added the debug option
prefetch_all_parquet_files