BUGFIX: Silent failure to write row groups with large lists #19376
Conversation
|
Thanks for the PR! This was added in #18578 which references #18512 I think the issue here is that we are doing this on the vector level, not at the record level which is the actual requirement in the spec. We should make sure records are aligned on page boundaries, but a vector contains multiple records (and can thus span multiple pages). CC @lnkuiper |
|
Maybe this PR can be modified to just change the cast to a |
|
Indeed, Mark summarized it nicely. This will cause us not to follow the Parquet spec anymore, as lists shouldn't span multiple pages. This is considered an invalid Parquet file. DuckDB can read it without issues, but other readers may not, so we cannot make this change. If the cast is changed to a This should be fixed properly by writing at the record level, but our current write infrastructure is at the vector level, so it's very coarse-grained, and requires batches of 2048 records to fit on a page (instead of a single record). This is a larger fix, outside of the scope of |
|
Made the requested changes, MacOS build failure is unrelated: |
|
Thanks! |
Fixes for CTE (de)serialization compatibility with older versions (duckdb/duckdb#19393) BUGFIX: Silent failure to write row groups with large lists (duckdb/duckdb#19376) Throw if non-`VARCHAR` key is passed to `json_object` (duckdb/duckdb#19365) add test tag support [vfs integration tests p1] (duckdb/duckdb#19331)
Fixes for CTE (de)serialization compatibility with older versions (duckdb/duckdb#19393) BUGFIX: Silent failure to write row groups with large lists (duckdb/duckdb#19376) Throw if non-`VARCHAR` key is passed to `json_object` (duckdb/duckdb#19365) add test tag support [vfs integration tests p1] (duckdb/duckdb#19331) Co-authored-by: krlmlr <[email protected]>
When the contents of all lists within a column within a rowgroup are > int32 max duckdb silently fails to write valid parquet files potentially causing loss of data.
This fix changes both it to loudly fail with an exception in the future, and implements a fix which allows writing files in this case, however the fix goes against a listed comment which I was unsure of the reasoning behind. I failed to find any requirements in the parquet spec for it, and am assuming it is for performance reasons, so only go against the comment when absolutely required.
I added a really, really slow test which is able to replicate the issue, on the current version of duckdb it fails on the read, but not the write with:
However, after just replacing that cast with the checked version it will instead fail on the write, then with the other changes it no longer fails at all, but switching to the checked cast seems like good practice in case there are any other similar issues anywhere else.
It seems that the test was too slow and was getting a SIGTERM (it may also be running out of RAM, I'm not sure)
Here is the relevant test, it's difficult to run all the time because operations with large lists in duckdb are generally quite slow, especially when getting up to 2GB in size: