Spark: Support Parquet dictionary encoded UUIDs #13324
Conversation
While fixing some issues on the PyIceberg ends to fully support UUIDs: apache/iceberg-python#2007 I noticed this issue, and was suprised since UUID used to work with Spark, but it turns out that the dictionary encoded UUID was not implemented yet. For PyIceberg we only generate little data, so therefore this wasn't caught previously.
There was a problem hiding this comment.
Generally LGTM
Is there a way to test this? Can we add a dictionary encoded UUID like this?
|
@dingo4dev Yes, we have tests for plain encoded UUIDs, let me add one for dictionary encoded UUIDs as well 👍 |
Just found out that the test above is not testing this code path, since Spark projects a UUID into a String. Test has been added and checked using breakpoints that it hits the newly added lines 👍 |
| Schema schema = new Schema(optional(100, "uuid", Types.UUIDType.get())); | ||
|
|
||
| File dataFile = File.createTempFile("junit", null, temp.toFile()); | ||
| assertThat(dataFile.delete()).as("Delete should succeed").isTrue(); |
There was a problem hiding this comment.
Why do we do this? Test looks fine just wondering why we create and then delete
There was a problem hiding this comment.
There are multiple tests that use the same directory. Removing this line causes:
> Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED
TestParquetVectorizedReads > testUuidReads() FAILED
org.apache.iceberg.exceptions.AlreadyExistsException: File already exists: /var/folders/q_/gj3w0fts15n9l1dz58630jg40000gp/T/junit-17282529623949094801/junit6226812078387717274.tmp
at app//org.apache.iceberg.Files$LocalOutputFile.create(Files.java:56)
There was a problem hiding this comment.
I've created an issue to fix this for all the tests: #13506
There was a problem hiding this comment.
Took a look at the issue, @TempDir make sure each test get a unique directory so multiple tests won't affect each other. However, the reason these tests need to manually delete the directory is that @TempDir always creates the directory before the test starts, and getParquetWriter throws an exception if the target folder already exists.
What we actually need here is just a randomly generated path without creating anything at that location. Unfortunately, @TempDir doesn’t currently support that level of control.
There was a problem hiding this comment.
I'm not a JUnit expert @liamzwbao, but this blog suggests that you can also recreate the directory after each test
|
Thanks @RussellSpitzer, @kevinjqliu and @dingo4dev for the review 🚀 |
|
@Fokko are you planning to port this over to Spark 4? I think it would be good to get this out with 1.10 |
|
and perhaps spark 3.4 as well |
While fixing some issues on the PyIceberg ends to fully support UUIDs: apache/iceberg-python#2007
I noticed this issue, and was suprised since UUID used to work with Spark, but it turns out that the dictionary encoded UUID was not implemented yet.
For PyIceberg we only generate little data, so therefore this wasn't caught previously.
Closes #4581