Core, Spark: Preserve the relative path in RewriteTablePathUtil on staging #13645

Elbehery · 2025-07-23T18:05:27Z

Elbehery · 2025-07-23T18:06:23Z

dramaticlly

I think we can update the pull request title to be something better, like "[Core, Spark] Preserve the relative path in RewriteTablePathUtil on staging"

core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java

core/src/test/java/org/apache/iceberg/RewriteTablePathUtilTest.java

dramaticlly

Looks reasonable to me. I dont think this will impact existing user as all staging files paths are written in file_list_location, distributed copy will rely on content of file listing when replicate to destination.

dramaticlly · 2025-07-23T23:46:27Z

@szehon-ho can you help approve the CI workflows?

Elbehery · 2025-07-24T07:08:46Z

@szehon-ho could you re-run the CI plz ?

pvary · 2025-07-24T08:41:49Z

@ebyhr: Does the RewriteTablePathUtil handle files out of the table root? Is this solution working there as well?

hpinca98 · 2025-07-24T08:59:19Z

I think on this topic, would it be a good idea to slightly modify the documentation for this procedure?
Currently it states "Stages a copy of the Iceberg table's metadata files", but the procedure also stages deletion files. I don t think there is anything else other than these. What do you think?

Elbehery · 2025-07-24T12:16:32Z

i dont know what is the fix for the CI

locally it is fine

export JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home && ./gradlew spotlessCheck

BUILD SUCCESSFUL

Elbehery · 2025-07-24T12:21:20Z

./gradlew spotlessApply                                                                                                        iceberg-13630
Starting a Gradle Daemon, 2 incompatible Daemons could not be reused, use --status for details
Configuration on demand is an incubating feature.

[Incubating] Problems report is available at: file:///Users/melbeher/go/src/apache/iceberg/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.14.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 4s
71 actionable tasks: 71 up-to-date

dramaticlly · 2025-07-24T16:39:06Z

./gradlew spotlessApply                                                                                                        iceberg-13630
Starting a Gradle Daemon, 2 incompatible Daemons could not be reused, use --status for details
Configuration on demand is an incubating feature.

[Incubating] Problems report is available at: file:///Users/melbeher/go/src/apache/iceberg/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.14.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 4s
71 actionable tasks: 71 up-to-date

@Elbehery
Usually I would run ./gradlew -DallVersions revapi spotlessApply checkStyleMain checkStyleTest before commit

Looks like CI is failing on spotless, I would double check if there's any files changed as the result of running linter (spotlessApply command). In case if there's files changed on disk but not reflect in the patch yet, I would add them and push a new commit and this would make next CI run pass build checks.

Elbehery · 2025-07-24T17:24:46Z

@dramaticlly i ran, no diff at all :'(

./gradlew -DallVersions revapi spotlessApply checkStyleMain checkStyleTest                iceberg-13630
Starting a Gradle Daemon (subsequent builds will be faster)
Configuration on demand is an incubating feature.

> Task :iceberg-api:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-api:testJar
file '/Users/melbeher/go/src/apache/iceberg/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '/Users/melbeher/go/src/apache/iceberg/api/build/resources/test/iceberg-build.properties', which has already been copied there.

> Task :iceberg-core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-azure:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-aliyun:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-gcp:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-nessie:compileJava
Note: /Users/melbeher/go/src/apache/iceberg/nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-aws:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-parquet:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-core:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-data:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-aws:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-hive-metastore:compileTestJava
Note: /Users/melbeher/go/src/apache/iceberg/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/melbeher/go/src/apache/iceberg/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCatalog.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-mr:compileJava
Note: /Users/melbeher/go/src/apache/iceberg/mr/src/main/java/org/apache/iceberg/mr/mapred/MapredIcebergInputFormat.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-data:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-parquet:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-kafka-connect:iceberg-kafka-connect:compileJava
Note: /Users/melbeher/go/src/apache/iceberg/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :iceberg-flink:iceberg-flink-2.0:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-flink:iceberg-flink-2.0:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-spark:iceberg-spark-4.0_2.13:compileScala
[Warn] : javac: [options] system modules path not set in conjunction with -source 11

> Task :iceberg-spark:iceberg-spark-4.0_2.13:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-spark:iceberg-spark-extensions-4.0_2.13:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

[Incubating] Problems report is available at: file:///Users/melbeher/go/src/apache/iceberg/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.14.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1m 4s
213 actionable tasks: 122 executed, 5 from cache, 86 up-to-date
melbeher@melbeher-mac iceberg % git status                                                                                iceberg-13630
On branch iceberg-13630
Your branch and 'origin/iceberg-13630' have diverged,
and have 7 and 1 different commits each, respectively.
  (use "git pull" if you want to integrate the remote branch with yours)

nothing to commit, working tree clean

Elbehery · 2025-07-24T22:12:03Z

@dramaticlly any other suggestions ?

ebyhr · 2025-07-25T00:54:04Z

@Elbehery You can reproduce CI failures by the below command:

./gradlew -DsparkVersions=3.5 :iceberg-spark:iceberg-spark-3.5_2.12:spotlessJavaCheck

Elbehery · 2025-07-25T06:08:50Z

@Elbehery You can reproduce CI failures by the below command:
./gradlew -DsparkVersions=3.5 :iceberg-spark:iceberg-spark-3.5_2.12:spotlessJavaCheck 

yes this helps 👍🏽

Elbehery · 2025-07-25T08:04:36Z

i think i got it now, it passes locally

would you kindly run the CI again ?

szehon-ho

sorry for the delay, approved workflow. Code changes look ok to me, thanks! some minor items

szehon-ho · 2025-07-28T19:51:14Z

...link/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2DistributionMode.java

      assertThat(addedDataFiles).hasSize(writeParallelism);
      // verify there is no overlap in min-max stats range
-      if (writeParallelism > 1) {
+      if (writeParallelism > 2) {


is this unrelated?

the test is perma failing, and this is the fix

please correct me if i am wrong

Generally speaking, if there's unrelated but flaky test, maintainer can help rerun the test to pass the CI, or a separate PR to fix the problem. From what i can tell, this test is configured with writeParallelism to either 1 or 2, so change if condition to > 2 means skip the test.

If this test is permanently failing then this is new.
I have seen flaky tests, and @stevenzwu is planning to take care some of them. The current failures seem unrelated.

I tried testRangeDistributionStatisticsMigration test in both TestFlinkIcebergSinkDistributionMode and TestFlinkIcebergSinkV2DistributionMode with Flink 2.0 in Intellij. In both cases, it is consistently failing.

Yes, there is a bit flakiness of this test as reported in issue #11835 . but this is definitely sth new that it fails consistently now.

merged. @Elbehery please rebase your PR

done 👍🏽

@Elbehery I think you want to remove this change flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2DistributionMode.java from your PR

fixed 👍🏽

sorry i forgot to revert this :)

szehon-ho · 2025-07-28T19:52:03Z

core/src/test/java/org/apache/iceberg/TestRewriteTablePathUtil.java

+
+import org.junit.jupiter.api.Test;
+
+public class TestRewriteTablePathUtil {


can you also add some tests in TestRewriteTablePathsAction, to test this nested directory case end-to-end?

@Elbehery can you ensure this comment is addressed as well.

updated 👍🏽

Elbehery · 2025-07-31T17:07:42Z

could you run the CI again

cc @dramaticlly

Elbehery · 2025-08-02T13:39:05Z

@szehon-ho hello

Could u plz re-run the CI ?

Signed-off-by: Mustafa Elbehery <[email protected]>

Elbehery · 2025-08-03T16:07:33Z

@szehon-ho hello

Could u plz re-run the CI ?

Elbehery · 2025-08-03T16:28:07Z

btw. the failure in Spark integration CI job is due to jdk version

JavaVersion javaVersion = JavaVersion.current()
if (javaVersion != JavaVersion.VERSION_17 && javaVersion != JavaVersion.VERSION_21) {
  throw new GradleException("Spark 4.0 build requires JDK 17 or 21 but was executed with JDK " + javaVersion)
}

Elbehery · 2025-08-04T13:52:41Z

🥳 🥳 🥳 🥳 🥳 🥳

Elbehery · 2025-08-04T22:33:36Z

@dramaticlly the CI is green

any more reviews ?

Elbehery · 2025-08-05T14:06:09Z

cc @nastra

szehon-ho · 2025-08-05T18:55:59Z

Merged, thanks @Elbehery and all for extra reviews !

github-actions bot added spark core KAFKACONNECT labels Jul 23, 2025

Elbehery force-pushed the iceberg-13630 branch from 4ce9d48 to 27b9bd0 Compare July 23, 2025 18:06

github-actions bot removed the KAFKACONNECT label Jul 23, 2025

dramaticlly reviewed Jul 23, 2025

View reviewed changes

Elbehery changed the title ~~Iceberg 13630~~ Iceberg-13630[Core]: Preserve the relative path in RewriteTablePathUtil on staging Jul 23, 2025

Elbehery force-pushed the iceberg-13630 branch from 27b9bd0 to b99b67a Compare July 23, 2025 22:11

ebyhr reviewed Jul 23, 2025

View reviewed changes

core/src/test/java/org/apache/iceberg/RewriteTablePathUtilTest.java Outdated Show resolved Hide resolved

Elbehery force-pushed the iceberg-13630 branch from b99b67a to 4a2af54 Compare July 23, 2025 22:44

dramaticlly approved these changes Jul 23, 2025

View reviewed changes

Elbehery force-pushed the iceberg-13630 branch from 4a2af54 to d38413e Compare July 24, 2025 06:48

Elbehery force-pushed the iceberg-13630 branch from d38413e to b3470f0 Compare July 24, 2025 10:38

github-actions bot added the flink label Jul 24, 2025

Elbehery force-pushed the iceberg-13630 branch from b3470f0 to 24a94ca Compare July 24, 2025 17:25

Elbehery force-pushed the iceberg-13630 branch from 24a94ca to 3b8785b Compare July 25, 2025 08:04

szehon-ho reviewed Jul 28, 2025

View reviewed changes

nastra changed the title ~~Iceberg-13630[Core]: Preserve the relative path in RewriteTablePathUtil on staging~~ Core, Spark: Preserve the relative path in RewriteTablePathUtil on staging Jul 29, 2025

stevenzwu mentioned this pull request Jul 30, 2025

Flink: disable flaky testRangeDistributionStatisticsMigration() #13711

Merged

Elbehery force-pushed the iceberg-13630 branch 3 times, most recently from f6ec05c to e872c2e Compare July 31, 2025 08:04

Elbehery force-pushed the iceberg-13630 branch from e872c2e to fffd316 Compare August 1, 2025 08:35

Iceberg-13630: fix RewriteTablePaths throws FileAlreadyExistsException

eaf01fa

Signed-off-by: Mustafa Elbehery <[email protected]>

Elbehery force-pushed the iceberg-13630 branch from fffd316 to eaf01fa Compare August 3, 2025 15:54

nastra requested a review from szehon-ho August 5, 2025 14:19

szehon-ho approved these changes Aug 5, 2025

View reviewed changes

szehon-ho merged commit 7ffc718 into apache:main Aug 5, 2025
42 checks passed

Elbehery deleted the iceberg-13630 branch August 5, 2025 21:26


		import org.junit.jupiter.api.Test;

		public class TestRewriteTablePathUtil {

Core, Spark: Preserve the relative path in RewriteTablePathUtil on staging #13645

Core, Spark: Preserve the relative path in RewriteTablePathUtil on staging #13645

Uh oh!

Conversation

Elbehery commented Jul 23, 2025

Uh oh!

Elbehery commented Jul 23, 2025

Uh oh!

dramaticlly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dramaticlly left a comment

Choose a reason for hiding this comment

Uh oh!

dramaticlly commented Jul 23, 2025

Uh oh!

Elbehery commented Jul 24, 2025

Uh oh!

pvary commented Jul 24, 2025

Uh oh!

hpinca98 commented Jul 24, 2025

Uh oh!

Elbehery commented Jul 24, 2025

Uh oh!

Elbehery commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dramaticlly commented Jul 24, 2025

Uh oh!

Elbehery commented Jul 24, 2025

Uh oh!

Elbehery commented Jul 24, 2025

Uh oh!

ebyhr commented Jul 25, 2025

Uh oh!

Elbehery commented Jul 25, 2025

Uh oh!

Elbehery commented Jul 25, 2025

Uh oh!

szehon-ho left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dramaticlly Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szehon-ho Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elbehery Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Elbehery commented Jul 24, 2025 •

edited

Loading

szehon-ho left a comment •

edited

Loading

dramaticlly Jul 29, 2025 •

edited

Loading

stevenzwu Jul 30, 2025 •

edited

Loading

szehon-ho Jul 28, 2025 •

edited

Loading

Elbehery Aug 1, 2025 •

edited

Loading