Spark 4 - fix context facetory #3934
Conversation
| return Optional.empty(); | ||
| } | ||
| SparkSession sparkSession = queryExecution.sparkSession(); | ||
| SparkSession sparkSession = Spark4CompatUtils.getSparkSession(queryExecution); |
There was a problem hiding this comment.
might be a stupid question, having very limited knowledge in Java, but how does it work without import of Spark4CompatUtils? I tried to build locally with this commit, but getting error: cannot find symbol. (using commands ./buildDependencies.sh && ./gradlew publishToMavenLocal -Pscala.binary.version=2.13 to build required jar). Though importing like this import io.openlineage.spark.agent.Spark4CompatUtils; did not work for me also, still can't find symbol. Maybe something missing in gradle config?
Another question, is that correct that Utils are defined as part of tests, but used in main source code here?
OpenLineage/integration/spark/app/src/test/java/io/openlineage/spark/agent/Spark4CompatUtils.java
OpenLineage/integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/ContextFactory.java
Does it work like this?
There was a problem hiding this comment.
@pawel-big-lebowski I tried to move the whole Spark4CompatUtils file to src instead of tests and it helped to build both code and tests, would that be a good way to fix build? I can commit such change here if you agree with it
Signed-off-by: Pawel Leszczynski <[email protected]>
Signed-off-by: Dominik Dębowczyk <[email protected]>
cbb0409 to
9cc79d8
Compare
Signed-off-by: Dominik Dębowczyk <[email protected]>
Closes #3931