Add dependency override for shapeless #1453
Conversation
Resolves a compiletime/runtime mismatch when Spark is in user classpath
build.sbt
Outdated
| "com.github.javaparser" % "javaparser-symbol-solver-core" % versions.javaparser, | ||
| "org.scalamock" %% "scalamock" % "4.4.0" % "test" | ||
| ), | ||
| dependencyOverrides += "com.chuusai" %% "shapeless" % "2.3.2", |
There was a problem hiding this comment.
Do we need to pick a specific version based on what matches Spark?
There was a problem hiding this comment.
yeah, I think so. I wish there were a way to tell sbt, "use the version that you get from Spark, not a later version" but there isn't as far as I know.
There was a problem hiding this comment.
Is there no like "provided" dependency? compileOnly or whatever?
There was a problem hiding this comment.
There is provided... but I don't see how that would solve this issue?
There was a problem hiding this comment.
Oh, by "compiletime/runtime" I mean cell compile time and cell runtime. The issue is that the shapeless in polynote-kernel-assembly (which needs to be there in the absence of Spark) gets used during compile time for some reason, and its macros are what get executed. But during runtime, Spark's shapeless gets used, which doesn't have the code that the macro referred to.
Resolves a compiletime/runtime mismatch when Spark is in user classpath