⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.85%. Comparing base (bc487ef) to head (71040e0).
Report is 31 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##           master     #598      +/-   ##
==========================================
- Coverage   91.43%   89.85%   -1.58%     
==========================================
  Files          18       21       +3     
  Lines         829     1065     +236     
  Branches       52      119      +67     
==========================================
+ Hits          758      957     +199     
- Misses         71      108      +37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

To check the result:

1. ./build/sbt
2. root / publishLocal
3. connect / publishLocal
4. databricksConnect / publishLocal

With a single command:

./build/sbt "root/publishLocal; connect/publishLocal; databricksConnect/publishLocal"

Is there an issue for why we need databricks-specific logic? should we link that explanation here in the PR?

Is there an issue for why we need databricks-specific logic? should we link that explanation here in the PR?

mrpowers-io/tsumugi-spark#55 (comment)

For unknown for me reason, databricks engineers are using an own shading rule that is different from an OSS one.

@WeichenXu123 Hello! Do you know, can we use the word "databricks" in the name of published artifacts (graphframes-databricks-connect) without violation of the trademark? Is there a better way to allow databricks users to use graphframes on "shared access" clusters?

cc: @rjurney

LGTM!

I'm waiting for the approve from @dmatrix about can we use the word "databricks" in the name of released artifacts, code and documentation.

@WeichenXu123 Hello! Do you know, can we use the word "databricks" in the name of published artifacts (graphframes-databricks-connect) without violation of the trademark? Is there a better way to allow databricks users to use graphframes on "shared access" clusters?

cc: @rjurney

I suggest to avoid using "databricks" in the name, but using name like graphframes-spark-connect.

but I think supporting Spark connect is not sufficient to make it work on Databricks shared cluster, we also need to whitelist a bunch of JVM methods in the lib for running on Databricks shared cluster (and needs our security team's approval for doing that)

Does the API change from protobuf.Any to Array[Byte] in Spark 4 fix the shading/binary compatibility issue, at least in Spark 4+? Might be a good compromise if it does to prevent a dedicated artifact for a closed source fork

Does the API change from protobuf.Any to Array[Byte] in Spark 4 fix the shading/binary compatibility issue, at least in Spark 4+? Might be a good compromise if it does to prevent a dedicated artifact for a closed source fork

I did not try but for me it looks like this change in Spark 4.0.x should resolve the mentioned problem.

@james-willis @Kimahriman @rjurney @dmatrix

Hello! After some discussions with engineers from Databricks and based on your comments I did the following updates:

1. I removed databricksConnect as a separate artifact
2. I changed the proto shading pattern from literal to variable
3. Anyone who wants a binary compatibility with Databricks Runtimes can build GraphFrames Connect with a command ./build/sbt -Dvendor.name=dbx connect/assembly

Please, take a look on the PR when you have time.
P.S. As I already mentioned, I'm not an experienced person in the topic of publishing anything to the sonatype central and this is my first experience with it. Please keep this in mind during the review. Thanks!

cc: @WeichenXu123

I looked into what sedona is doing here and it seems that it shades in its own version of dependencies when databricks has some kind of conflict:

https://github.com/apache/sedona/blob/e8c0d5af41670757c6d319d9c0d393b0154ec929/common/pom.xml#L143

will making users compile from source for dbx impede adoption?

Running Spark Connect plugins on Databricks is not an easy task anyway. I think we should be fine. Spark 4.0 should resolve this.