At the moment PySpark GraphFrames is relying on both py4j and connect. But we can actually use connect utils and protobuf instead of py4j for PySpark Classic too. As well, as we can create a pure Java API (and maybe other APIs, like .NET, GoLang, etc.).

Benefits:

• simple testing
• single entry point for all the APIs
• decoupling of the Spark Scala core from APIs
• not necessary to maintain Java-compatible API for py4j (that is terrible for calling of Scala)

cc: @rjurney