Allow setting default ConnectedComponents configurations from spark config#563
Conversation
|
latm (= looks asymmetric to me) Why allow only the |
|
I like where you're going with this... |
Because I'm lazy and greedy. And I'm gauging interest in this kind of approach. |
cae580b to
633b3da
Compare
Didn't know lazy people actually propose new things ... 😂 |
|
@james-willis jfyi, you can use a Alternatively, just run |
|
Hey, and what about the Spark Connect feature in Graphframes? isn't it affected also by this change? |
The new interface here is just spark config so I don't think there is any need to change anything about the connect integration but if you think there is some change to be made please point me in the right direction. |
91dd0a0 to
c368a39
Compare
It isnt, there is no problem, it is just an alternative way of passing configs. |
|
@james-willis in the latest changes we introduced a new mixin (WithAlgorithmChoice) to avoid code repeating that was added to ConnectedComponents. So, instead of redefining the default you should probably just call a setter. |
|
Bogged down at work. will get back to this eventually. |
4fc31ce to
de50969
Compare
src/main/scala/org/apache/spark/sql/graphframes/GraphFramesConf.scala
Outdated
Show resolved
Hide resolved
SemyonSinchenko
left a comment
There was a problem hiding this comment.
LGTM! Nice work @james-willis !
What changes were proposed in this pull request?
Enable setting the connected components algorithm from spark configurations.
Why are the changes needed?
spark configurations are a standard way to control implementation details across a spark runtime. This pattern will help runtime environment owners control how their users use graphframes