GraphX
Graph Analytics in Spark
Ankur Dave
Graduate Student, UC Berkeley AMPLab
Joint work with Joseph Gonzalez, Reynold Xin, Daniel
Crankshaw, Michael Franklin, and Ion Stoica UC
BERKELEY
Machine Learning Landscape
Model &
Dependencies
Small & Dense Sparse Large & Dense
Architecture
MapReduce Graph-Parallel Parameter Server
Machine Learning Landscape
Model &
Dependencies
Small & Dense Sparse Large & Dense
GraphX
Architecture
Spark Dataflow
Framework Parameter Server
Graphs
Social Networks
Web Graphs
User-Item Graphs
Graph Algorithms
PageRank
Triangle Counting
Collaborative Filtering
Products
x
Users
Ratings f(j)
Users
f(i)
Products
Collaborative Filtering
f(3)
r13
f(1)
Product Factors
r14
User Factors
f(4)
r24
f(2)
r25 f(5)
X 2
T
f [i] = arg min rij w f [j] + ||w||22
w2Rd
j2Nbrs(i)
The Graph-Parallel Pattern
The Graph-Parallel Pattern
The Graph-Parallel Pattern
Many Graph-Parallel Algorithms
Collaborative Filtering Community Detection
Alternating Least Squares Triangle-Counting
Stochastic Gradient Descent K-core Decomposition
Tensor Factorization K-Truss
Structured Prediction Graph Analytics
Loopy Belief Propagation PageRank
Max-Product Linear Personalized PageRank
Programs Shortest Path
Gibbs Sampling Graph Coloring
Semi-supervised ML Classification
Graph SSL Neural Networks
CoEM
Modern Analytics
Link Table Hyperlinks PageRank Top 20 Pages
Title Link Title PR
Raw
Wikipedia
Com. PR..
<</ />>
</> Top Communities
XML
Editor Community User
Table Editor Graph Detection Community
Editor Title User Com.
Tables
Link Table Hyperlinks PageRank Top 20 Pages
Title Link Title PR
Raw
Wikipedia
Com. PR..
<</ />>
</> Top Communities
XML
Editor Community User
Table Editor Graph Detection Community
Editor Title User Com.
Graphs
Link Table Hyperlinks PageRank Top 20 Pages
Title Link Title PR
Raw
Wikipedia
Com. PR..
<</ />>
</> Top Communities
XML
Editor Community User
Table Editor Graph Detection Community
Editor Title User Com.
The GraphX API
Property Graphs
Vertex Property:
User Profile
Current PageRank Value
Edge Property:
Weights
Relationships
Timestamps
Creating a Graph (Scala)
type
VertexId
=
Long
Graph
val
vertices:
RDD[(VertexId,
String)]
=
sc.parallelize(List(
(1L,
Alice),
1 Alice
(2L,
Bob),
(3L,
Charlie)))
coworker
class
Edge[ED](
val
srcId:
VertexId,
val
dstId:
VertexId,
2 Bob
val
attr:
ED)
val
edges:
RDD[Edge[String]]
=
friend
sc.parallelize(List(
Edge(1L,
2L,
coworker),
Edge(2L,
3L,
friend)))
3 Charlie
val
graph
=
Graph(vertices,
edges)
Graph Operations (Scala)
class
Graph[VD,
ED]
{
//
Table
Views
-----------------------
def
vertices:
RDD[(VertexId,
VD)]
def
edges:
RDD[Edge[ED]]
def
triplets:
RDD[EdgeTriplet[VD,
ED]]
//
Transformations
-------------------------------------------
def
mapVertices[VD2](f:
(VertexId,
VD)
=>
VD2):
Graph[VD2,
ED]
def
mapEdges[ED2](f:
Edge[ED]
=>
ED2):
Graph[VD2,
ED]
def
reverse:
Graph[VD,
ED]
def
subgraph(epred:
EdgeTriplet[VD,
ED]
=>
Boolean,
vpred:
(VertexId,
VD)
=>
Boolean):
Graph[VD,
ED]
//
Joins
----------------------------------------
def
outerJoinVertices[U,
VD2]
(tbl:
RDD[(VertexId,
U)])
(f:
(VertexId,
VD,
Option[U])
=>
VD2):
Graph[VD2,
ED]
//
Computation
----------------------------------
def
mapReduceTriplets[A](
sendMsg:
EdgeTriplet[VD,
ED]
=>
Iterator[(VertexId,
A)],
mergeMsg:
(A,
A)
=>
A):
RDD[(VertexId,
A)]
Built-in Algorithms (Scala)
//
Continued
from
previous
slide
def
pageRank(tol:
Double):
Graph[Double,
Double]
def
triangleCount():
Graph[Int,
ED]
def
connectedComponents():
Graph[VertexId,
ED]
//
...and
more:
org.apache.spark.graphx.lib
}
PageRank Triangle Count Connected
Components
The triplets view
class
Graph[VD,
ED]
{
def
triplets:
RDD[EdgeTriplet[VD,
ED]]
}
class
EdgeTriplet[VD,
ED](
val
srcId:
VertexId,
val
dstId:
VertexId,
val
attr:
ED,
val
srcAttr:
VD,
val
dstAttr:
VD)
Graph
1 Alice RDD
coworker srcAttr dstAttr attr
triplets
Alice coworker Bob
2 Bob
Bob friend Charlie
friend
3 Charlie
The subgraph transformation
class
Graph[VD,
ED]
{
def
subgraph(epred:
EdgeTriplet[VD,
ED]
=>
Boolean,
vpred:
(VertexId,
VD)
=>
Boolean):
Graph[VD,
ED]
}
graph.subgraph(epred
=
(edge)
=>
edge.attr
!=
relative)
Graph Graph
Alice coworker Bob Alice coworker Bob
relative subgraph
friend friend
Charlie relative David Charlie David
The subgraph transformation
class
Graph[VD,
ED]
{
def
subgraph(epred:
EdgeTriplet[VD,
ED]
=>
Boolean,
vpred:
(VertexId,
VD)
=>
Boolean):
Graph[VD,
ED]
}
graph.subgraph(vpred
=
(id,
name)
=>
name
!=
Bob)
Graph Graph
Alice coworker Bob Alice
relative subgraph
relative
friend
Charlie relative David Charlie relative David
Computation with mapReduceTriplets
class
Graph[VD,
ED]
{
def
mapReduceTriplets[A](
upgrade to aggregateMessages
sendMsg:
EdgeTriplet[VD,
ED]
=>
Iterator[(VertexId,
A)],
in Spark 1.2.0
mergeMsg:
(A,
A)
=>
A):
RDD[(VertexId,
A)]
}
graph.mapReduceTriplets(
edge
=>
Iterator(
(edge.srcId,
1),
(edge.dstId,
1)),
_
+
_)
RDD
Graph vertex id degree
Alice Bob
Alice 2
coworker
mapReduceTriplets
Bob 2
relative
friend
Charlie 3
Charlie relative David
David 1
How GraphX Works
Encoding Property Graphs as RDDs
Vertex Routing Edge Table
Property Graph Table Table (RDD)
(RDD) (RDD)
Part. 1 A B
A A 1
2
B C A C
Machine 1
B B 1
B C
C D
ACut
VertexA
D
D C C 1
A D
A E
D D 1
2
Machine 2
A F
E E 2
F E E D
Part. 2 F F 2
E F
Graph System Optimizations
Specialized Vertex-Cuts Remote
Data-Structures Partitioning Caching / Mirroring
Message Combiners Active Set Tracking
PageRank Benchmark
EC2 Cluster of 16 x m2.4xLarge (8 cores) + 1GigE
Twitter Graph (42M Vertices,1.5B Edges) UK-Graph (106M Vertices, 3.7B Edges)
3500 9000
3000 8000
Runtime (Seconds)
7000
2500
6000
2000 5000
7x 18x
1500 4000
3000
1000
2000
500 1000
0 0
GraphX performs comparably to
state-of-the-art graph processing systems.
Future of GraphX
1. Language support
a) Java API: PR #3234
b) Python API: collaborating with Intel, SPARK-3789
2. More algorithms
a) LDA (topic modeling): PR #2388
b) Correlation clustering
c) Your algorithm here?
3. Speculative
a) Streaming/time-varying graphs
b) Graph databaselike queries
Thanks!
http://spark.apache.org/graphx
[email protected]
[email protected]
[email protected]
[email protected]
