> Retrieving RDD Information
> Reshaping Data
Basic Information Re ducing
Python For Data Science
>>> rdd.getNumPartitions() #List the number of partitions
>>> rdd.reduceByKey(lambda x,y : x+y).collect() #Merge the
[('a',9),('b',2)]
>>> rdd.reduce(lambda a, b: a + b) #Merge the rdd values
rdd values for
each key
>>> rdd.count() #Count RDD instances 3
('a',7,'a',2,'b',2)
>>> rdd.countByKey() #Count RDD instances by key
PySpark RDD Cheat Sheet defaultdict(<type 'int'>,{'a':2,'b':1})
>>> rdd.countByValue() #Count RDD instances by value
Grouping by
>>> rdd3.groupBy(lambda x: x % 2) #Return RDD of grouped values
defaultdict(<type 'int'>,{('b',2):1,('a',2):1,('a',7):1})
.mapValues(list)
>>> rdd.collectAsMap() #Return (key,value) pairs as a
dictionary
Learn PySpark RDD online at www.DataCamp.com .collect()
{'a': 2,'b': 2}
>>> rdd.groupByKey() #Group rdd by key
>>> rdd3.sum() #Sum of RDD elements 4950
.mapValues(list)
>>> sc.parallelize([]).isEmpty() #Check whether RDD is empty
.collect()
True
[('a',[7,2]),('b',[2])]
Aggregating
Spark S ummary >>> seqOp = (lambda x,y: (x[0]+y,x[1]+1))
>>> combOp = (lambda x,y:(x[0]+y[0],x[1]+y[1]))
>>> r dd3.max() #Maximum value of RDD elements
#Aggregate RDD elements of each
partition and then the results
99
>>> rdd3.aggregate((0,0),seqOp,combOp)
PySpark is the Spark Python API that exposes
>>> r dd3.min() #Minimum value of RDD elements
(4950,100)
the Spark programming model to Python. #Aggregate values of each RDD key
>>> rdd3.mean() #Mean value of RDD elements
>>> rdd.aggregateByKey((0,0),seqop,combop).collect()
49.5
[('a',(9,2)), ('b',(2,1))]
>>> rdd3.stdev() #Standard deviation of RDD elements
#Aggregate the elements of each
partition, and then the results
28.866070047722118
>>> rdd3.fold(0,add)
>>> rdd3.variance() #Compute variance of RDD elements
> Initializing Spark 833.25
4950
#Merge the values for each key
>>> rdd3.histogram(3) #Compute histogram by bins
>>> rdd.foldByKey(0, add).collect()
([0,33,66,99],[33,33,34])
SparkC ontext >>> rdd3.stats() #Summary statistics (count, mean, stdev, x &
ma min)
[('a',9),('b',2)]
#Create tuples of RDD elements by
applying a function
>>> rdd3.keyBy(lambda x: x+x).collect()
>>> from pyspark import SparkContext
>>> sc = SparkContext(master = 'local[2]')
> Applying Functions
Inspect SparkContext > Mathematical Operations
A
# pply a function to each RDD element
>>> sc .version #Retrieve SparkContext version
>>> rdd.map(lambda x: x+(x[1],x[0])).collect()
>>> rdd.subtract(rdd2).collect() #Return each rdd value not contained in rdd 2
>>> sc.pythonVer #Retrieve Python version
[('a',7,7,'a'),('a',2,2,'a'),('b',2,2,'b')]
[('b',2),('a',7)]
>>> sc.master #Master URL to connect to
#Apply a function to each RDD element and flatten the result
#Return each (key,value) pair of rdd2 with no matching key in rd d
>>> str(sc.sparkHome) #Path where Spark is installed on worker nodes
>>> rdd5 = rdd.flatMap(lambda x: x+(x[1],x[0]))
>>> rdd2.subtractByKey(rdd).collect()
>>> str(sc.sparkUser()) #Retrieve name of the Spark User running SparkContext
>>> rdd5.collect()
[('d', 1)]
>>> sc.appName #Return application name
['a',7,7,'a','a',2,2,'a','b',2,2,'b']
>>> rdd.cartesian(rdd2).collect() #Return the Cartesian product of rdd and rdd 2
>>> sc.applicationId #Retrieve application ID
#Apply a flatMap function to each (key,value)
pair of g g
rdd4 without chan in s
the key
>>> sc.defaultParallelism #Return default level of parallelism
>>> rdd4.flatMapValues(lambda x: x).collect()
>>> sc.defaultMinPartitions #Default minimum number of partitions for
RDDs
[('a','x'),('a','y'),('a','z'),('b','p'),('b','r')]
> Sort
C onfiguration
>>> from pyspark import SparkConf , SparkContext
> Selecting Data >>> rdd2.sortBy(lambda x: x[1]).collect() #Sort RDD by given function
[('d',1),('b',1),('a',2)]
>>> conf = (SparkConf()
>>> rdd2.sortByKey().collect() #Sort (key, value) RDD by key
.setMaster("local")
Getting [('a',2),('b',1),('d',1)]
.setAppName("My app")
.set("spark.executor.memory", "1g"))
>>> rdd.collect() #Return a list with all RDD elements
[('a', 7), ('a', 2), ('b', 2)]
>>> sc = SparkContext(conf = conf)
>>> rdd.take(2) #Take first 2 RDD elements
[('a', 7), ('a', 2)]
Using The Shell >>> rdd.first() #Take first RDD element
> Repartitioning
('a', 7)
>>> rdd.top(2) #Take top 2 RDD elements
In the PySpark shell, a special interpreter-aware SparkContext is already
created in the variable called sc. >>> r dd.repartition(4) #New RDD with 4 partitions
[('b', 2), ('a', 7)]
>>> rdd.coalesce(1) #Decrease the number of partitions in the RDD to 1
$ ./bin/spark-shell --master local[2]
$ ./bin/pyspark --master local[4] --py-files d .
co e py
Samplin g
>>> rdd3.sample(False, 0.15, 81).collect() #Return sampled subset of rdd3
Set which master the context connects to with the --master argument, and
add Python .zip, .egg or .py files to the
[3,4,27,31,40,41,42,43,60,76,79,80,86,97]
runtime path by passing a
comma-separated list to --py-files.
Filtering > Saving
>>> rdd.filter(lambda x: "a" in x).collect()
#Filter the RDD
[('a',7),('a',2)]
>>> r dd.saveAsTextFile("rdd.txt")
> Loading Data >>> rdd5.distinct().collect() #Return distinct RDD values
['a',2,'b',7]
>>> rdd.saveAsHadoopFile("hdfs://namenodehost/parent/child",
’org.apache.hadoop.mapred.TextOutputFormat')
>>> rdd.keys().collect() #Return (key,value) RDD's keys
Para e ll lized Collections ['a', 'a', 'b']
>>> r dd = sc.parallelize([('a',7),('a',2),('b',2)])
> Stopping SparkContext
>>> rdd2 = sc.parallelize([('a',2),('d',1),('b',1)])
>>> rdd3 = sc.parallelize(range(100))
> Iterating .
>>> sc stop()
>>> rdd4 = sc.parallelize([("a",["x","y","z"]),
("b",["p", "r"])])
>>> def g(x): print(x)
>>> rdd.foreach(g) #Apply a function to all RDD elements
External Data ('a', 7)
('b', 2)
> Execution
('a', 2)
Rea d either one text file from HDFS, a local file system or or any
Hadoop-supported file system URI with textFile(),
or read in a directory
of text files with wholeTextFiles() $ ./bin/spark-submit / / / h / .
examples src main pyt on pi py
F .
>>> text ile = sc text ile( F "/my/directory/*.txt")
>>> textFile2 = sc.wholeTextFiles("/my/directory/")
Learn Data Skill s Online at www.DataCamp.com
