Python
Follow me on
Sumit Khanna for more updates
Comprehensive Guide on PySpark
Framework
PySpark is the Python API for Apache Spark, an open-source, distributed computing system that provides an
interface for programming entire clusters with implicit data parallelism and fault tolerance. PySpark allows
Python developers to harness the simplicity of Python and the power of Apache Spark to process large
datasets efficiently.
Cheat Sheet Table
Method Name Definition
Returns a DataFrameReader that provides methods for reading data from
spark.read
various data sources.
df.show Displays the content of the DataFrame in a tabular format.
df.select Selects a set of columns from a DataFrame.
df.filter Filters rows that meet a specified condition.
df.groupBy Groups the DataFrame using the specified columns.
df.agg Applies aggregate functions to the DataFrame.
df.join Joins two DataFrames based on a common column(s).
df.withColumn Adds a new column or replaces an existing column in the DataFrame.
df.drop Drops specified columns from the DataFrame.
df.distinct Returns a new DataFrame with distinct rows.
df.union Returns a new DataFrame containing union of rows from two DataFrames.
Returns a new DataFrame containing intersection of rows from two
df.intersect
DataFrames.
Method Name Definition
Returns a new DataFrame containing rows that are in one DataFrame but
df.except
not the other.
df.sort Sorts the DataFrame based on specified columns.
df.orderBy Orders the DataFrame based on specified columns.
df.cache Caches the DataFrame in memory for quicker access.
df.persist Persists the DataFrame with specified storage level.
df.repartition Repartitions the DataFrame into specified number of partitions.
df.coalesce Reduces the number of partitions in the DataFrame.
df.na.drop Drops rows containing null values.
df.na.fill Fills null values with specified value.
df.na.replace Replaces specified values in the DataFrame.
df.describe Computes basic statistics for numeric columns.
df.summary Provides a summary of the DataFrame.
df.crosstab Computes a cross-tabulation of two columns.
df.stat.corr Computes correlation between two columns.
df.stat.cov Computes covariance between two columns.
df.stat.freqItems Finds frequent items for columns.
df.stat.approxQuantile Calculates approximate quantiles.
Returns a stratified sample without replacement based on the fraction given
df.stat.sampleBy
on each stratum.
df.withColumnRenamed Renames an existing column in the DataFrame.
df.dropDuplicates Drops duplicate rows based on specified columns.
df.sample Returns a sampled subset of the DataFrame.
df.randomSplit Randomly splits the DataFrame into multiple DataFrames.
df.take Returns the first n rows of the DataFrame.
df.head Returns the first row or n rows of the DataFrame.
df.first Returns the first row of the DataFrame.
df.collect Returns all rows as a list of Row objects.
Method Name Definition
df.toPandas Converts the DataFrame to a Pandas DataFrame.
Interface for saving the content of the DataFrame out into external storage
df.write
systems.
df.schema Returns the schema of the DataFrame.
df.dtypes Returns a list of tuples containing the column name and data type.
df.columns Returns a list of column names.
df.count Returns the number of rows in the DataFrame.
df.isEmpty Returns whether the DataFrame is empty.
df.sample Returns a random sample of rows from the DataFrame.
df.fillna Fills null values with specified value.
df.replace Replaces values with specified values.
df.dropna Drops rows with null values.
Computes aggregates with specified columns and aggregators, using cube
df.cube
computation.
Computes aggregates with specified columns and aggregators, using rollup
df.rollup
computation.
Explanation and Usage of Each Method
spark.read
Definition:
Returns a DataFrameReader that provides methods for reading data from various data sources.
Example:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("PySpark Guide").getOrCreate()
# Reading a CSV file
df = spark.read.csv("path/to/csvfile.csv", header=True, inferSchema=True)
df.show()
df.show
Definition:
Displays the content of the DataFrame in a tabular format.
Example:
# Show the first 20 rows of the DataFrame
df.show()
# Show the first 10 rows
df.show(10)
df.select
Definition:
Selects a set of columns from a DataFrame.
Example:
# Select specific columns
selected_df = df.select("column1", "column2")
selected_df.show()
df.filter
Definition:
Filters rows that meet a specified condition.
Example:
# Filter rows where the value in column1 is greater than 100
filtered_df = df.filter(df.column1 > 100)
filtered_df.show()
df.groupBy
Definition:
Groups the DataFrame using the specified columns.
Example:
# Group by column1 and calculate the count
grouped_df = df.groupBy("column1").count()
grouped_df.show()
df.agg
Definition:
Applies aggregate functions to the DataFrame.
Example:
from pyspark.sql.functions import avg
# Calculate the average value of column1
agg_df = df.agg(avg("column1"))
agg_df.show()
df.join
Definition:
Joins two DataFrames based on a common column(s).
Example:
# Join df1 and df2 on column1
joined_df = df1.join(df2, df1.column1 == df2.column1)
joined_df.show()
df.withColumn
Definition:
Adds a new column or replaces an existing column in the DataFrame.
Example:
from pyspark.sql.functions import lit
# Add a new column with a constant value
new_df = df.withColumn("new_column", lit("constant_value"))
new_df.show()
df.drop
Definition:
Drops specified columns from the DataFrame.
Example:
# Drop column1
dropped_df = df.drop("column1")
dropped_df.show()
df.distinct
Definition:
Returns a new DataFrame with distinct rows.
Example:
# Get distinct rows
distinct_df = df.distinct()
distinct_df.show()
df.union
Definition:
Returns a new DataFrame containing union of rows from two DataFrames.
Example:
# Union of df1 and df2
union_df = df1.union(df2)
union_df.show()
df.intersect
Definition:
Returns a new DataFrame containing intersection of rows from two DataFrames.
Example:
# Intersection of df1 and df2
intersect_df = df1.intersect(df2)
intersect_df.show()
df.except
Definition:
Returns a new DataFrame containing rows that are in one DataFrame but not the other.
Example:
# Except of df1 and df2
except_df = df1.except(df2)
except_df.show()
df.sort
Definition:
Sorts the DataFrame based on specified columns.
Example:
# Sort by column1
sorted_df = df.sort("column1")
sorted_df.show()
df.orderBy
Definition:
Orders the DataFrame based on specified columns.
Example:
# Order by column1
ordered_df = df.orderBy("column1")
ordered_df.show()
df.cache
Definition:
Caches the DataFrame in memory for quicker access.
Example:
# Cache the DataFrame
df.cache()
df.show()
df.persist
Definition:
Persists the DataFrame with specified storage level.
Example:
from pyspark import StorageLevel
# Persist the DataFrame
df.persist(StorageLevel.MEMORY_ONLY)
df.show()
df.repartition
Definition:
Repartitions the DataFrame into specified number of partitions.
Example:
# Repartition to 10 partitions
repartitioned_df = df.repartition(10)
repartitioned_df.show()
df.coalesce
Definition:
Reduces the number of partitions in the DataFrame.
Example:
# Coalesce to 2 partitions
coalesced_df = df.coalesce(2)
coalesced_df.show()
df.na.drop
Definition:
Drops rows containing null values.
Example:
# Drop rows with any null values
df_na_dropped = df.na.drop()
df_na_dropped.show()
df.na.fill
Definition:
Fills null values with specified value.
Example:
# Fill null values with 0
df_na_filled = df.na.fill(0)
df_na_filled.show()
df.na.replace
Definition:
Replaces specified values in the DataFrame.
Example:
# Replace 'old_value' with 'new_value'
df_na_replaced = df.na.replace('old_value', 'new_value')
df_na_replaced.show()
df.describe
Definition:
Computes basic statistics for numeric columns.
Example:
# Describe the DataFrame
df.describe().show()
df.summary
Definition:
Provides a summary of the DataFrame.
Example:
# Summary of the DataFrame
df.summary().show()
df.crosstab
Definition:
Computes a cross-tabulation of two columns.
Example:
# Crosstab of column1 and column2
crosstab_df = df.crosstab("column1", "column2")
crosstab_df.show()
df.stat.corr
Definition:
Computes correlation between two columns.
Example:
# Correlation between column1 and column2
correlation = df.stat.corr("column1", "column2")
print(correlation)
df.stat.cov
Definition:
Computes covariance between two columns.
Example:
# Covariance between column1 and column2
covariance = df.stat.cov("column1", "column2")
print(covariance)
df.stat.freqItems
Definition:
Finds frequent items for columns.
Example:
# Frequent items for column1
freq_items_df = df.stat.freqItems(["column1"])
freq_items_df.show()
df.stat.approxQuantile
Definition:
Calculates approximate quantiles.
Example:
# Approximate quantiles for column1
quantiles = df.stat.approxQuantile("column1", [0.25, 0.5, 0.75], 0.01)
print(quantiles)
df.stat.sampleBy
Definition:
Returns a stratified sample without replacement based on the fraction given on each stratum.
Example:
# Stratified sample by column1
sampled_df = df.stat.sampleBy("column1", fractions={"A": 0.1, "B": 0.2}, seed=0)
sampled_df.show()
df.withColumnRenamed
Definition:
Renames an existing column in the DataFrame.
Example:
# Rename column1 to new_column1
renamed_df = df.withColumnRenamed("column1", "new_column1")
renamed_df.show()
df.dropDuplicates
Definition:
Drops duplicate rows based on specified columns.
Example:
# Drop duplicates based on column1
deduped_df = df.dropDuplicates(["column1"])
deduped_df.show()
df.sample
Definition:
Returns a sampled subset of the DataFrame.
Example:
# Sample 10% of the DataFrame
sampled_df = df.sample(fraction=0.1, seed=0)
sampled_df.show()
df.randomSplit
Definition:
Randomly splits the DataFrame into multiple DataFrames.
Example:
# Split the DataFrame into training (70%) and testing (30%) sets
train_df, test_df = df.randomSplit([0.7, 0.3], seed=0)
train_df.show()
test_df.show()
df.take
Definition:
Returns the first n rows of the DataFrame.
Example:
# Take the first 5 rows
rows = df.take(5)
for row in rows:
print(row)
df.head
Definition:
Returns the first row or n rows of the DataFrame.
Example:
# Head of the DataFrame
head_row = df.head()
print(head_row)
df.first
Definition:
Returns the first row of the DataFrame.
Example:
# First row of the DataFrame
first_row = df.first()
print(first_row)
df.collect
Definition:
Returns all rows as a list of Row objects.
Example:
# Collect all rows
all_rows = df.collect()
for row in all_rows:
print(row)
df.toPandas
Definition:
Converts the DataFrame to a Pandas DataFrame.
Example:
# Convert to Pandas DataFrame
pandas_df = df.toPandas()
print(pandas_df.head())
df.write
Definition:
Interface for saving the content of the DataFrame out into external storage systems.
Example:
# Write DataFrame to CSV
df.write.csv("path/to/output.csv")
df.schema
Definition:
Returns the schema of the DataFrame.
Example:
# Schema of the DataFrame
schema = df.schema
print(schema)
df.dtypes
Definition:
Returns a list of tuples containing the column name and data type.
Example:
# Data types of the DataFrame
dtypes = df.dtypes
print(dtypes)
df.columns
Definition:
Returns a list of column names.
Example:
# Columns of the DataFrame
columns = df.columns
print(columns)
df.count
Definition:
Returns the number of rows in the DataFrame.
Example:
# Count of rows
row_count = df.count()
print(row_count)
df.isEmpty
Definition:
Returns whether the DataFrame is empty.
Example:
# Check if DataFrame is empty
is_empty = df.isEmpty()
print(is_empty)
Additional Examples
Example 1: Word Count
from pyspark.sql.functions import explode, split
text_df = spark.read.text("path/to/textfile.txt")
word_df = text_df.select(explode(split(text_df.value, " ")).alias("word"))
word_count_df = word_df.groupBy("word").count()
word_count_df.show()
Example 2: Join DataFrames
df1 = spark.read.csv("path/to/csvfile1.csv", header=True, inferSchema=True)
df2 = spark.read.csv("path/to/csvfile2.csv", header=True, inferSchema=True)
joined_df = df1.join(df2, df1.id == df2.id, "inner")
joined_df.show()
Example 3: Aggregations
from pyspark.sql.functions import sum, avg
agg_df = df.groupBy("category").agg(sum("sales").alias("total_sales"), avg("sales").alias("average_sal
agg_df.show()
Example 4: Handling Missing Data
# Fill missing values
filled_df = df.na.fill({"column1": 0, "column2": "unknown"})
filled_df.show()
# Drop rows with missing values
dropped_df = df.na.drop()
dropped_df.show()
Example 5: Creating UDFs
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
def my_upper(s):
return s.upper()
my_upper_udf = udf(my_upper, StringType())
df_with_udf = df.withColumn("upper_column", my_upper_udf(df.column1))
df_with_udf.show()
Example 6: Reading and Writing Parquet Files
# Reading Parquet file
parquet_df = spark.read.parquet("path/to/parquetfile.parquet")
parquet_df.show()
# Writing Parquet file
df.write.parquet("path/to/output.parquet")
Example 7: DataFrame Transformation
transformed_df = df.withColumn("new_column", df.column1 * 2)
transformed_df.show()
Example 8: Filtering Data
filtered_df = df.filter((df.column1 > 100) & (df.column2 == "value"))
filtered_df.show()
Example 9: Date and Time Operations
from pyspark.sql.functions import current_date, current_timestamp
date_df = df.withColumn("current_date", current_date()).withColumn("current_timestamp", current_timest
date_df.show()
Example 10: Working with JSON Data
# Reading JSON file
json_df = spark.read.json("
path/to/jsonfile.json")
json_df.show()
# Writing JSON file
df.write.json("path/to/output.json")
Example 11: SQL Queries
df.createOrReplaceTempView("table")
result_df = spark.sql("SELECT * FROM table WHERE column1 > 100")
result_df.show()
Example 12: Using Window Functions
from pyspark.sql.window import Window
from pyspark.sql.functions import rank
window_spec = Window.partitionBy("category").orderBy("sales")
ranked_df = df.withColumn("rank", rank().over(window_spec))
ranked_df.show()
Example 13: Pivoting Data
pivot_df = df.groupBy("category").pivot("year").sum("sales")
pivot_df.show()
Example 14: Unpivoting Data
unpivot_df = pivot_df.selectExpr("category", "stack(3, '2020', `2020`, '2021', `2021`, '2022', `2022`)
unpivot_df.show()
Example 15: Sampling Data
sampled_df = df.sample(fraction=0.1, seed=0)
sampled_df.show()
Example 16: Creating a DataFrame from RDD
rdd = spark.sparkContext.parallelize([(1, "a"), (2, "b"), (3, "c")])
rdd_df = spark.createDataFrame(rdd, ["id", "value"])
rdd_df.show()
Example 17: Reading Avro Files
avro_df = spark.read.format("avro").load("path/to/avrofile.avro")
avro_df.show()
Example 18: Writing Avro Files
df.write.format("avro").save("path/to/output.avro")
Example 19: Handling Complex Types
from pyspark.sql.functions import explode
complex_df = df.withColumn("exploded_column", explode(df.complex_column))
complex_df.show()
Example 20: Machine Learning with PySpark
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
# Prepare data for ML
assembler = VectorAssembler(inputCols=["column1", "column2"], outputCol="features")
ml_df = assembler.transform(df)
# Train a Linear Regression model
lr = LinearRegression(featuresCol="features", labelCol="label")
lr_model = lr.fit(ml_df)
# Make predictions
predictions = lr_model.transform(ml_df)
predictions.show()
Follow me on
Sumit Khanna for more updates
