Prerequisite:
1. Anaconda installation
2. Java 1.8
3. Spark 2.3
1. Verify the installation
Log in to EC2 user as the root user
a. Run the following command to verify the anaconda installation
conda --version
b. Run the following command to verify Java 8 installation
java -version
c. Run the following command to verify Spark2 installation
spark2-shell --version
2. Configure Jupyter(One-time process)
Jupyter notebook is already installed in our CDH instance with Anaconda. However, we need to
configure it before we can actually run it.
a. Run the following command to generate jupyter configuration file.
jupyter notebook --generate-config
You can see jupyter_notebook_config.py has been created inside the /root/.jupyter directory.
b. Allow access to your remote Jupyter server
Open the jupyter_notebook_config.py file
vi .jupyter/jupyter_notebook_config.py
Press “I” to get into the insert mode
Copy and paste the below two lines as shown in the screenshot.
c.NotebookApp.allow_origin = ' *' #allow all origins
c.NotebookApp.ip = '0.0.0.0' # listen on all IPs
To exit
> Press ‘Esc’ > Type :wq! > Hit Enter
3. Now, use the following command to run the Jupyter Notebook
jupyter notebook --port 7861 --allow-root
You will see the Jupyter server as started.
Note: Don’t kill this process, keep it running until you are done using your Jyputer notebook.
To stop/close your Jyputer Notebook open this terminal and press Ctrl + C
You can see it has given you a URL where your Jupyter notebook is running. In my case the
URL is
http://(ip-10-0-0-228.ec2.internal or 127.0.0.1):7861/?token=5935c2bdf10c2212013af0b4e9e7cd98cd22ca03eeaf4005
In order to open the Jupyter notebook on your browser, replace the highlighted part with the
Public IP of your EC2 Instance.
In my case, the final URL will be-
http://3.89.129.54:7861/?token=5935c2bdf10c2212013af0b4e9e7cd98cd22ca03eeaf4005
Now, you can open your web browser and copy-paste your final URL to open the Jupyter
notebook. You should be able to see the Jupyter home page.
4. PySpark on Jupiter
In order to run PySpark on Jupyter notebook, you need to paste and run the following
few lines of codes in your every PySpark Notebook.
import os
import sys
os.environ["PYSPARK_PYTHON"] = "/opt/cloudera/parcels/Anaconda/bin/python"
os.environ["JAVA_HOME"] = "/usr/java/jdk1.8.0_161/jre"
os.environ["SPARK_HOME"] = "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/"
os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.10.6-src.zip")
sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")
Note: The value for the above environment variables may be different in your case, we suggest
you verify these values before using it. Please follow the below-mentioned steps for the same.
i. Anaconda and Python
os.environ["PYSPARK_PYTHON"] = "/opt/cloudera/parcels/Anaconda/bin/python"
Make sure Anaconda is installed in the /opt/cloudera/parcels/ director. To Verify please check
whether the Anaconda directory is present under /opt/cloudera/parcels/ path.
ls /opt/cloudera/parcels/
ii. Java Home
os.environ["JAVA_HOME"] = "/usr/java/jdk1.8.0_161/jre"
Run the following command to get your Java_Home path
echo $JAVA_HOME
Hence /usr/java/jdk1.8.0_161/jre will be the value of our JAVA_HOME variable
iii. Spark Home
For the Spark home path in the following line-
os.environ["SPARK_HOME"] = "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/"
The name of this Spark file(highlighted above) be exactly the same as it is present under your
/opt/cloudera/parcels/
Run the following command to check/verify and replace it in case you have a different
version/distribution installed.
ls /opt/cloudera/parcels/
iv. Version of py4j and pyspark.zip file
sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.10.6-src.zip")
sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")
After you have identified your correct Spark home path this step is to verify the version of the
py4j file.
For that run the following command(make sure to modify this command according to your Spark
home identified in the 3rd point)
ls /opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib
You should be able to see the py4j and p
yspark file. Make sure you modify this according to
your instance in the above code.
5. Test your PySpark setup
a. Open a new Jupyter Notebook
b. Copy-paste the environment variable that you have finalized in the 4th point into a cell.
(You need to do it for every pySpark notebook which you will create)
c. Run the cell you should not see any error.
d. Now, let’s initialize the SparkContext object. Copy-paste the following code in a new cell
> Run the cell
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("jupyter_Spark").setMaster("yarn-client")
sc = SparkContext(conf=conf)
sc
You should be able see the following output
This means your pySpark is working fine.
6. Closing Jupyter Notebook
To stop the notebook open the terminal/Putty window where the Jupyter process is running and
press Ctrl + C.
Enter ‘y’ to shut down the cluster.
