University of Mysore

Mysore University School of Engineering

Manasa Gangotri campus, Mysore 570006
(Approved by AICTE, New Delhi)
Department of Computer Science And Design

BIG DATA ANALYTICS (21AD62) Lab Manual

DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND DATA SCIENCE
(AI&DS)

VISION

➢ To achieve high quality in technical education that provides the skills and attitude to adapt to
the global needs of the Information Technology sector, through academic and research
excellence.

MISSION

➢ To equip the students with the cognizance for problem solving and to improve the teaching
learning pedagogy by using innovative techniques.
➢ To strengthen the knowledge base of the faculty and students with motivation towards
possession of effective academic skills and relevant research experience.
➢ To promote the necessary moral and ethical values among the engineers, for the betterment
of the society.
 PROGRAMME EDUCATIONAL OBJECTIVES (PEOs)

PEO1: PROFESSIONALISM & CITIZENSHIP

To create and sustain a community of learning in which students acquire knowledge and learn
to apply it professionally with due consideration for ethical, ecological and economic issues.

PEO2: TECHNICAL ACCOMPLISHMENTS

To provide knowledge-based services to satisfy the needs of society and the industry by
providing hands on experience in various technologies in core field.

PEO3: INVENTION, INNOVATION AND CREATIVITY

To make the students to design, experiment, analyze, interpret in the core field with the help
of other multi-disciplinary concepts wherever applicable.

PEO4: PROFESSIONAL DEVELOPMENT

To educate the students to disseminate research findings with good soft skills and become a
successful entrepreneur.

PEO5: HUMAN RESOURCE DEVELOPMENT

To graduate the students in building national capabilities in technology, education and

research.
 PROGRAM SPECIFIC OUTCOMES (PSOs)

After the completion of the course, B.E Artificial Intelligence and Data Science (AI&DS), the
graduates will have the following Program Specific Outcomes:

1. Fundamentals and critical knowledge of the Computer System: Able to understand the
working principles of the computer System and its components, Apply the knowledge to
build, asses, and analyze the software and hardware aspects of it.

2. The comprehensive and Applicative knowledge of Software Development:

Comprehensive skills of Programming Languages, Software process models, methodologies,
and able to plan, develop, test, analyze, and manage the software and hardware intensive
systems in heterogeneous platforms individually or working in teams.

3. Applications of Computing Domain & Research: Able to use the professional, managerial,
interdisciplinary skill set, and domain specific tools in development processes, identify the
research gaps, and provide innovative solutions to them.
 PROGRAM OUTCOMES (POs)
Engineering Graduates should possess the following:

1. Engineering knowledge: Apply the knowledge of mathematics, science, engineering

fundamentals, and an engineering specialization to the solution of complex engineering
problems.

2. Problem analysis: Identify, formulate, review research literature, and analyze complex
engineering problems reaching substantiated conclusions using first principles of
mathematics, natural sciences, and engineering sciences.

3. Design / development of solutions: Design solutions for complex engineering problems

and design system components or processes that meet the specified needs with appropriate
consideration for the public health and safety, and the cultural, societal, and environmental
considerations.

4. Conduct investigations of complex problems: Use research-based knowledge and

research methods including design of experiments, analysis and interpretation of data, and
synthesis of the information to provide valid conclusions.

5. Modern tool usage: Create, select, and apply appropriate techniques, resources, and
modern engineering and IT tools including prediction and modeling to complex
engineering activities with an understanding of the limitations.

6. The engineer and society: Apply reasoning informed by the contextual knowledge to
assess societal, health, safety, legal and cultural issues and the consequent responsibilities
relevant to the professional engineering practice.

7. Environment and sustainability: Understand the impact of the professional engineering

solutions in societal and environmental contexts, and demonstrate the knowledge of, and
need for sustainable development.

8. Ethics: Apply ethical principles and commit to professional ethics and responsibilities and
norms of the engineering practice.

9. Individual and team work: Function effectively as an individual, and as a member or

leader in diverse teams, and in multidisciplinary settings.

10. Communication: Communicate effectively on complex engineering activities with the

engineering community and with society at large, such as, being able to comprehend and
write effective reports and design documentation, make effective presentations, and give
and receive clear instructions.

11. Project management and finance: Demonstrate knowledge and understanding of the
engineering and management principles and apply these to one’s own work, as a member
and leader in a team, to manage projects and in multi-disciplinary environments.

12. Life- long learning: Recognize the need for, and have the preparation and ability to engage
in independent and life-long learning in the broadest context of technological change.
 GENERAL LABORATORY INSTRUCTIONS

1. Students are advised to come to the laboratory at least 5 minutes before (to the starting time),
those who come after 5 minutes will not be allowed into the lab.
2. Plan your task properly much before to the commencement, come prepared to the lab with the
synopsis / program / experiment details.
3. Student should enter into the laboratorywith:
a. Laboratory observation notes with all the details (Problem statement, Aim, Algorithm,
Procedure, Program, Expected Output, etc.,) filled in for the lab session.
b. Laboratory Record updated up to the last session experiments and other utensils (if any) needed
in the lab.
c. Proper Dress code and Identity card.
4. Sign in the laboratory login register, write the TIME-IN, and occupy the computer system allotted
to you by the faculty.
5. Execute your task in the laboratory, and record the results / output in the lab observation note
book, and get certified by the concerned faculty.
6. All the students should be polite and cooperative with the laboratory staff, must maintain the
discipline and decency in the laboratory.
7. Computer labs are established with sophisticated and high-end branded systems, which should be
utilized properly.
8. Students must keep their mobile phones in SWITCHED OFF mode during the lab sessions.
Misuse of the equipment, misbehaviors with the staff and systems etc., will attract severe
punishment.
9. Students must take the permission of the faculty in case of any urgency to go out; if anybody
found loitering outside the lab / class without permission during working hours will be treated
seriously and punished appropriately.
10. Students should LOG OFF/ SHUT DOWN the computer system before he/she leaves the lab after
completing the task (experiment) in all aspects. He/she must ensure the system / seat is kept
properly.

HEAD OF THE DEPARTMENT

COURSE OBJECTIVES:
The objectives of this course are,
1. Understand fundamentals of Big Data analytics.
2. Investigate Hadoop framework and Hadoop Distributed File system.
3. Illustrate the concepts of NoSQL using MongoDB and Cassandra for Big Data.
4. Demonstrate the MapReduce programming model to process the big data along with Hadoop
tools.
5. Use Machine Learning algorithms for real world big data.
6. Analyze web contents and Social Networks to provide analytics with relevant
visualization tools.

List of Experiments
1. Install, configure and run python, numPy and Pandas.
2. Install, configure and run Hadoop and HDFS.
3. Visualize data using basic plotting techniques in Python.
4. Implement NoSQL Database Operations: CRUD operations, Arrays using MongoDB.
5. Implement Functions: Count – Sort – Limit – Skip – Aggregate using MongoDB.
6. Implement word count / frequency programs using MapReduce.
7. Implement a MapReduce program that processes a dataset.
8. Implement clustering techniques using SPARK.
9. Implement an application that stores big data in MongoDB / Pig using Hadoop / R.
 BIG DATA ANALYTICS LAB

Table of Contents

S.No Name of the Experiment Page No.

1. Install, configure and run python, numPy and Pandas. 9

2. Install, configure and run Hadoop and HDFS. 25

3. Visualize data using basic plotting techniques in Python. 40

Implement NoSQL Database Operations: CRUD operations, 46

Arrays using MongoDB).
4.
Implement Functions: Count – Sort – Limit – Skip – Aggregate 53
5. using MongoDB.

6. Implement word count / frequency programs using MapReduce. 61

7. Implement a MapReduce program that processes a dataset. 64

8. Implement clustering techniques using SPARK. 67

9. Implement an application that stores big data in MongoDB / Pig 69

using Hadoop / R.
 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

EXPERIMENT: 1
Install, configure and run python, numpy and pandas.

PROGRAM:
AIM: To Installing and Running Applications On python, numpy and pandas.
How to Install Anaconda on Windows?

Anaconda is an open-source software that contains Jupyter, spyder, etc that are used for large data
processing, data analytics, heavy scientific computing. Anaconda works for R and python
programming language. Spyder(sub-application of Anaconda) is used for python. Opencv for python
will work in spyder. Package versions are managed by the package management system called conda.
To begin working with Anaconda, one must get it installed first. Follow the below instructions to
Download and install Anaconda on your system:
Download and install Anaconda:
Head over to anaconda.com and install the latest version of Anaconda. Make sure to download the
―Python 3.7 Version‖ for the appropriate architecture.

Begin with the installation process:

• Getting Started

Getting through the License Agreement:

Mr. Rajesh S Asst Professor MUSE Page 9

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Select Installation Type: Select Just Me if you want the software to be used by a single User

Choose Installation Location:

Mr. Rajesh S Asst Professor MUSE Page 10

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Advanced Installation Option:

Getting through the Installation Process:

Mr. Rajesh S Asst Professor MUSE Page 11

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Recommendation to Install Pycharm:

Finishing up the Installation:

Mr. Rajesh S Asst Professor MUSE Page 12

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Working with Anaconda:

Once the installation process is done, Anaconda can be used to perform multiple operations. To

begin using Anaco nda, search for

Anaconda Navigator from the Start Menu in Windows

Mr. Rajesh S Asst Professor MUSE Page 13

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

#import pandas in jupyter notebook

import pandas

#loading the dataset which is excel file

dataset = pandas.read_csv("crime.csv")

#displaying the data

dataset

import pandas as pd
dataset1 = pd.read_csv("crime.csv")
dataset1

Mr. Rajesh S Asst Professor MUSE Page 14

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

dataset1.head()

dataset1.tail()

dataset1.head(10)

Mr. Rajesh S Asst Professor MUSE Page 15

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

dataset1.tail(10)

type(dataset1)
pandas.core.frame.DataFrame

#to find any null values in the last 5 rows

dataset1.isnull().tail()

#to makesure that no null values exists

dataset1.notnull().tail()

Mr. Rajesh S Asst Professor MUSE Page 16

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

#displays the number of null values in each column

dataset1.isnull().sum()

#helps to find null values with respect to ROBBERY column

dataset1[dataset1.Robbery.isnull()]

dataset1.shape

#helps to find how many times values in a particular column has repeated
dataset1['Robbery'].value_counts()

#consolidated value counts for all the columns in the dataset

for col in dataset1.columns:
display(dataset1[col].value_counts())

Mr. Rajesh S Asst Professor MUSE Page 17

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

#helps to find number of rows in the dataset

dataset_length=len(dataset1)
dataset_length

#helps to find number of columns in the dataset

dataset_col=len(dataset1.columns)
dataset_col

#helps to find the summary of numerical columns

dataset1.describe()

#helps to describe individual column

dataset1.Murder.describe()

dataset1.skew()

Mr. Rajesh S Asst Professor MUSE Page 18

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

dataset1.var()

dataset1.kurtosis()

print(dataset1.dtypes)

NUMPY

Numpy is the core library for scientific and numerical computing in Python. It provides high
performance multi dimensional array object and tools for working with arrays.
Numpy main object is the multidimensional array, it is a table of elements (usually numbers) all of
the same type indexed by a positive integers.

Mr. Rajesh S Asst Professor MUSE Page 19

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
In Numpy dimensions are called as axes.
Numpy is fast, convenient and occupies less memory when compared to python list.

import numpy
arr = numpy.array([1, 2, 3, 4, 5])
print(arr)

NumPy is usually imported under the np alias.

import numpy as np

Now the NumPy package can be referred to as np instead of numpy.

import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)

Checking NumPy Version

The version string is stored under version attribute.

import numpy as np
print(np. version )

Create a NumPy ndarray Object

NumPy is used to work with arrays. The array object in NumPy is called ndarray.
We can create a NumPy ndarray object by using the array() function.

import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)
print(type(arr))

type(): This built-in Python function tells us the type of the object passed to it. Like in above
code it shows that arr is numpy.ndarray type.
To create an ndarray, we can pass a list, tuple or any array-like object into the array() method, and it
will be converted into an ndarray:

Use a tuple to create a NumPy array:

import numpy as np
arr = np.array((1, 2, 3, 4, 5))
print(arr)

Dimensions in Arrays
A dimension in arrays is one level of array depth (nested arrays).

Mr. Rajesh S Asst Professor MUSE Page 20

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

nested array: are arrays that have arrays as their elements.

0-D Arrays
0-D arrays, or Scalars, are the elements in an array. Each value in an array is a 0-D array.

#Create a 0-D array with value 42

import numpy as np
arr = np.array(42)
print(arr)

1-D Arrays
These are the most common and basic arrays.

#Create a 1-D array containing the values 1,2,3,4,5:

import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)

2-D Arrays
An array that has 1-D arrays as its elements is called a 2-D array.
These are often used to represent matrix or 2nd order tensors.

#Create a 2-D array containing two arrays with the values 1,2,3 and 4,5,6:
import numpy as np
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)

3-D arrays
An array that has 2-D arrays (matrices) as its elements is called 3-D array.
These are often used to represent a 3rd order tensor.

#Create a 3-D array with two 2-D arrays, both containing two arrays with the values 1,2,3 and 4,5,6:
import numpy as np
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(arr)

Check Number of Dimensions?

NumPy Arrays provides the ndim attribute that returns an integer that tells us how many dimensions
the array have.

Mr. Rajesh S Asst Professor MUSE Page 21

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
#Check how many dimensions the arrays have:
import numpy as np
a = np.array(42)
b = np.array([1, 2, 3, 4, 5])
c = np.array([[1, 2, 3], [4, 5, 6]])
d = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(a.ndim)
print(b.ndim)
print(c.ndim)
print(d.ndim)

#Create an array with 5 dimensions and verify that it has 5 dimensions:

import numpy as np
arr = np.array([1, 2, 3, 4], ndmin=5)
print(arr)
print('number of dimensions :', arr.ndim)

NumPy Array Indexing

Access Array Elements
Array indexing is the same as accessing an array element.
You can access an array element by referring to its index number.
The indexes in NumPy arrays start with 0, meaning that the first element has index 0, and the second
has index 1 etc.

#Get the first element from the following array:

import numpy as np
arr = np.array([1, 2, 3, 4])
print(arr[0])

#Get the second element from the following array.

import numpy as np
arr = np.array([1, 2, 3, 4])
print(arr[1])

#Get third and fourth elements from the following array and add them.
import numpy as np
arr = np.array([1, 2, 3, 4])
print(arr[2] + arr[3])

Output:

Mr. Rajesh S Asst Professor MUSE Page 22

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Access 2-D Arrays
To access elements from 2-D arrays we can use comma separated integers representing the dimension and
the index of the element.
Think of 2-D arrays like a table with rows and columns, where the dimension represents the row and the
index represents the column.

#Access the element on the first row, second column:

import numpy as np
arr = np.array([[1,2,3,4,5], [6,7,8,9,10]])
print('2nd element on 1st row: ', arr[0, 1])

#Access the element on the 2nd row, 5th column:

import numpy as np
arr = np.array([[1,2,3,4,5], [6,7,8,9,10]])
print('5th element on 2nd row: ', arr[1, 4])

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 23

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 24

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

EXPERIMENT: 2
Install, Configure and Run Hadoop and HDFS
PROGRAM:
AIM: To Installing and Running Applications On Hadoop and HDFS.
HADOOP INSTALATION IN WINDOWS
1. Prerequisites
Hardware Requirement
* RAM — Min. 8GB, if you have SSD in your system then 4GB RAM would also work.
* CPU — Min. Quad core, with at least 1.80GHz
2. JRE 1.8 — Offline installer for JRE
3. Java Development Kit — 1.8
4. A Software for Un-Zipping like 7Zip or Win Rar
* I will be using a 64-bit windows for the process, please check and download the version supported
by your system x86 or x64 for all the software.
5. Download Hadoop zip
* I am using Hadoop-2.9.2, you can use any other STABLE version for hadoop.

Once we have Downloaded all the above software, we can proceed with next steps in installing the
Hadoop.
2. Unzip and Install Hadoop
After Downloading the Hadoop, we need to Unzip the hadoop-2.9.2.tar.gz file.

Once extracted, we would get a new file hadoop-2.9.2.tar.

Mr. Rajesh S Asst Professor MUSE Page 25

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Now, once again we need to extract this tar file.

Now we can organize our Hadoop installation, we can create a folder and move the final extracted
file in it. For Eg. :-

Please note while creating folders, DO NOT ADD SPACES IN BETWEEN THE FOLDER
NAME.(it can cause issues later)
I have placed my Hadoop in D: drive you can use C: or any other drive also.
3. Setting Up Environment Variables
Another important step in setting up a work environment is to set your Systems environment
variable.
To edit environment variables, go to Control Panel > System > click on the ―Advanced system
settings‖ link
Alternatively, We can Right click on This PC icon and click on Properties and click on the
―Advanced system settings‖ link
Or, easiest way is to search for Environment Variable in search bar and there you GO…

Mr. Rajesh S Asst Professor MUSE Page 26

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Mr. Rajesh S Asst Professor MUSE Page 27

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

3.1 Setting JAVA_HOME

Open environment Variable and click on ―New‖ in ―User Variable‖

Mr. Rajesh S Asst Professor MUSE Page 28

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

On clicking ―New‖, we get below screen.

Now as shown, add JAVA_HOME in variable name and path of Java(jdk) in Variable Value.
Click OK and we are half done with setting JAVA_HOME.

Mr. Rajesh S Asst Professor MUSE Page 29

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
3.2 Setting HADOOP_HOME
Open environment Variable and click on ―New‖ in ―User Variable‖

On clicking ―New‖, we get below screen.

Now as shown, add HADOOP_HOME in variable name and path of Hadoop folder in Variable
Value.
Click OK and we are half done with setting HADOOP_HOME.
Note:- If you want the path to be set for all users you need to select ―New‖ from System Variables.
3.3 Setting Path Variable
Last step in setting Environment variable is setting Path in System Variable.

Mr. Rajesh S Asst Professor MUSE Page 30

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Select Path variable in the system variables and click on ―Edit‖.

Now we need to add these paths to Path Variable one by one:-

* %JAVA_HOME%\bin
* %HADOOP_HOME%\bin
* %HADOOP_HOME%\sbin
Click OK and OK. & we are done with Setting Environment Variables.
3.4 Verify the Paths
Now we need to verify that what we have done is correct and reflecting.
Open a NEW Command Window
Run following commands
echo %JAVA_HOME%
echo %HADOOP_HOME%
echo %PATH%
4. Editing Hadoop files
Once we have configured the environment variables next step is to configure Hadoop. It has 3 parts:-
4.1 Creating Folders
We need to create a folder data in the hadoop directory, and 2 sub folders namenode and datanode

Mr. Rajesh S Asst Professor MUSE Page 31

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Create DATA folder in the Hadoop directory

Once DATA folder is created, we need to create 2 new folders namely, namenode and datanode
inside the data folder
These folders are important because files on HDFS resides inside the datanode.
4.2 Editing Configuration Files
Now we need to edit the following config files in hadoop for configuring it :-
(We can find these files in Hadoop -> etc -> hadoop)
* core-site.xml
* hdfs-site.xml
* mapred-site.xml
* yarn-site.xml
* hadoop-env.cmd
4.2.1 Editing core-site.xml
Right click on the file, select edit and paste the following content within <configuration>
</configuration> tags.
Note:- Below part already has the configuration tag, we need to copy only the part inside it.
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
4.2.2 Editing hdfs-site.xml
Mr. Rajesh S Asst Professor MUSE Page 32
 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Right click on the file, select edit and paste the following content within
<configuration></configuration>tags.
Note:- Below part already has the configuration tag, we need to copy only the part inside it.
Also replace PATH~1 and PATH~2 with the path of namenode and datanode folder that we created
recently(step 4.1).
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>C:\hadoop\data\namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>C:\hadoop\data\datanode</value>
</property>
</configuration>
4.2.3 Editing mapred-site.xml
Right click on the file, select edit and paste the following content within <configuration>
</configuration> tags.
Note:- Below part already has the configuration tag, we need to copy only the part inside it.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
4.2.4 Editing yarn-site.xml
Right click on the file, select edit and paste the following content within <configuration>
</configuration> tags.
Note:- Below part already has the configuration tag, we need to copy only the part inside it.
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
4.2.5 Verifying hadoop-env.cmd
Right click on the file, select edit and check if the JAVA_HOME is set correctly or not.
We can replace the JAVA_HOME variable in the file with your actual JAVA_HOME that we
configured in the System Variable.
set JAVA_HOME=%JAVA_HOME%
OR

Mr. Rajesh S Asst Professor MUSE Page 33

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
set JAVA_HOME="C:\Program Files\Java\jdk1.8.0_221"
4.3 Replacing bin
Last step in configuring the hadoop is to download and replace the bin folder.

Mr. Rajesh S Asst Professor MUSE Page 34

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
* Go to this GitHub Repo and download the bin folder as a zip.
* Extract the zip and copy all the files present under bin folder to %HADOOP_HOME%\bin
Note:- If you are using different version of Hadoop then please search for its respective bin folder
and download it.
5. Testing Setup
Congratulation..!!!!!
We are done with the setting up the Hadoop in our System.
Now we need to check if everything works smoothly…
5.1 Formatting Namenode

Before starting hadoop we need to format the namenode for this we need to start a NEW Command
Prompt and run below command
hadoop namenode –format

Note:- This command formats all the data in namenode. So, its advisable to use only at the start and
do not use it every time while starting hadoop cluster to avoid data loss.
5.2 Launching Hadoop
Now we need to start a new Command Prompt remember to run it as administrator to avoid
permission issues and execute below commands
start-all.cmd

Mr. Rajesh S Asst Professor MUSE Page 35

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
This will open 4 new cmd windows running 4 different Daemons of hadoop:-
* Namenode
* Datanode
* Resourcemanager
* Nodemanager

Mr. Rajesh S Asst Professor MUSE Page 36

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Note:- We can verify if all the daemons are up and running using jps command in new cmd window.
6. Running Hadoop (Verifying Web UIs)
6.1 Namenode
Open localhost:50070 in a browser tab to verify namenode health.

6.2 Resourcemanger
Open localhost:8088 in a browser tab to check resourcemanager details.

6.3 Datanode
Open localhost:50075 in a browser tab to checkout datanode.

Mr. Rajesh S Asst Professor MUSE Page 37

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 38

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Facul

Mr. Rajesh S Asst Professor MUSE Page 39

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
EXPERIMENT: 3
Visualize Data Using Basic Plotting Techniques In Python.

PROGRAM:
AIM: To create an application that takes the Visualize Data Using Basic Plotting Techniques.
import pandas as pb
import matplotlib.pyplot as plt
import seaborn as sns
crime=pb.read_csv('crime.csv')
crime

plt.plot(crime.Murder,crime.Assault);

Mr. Rajesh S Asst Professor MUSE Page 40

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026
import seaborn as sns
sns.scatterplot(crime.Murder,crime.Assault);

sns.scatterplot(crime.Murder,crime.Assault,hue=crime.Murder,s=100);

plt.figure(figsize=(12,6))
plt.title('Murder Vs Assault')
sns.scatterplot(crime.Murder,crime.Assault,hue=crime.Murder,s=100);

Mr. Rajesh S Asst Professor MUSE Page 41

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

plt.title('Histogram for Robbery')

plt.hist(crime.Robbery);

plt.bar(crime_bar.index,crime_bar.Robbery);

sns.barplot('Robbery','Year',data=crime);

Mr. Rajesh S Asst Professor MUSE Page 42

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

import matplotlib.pyplot as plt

import pandas as pd
import numpy as np
data=pd.read_csv('crime.csv')
x=data.Population
y=data.CarTheft
plt.scatter(x,y)
plt.xlabel('Population')
plt.ylabel('CarTheft')
plt.title('Population Vs CarTheft')
plt.show();

Mr. Rajesh S Asst Professor MUSE Page 43

 BIG DATA ANALYTICS (21AD62) LAB 2025-2026

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 44

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 45

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

EXPERIMENT: 4
Implement no sql Database Operations: Crud Operations, Arrays Using MONGODB.
PROGRAM:
AIM: To Create a operations for crud and arrays without no sql datasbase.
TITLE: Basic CRUD operations in MongoDB.
CRUD operations refer to the basic Insert, Read, Update and Delete operations.
Inserting a document into a collection (Create)
➢ The command db.collection.insert()will perform an insert operation into a collection of a
document. ➢ Let us insert a document to a student collection. You must be connected to a database
for doing any insert. It is done as follows:
db.student.insert({
regNo: "3014",
name: "Test Student",
course: { courseName: "MCA", duration: "3 Years" },
address: {
city: "Bangalore",
state: "KA",
country: "India" } })
An entry has been made into the collection called student.

Querying a document from a collection (Read)

To retrieve (Select) the inserted document, run the below command. The find() command will
retrieve all the documents of the given collection.
db.collection_name.find()
➢ If a record is to be retrieved based on some criteria, the find() method should be called passing
parameters, then the record will be retrieved based on the attributes specified.
db.collection_name.find({"fieldname":"value"})
➢ For Example: Let us retrieve the record from the student collection where the attribute regNo is
3014and the query for the same is as shown below:
db.students.find({"regNo":"3014"})

Updating a document in a collection (Update) In order to update specific field values of a collection
in MongoDB, run the below query. db.collection_name.update()

Mr. Rajesh S Asst Professor MUSE Page 46

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
➢ update() method specified above will take the fieldname and the new value as argument to update
a document.
➢ Let us update the attribute name of the collection student for the document with regNo 3014.
db.student.update({
"regNo": "3014"
},
$set:
{
"name": "Viraj"
})
Removing an entry from the collection (Delete)
➢ Let us now look into the deleting an entry from a collection. In order to delete an entry from a
collection, run the command as shown below : db.collection_name.remove({"fieldname":"value"})
➢ For Example : db.student.remove({"regNo":"3014"})

Note that after running the remove() method, the entry has been deleted from the student collection.

Working with Arrays in MongoDB

1. Introduction
In a MongoDB database, data is stored in collections and a collection has documents. A document
has fields and values, like in a JSON. The field types include scalar types (string, number, date, etc.)
and composite types (arrays and objects). In this article we will look at an example of using the array
field type.
The example is an application where users create blog posts and write comments for the posts. The
relationship between the posts and comments is One-to-Many; i.e., a post can have many comments.
We will consider a collection of blog posts with their comments. That is a post document will also
store the related comments. In MongoDB's document model, a 1:N relationship data can be stored
within a collection; this is a de-normalized form of data. The related data is stored together and can
be accessed (and updated) together. The comments are stored as an array; an array of comment
objects.
A sample document of the blog posts with comments:
{
"_id" : ObjectId("5ec55af811ac5e2e2aafb2b9"),
"name" : "Working with Arrays",
"user" : "Database Rebel",
"desc" : "Maintaining an array of objects in a document",
"content" : "some content ...",
"created" : ISODate("2020-05-20T16:28:55.468Z"),
"updated" : ISODate("2020-05-20T16:28:55.468Z"),
"tags" : [ "mongodb", "arrays" ],
"comments" : [
{
"user" : "DB Learner",
"content" : "Nice post.",
"updated" : ISODate("2020-05-20T16:35:57.461Z")

Mr. Rajesh S Asst Professor MUSE Page 47

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
}
]
}
In an application, a blog post is created, comments are added, queried, modified or deleted by users.
In the example, we will write code to create a blog post document, and do some CRUD operations
with comments for the post.

2. Create and Query a Document

Let's create a blog post document. We will use a database called as blogs and a collection called as
posts. The code is written in mongoshell (an interactive JavaScript interface to MongoDB). Mongo
shell is started from the command line and is connected to the MongoDB server. From the shell:
use blogs
NEW_POST =
{
name: "Working with Arrays",
user: "Database Rebel",
desc: "Maintaining an array of objects in a document",
content: "some content...",
created: ISODate(),
updated: ISODate(),
tags: [ "mongodb", "arrays" ]
}
db.posts.insertOne(NEW_POST)
Returns a result { "acknowledged" : true, "insertedId" : ObjectId("5ec55af811ac5e2e2aafb2b9") }
indicating that a new document is created. This is a common acknowledgement when you perform a
write operation. When a document is inserted into a collection for the first time, the collection gets
created (if it doesn't exist already). The insertOne method inserts a document into the collection.
Now, let's query the collection :
db.posts.findOne()
{
"_id" : ObjectId("5ec55af811ac5e2e2aafb2b9"),
"name" : "Working with Arrays",
"user" : "Database Rebel",
"desc" : "Maintaining an array of objects in a document",
"content" : "some content...",
"created" : ISODate("2020-05-20T16:28:55.468Z"),
"updated" : ISODate("2020-05-20T16:28:55.468Z"),
"tags" : [
"mongodb",
"arrays"
]
}
The findOne method retrieves one matching document from the collection. Note the scalar fields
name (string type) and created (date type), and the array field tags. In the newly inserted document
there are no comments, yet.

3. Add an Array Element

Let's add a comment for this post, by a user "DB Learner":
NEW_COMMENT = {
user: "DB Learner",
text: "Nice post, can I know more about the arrays in MongoDB?",

Mr. Rajesh S Asst Professor MUSE Page 48

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
updated: ISODate()
}
db.posts.updateOne(
{ _id : ObjectId("5ec55af811ac5e2e2aafb2b9") },
{ $push: { comments: NEW_COMMENT } }
)
Returns: { "acknowledged" : true, "matchedCount" : 1, "modifiedCount" : 1 }
The updateOne method updates a document's fields based upon the specified condition. $push is an
array update operator which adds an element to an array. If the array doesn't exist, it creates an array
field and then adds the element.
Let's query the collection and confirm the new comment visually, using the findOne method:
{
"_id" : ObjectId("5ec55af811ac5e2e2aafb2b9"),
"name" : "Working with Arrays",
...
"comments" : [
{
"user" : "DB Learner",
"text" : "Nice post, can I know more about the arrays in MongoDB?",
"updated" : ISODate("2020-05-20T16:35:57.461Z")
}
]
}
Note the comments array field has comment objects as elements. Let's add one more comment using
the same $push update operator. This new comment (by user "Database Rebel") is appended to the
comments array:
"comments" : [
{
"user" : "DB Learner",
"text" : "Nice post, can I know more about the arrays in MongoDB?",
"updated" : ISODate("2020-05-20T16:35:57.461Z")
},
{
"user" : "Database Rebel",
"text" : "Thank you, please look for updates",
"updated" : ISODate("2020-05-20T16:48:25.506Z")
}
]

4. Update an Array Element

Let's update the comment posted by "Database Rebel" with modified text field :
NEW_CONTENT = "Thank you, please look for updates - updated the post".
db.posts.updateOne(
{ _id : ObjectId("5ec55af811ac5e2e2aafb2b9"), "comments.user": "Database Rebel" },
{ $set: { "comments.$.text": NEW_CONTENT } }
)
The $set update operator is used to change a field's value. The positional $ operator identifies an
element in an array to update without explicitly specifying the position of the element in the array.
The first matching element is updated. The updated comment object:
"comments" : [
{
"user" : "Database Rebel",

Mr. Rajesh S Asst Professor MUSE Page 49

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
"text" : "Thank you, please look for updates - updated",
"updated" : ISODate("2020-05-20T16:48:25.506Z")
}
]

5. Delete an Array Element

The user changed his mind and wanted to delete the comment, and then add a new one.
db.posts.updateOne(
{ _id" : ObjectId("5ec55af811ac5e2e2aafb2b9") },
{ $pull: { comments: { user: "Database Rebel" } } }
)
The $pull update operator removes elements from an array which match the specified condition - in
this case { comments: { user: "Database Rebel" } }.
A new comment is added to the array after the above delete operation, with the following text:
"Thank you for your comment. I have updated the post with CRUD operations on an array field".

6. Add a New Field to all Objects in the Array

Let's add a new field likes for all the comments in the array.
db.posts.updateOne(
{ "_id : ObjectId("5ec55af811ac5e2e2aafb2b9") },
{ $set: { "comments.$[].likes": 0 } }
)
The all positional operator $[] specifies that the update operator $set should modify all elements in
the specified array field. After the update, all comment objects have the likes field, for example:
{
"user" : "DB Learner",
"text" : "Nice post, can I know more about the arrays in MongoDB?",
"updated" : ISODate("2020-05-20T16:35:57.461Z"),
"likes" : 0
}

7. Update a Specific Array Element Based on a Condition

First, let's add another new comment using the $push update operator:
NEW_COMMENT = {
user: "DB Learner",
text: "Thanks for the updates!",
updated: ISODate()
}
Note the likes field is missing in the input document. We will update this particular comment in the
comments array with the condition that the likes field is missing.
db.posts.updateOne(
{ "_id" : ObjectId("5ec55af811ac5e2e2aafb2b9") },
{ $inc: { "comments.$[ele].likes": 1 } },
{ arrayFilters: [ { "ele.user": "DB Learner", "ele.likes": { $exists: false } } ] }
)
The likes field is updated using the $inc update operator (this increments a field's value, or if not
exists adds the field and then increments). The filtered positional operator $[<identifier>] identifies
the array elements that match the arrayFilters conditions for an update operation.

Mr. Rajesh S Asst Professor MUSE Page 50

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 51

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Record Notes

Signature of the faculty

Mr. Rajesh S Asst Professor MUSE Page 52

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
EXPERIMENT: 5
Implement Functions: Count – Sort – Limit – Skip – Aggregate Using MONGODB.
PROGRAM:

AIM: To create function operations for sort, limit, skip and aggregate.

1. COUNT
How do you get the number of Debit and Credit transactions? One way to
do it is by using count() function as below
> db.transactions.count({cr_dr : "D"});
or

> db.transactions.find({cr_dr : "D"}).length();

But what if you do not know the possible values of cr_dr upfront. Here
Aggregation framework comes to play. See the below Aggregate query.
> db.transactions
.aggregate( [
{
$group : {
_id : '$cr_dr', // group by type of transaction
// Add 1 for each document to the count for this type of
transaction
count : {$sum : 1}
}
}
]
);

And the result is

{
"_id" : "C",
"count" : 3
}
{
"_id" : "D",
"count" : 5
}

2. SORT
Definition
$sort
Sorts all input documents and returns them to the pipeline in sorted order.

The
$sort

Mr. Rajesh S Asst Professor MUSE Page 53

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
stage has the following prototype form:

{ $sort: { <field1>: <sort order>, <field2>: <sort order> ... } }

$sort
takes a document that specifies the field(s) to sort by and the respective
sort order. <sort order> can have one of the following values:

Value
Description
1
Sort ascending.
-1
Sort descending.
{ $meta: "textScore" }
Sort by the computed textScore metadata in descending order. See
Text Score Metadata Sort
for an example.
If sorting on multiple fields, sort order is evaluated from left to right. For
example, in the form above, documents are first sorted by <field1>. Then
documents with the same <field1> values are further sorted by <field2>.

Behavior
Limits
You can sort on a maximum of 32 keys.

Sort Consistency
MongoDB does not store documents in a collection in a particular order.
When sorting on a field which contains duplicate values, documents
containing those values may be returned in any order.

If consistent sort order is desired, include at least one field in your sort that
contains unique values. The easiest way to guarantee this is to include the
_id field in your sort query.

Consider the following restaurant collection:

db.restaurants.insertMany( [
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan"},
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" :
"Queens"},
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn"},
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan"},
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn"},
])

Mr. Rajesh S Asst Professor MUSE Page 54

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

The following command uses the

$sort
stage to sort on the borough field:

db.restaurants.aggregate(
[
{ $sort : { borough : 1 } }
]
)

In this example, sort order may be inconsistent, since the borough field
contains duplicate values for both Manhattan and Brooklyn. Documents
are returned in alphabetical order by borough, but the order of those
documents with duplicate values for borough might not the be the same
across multiple executions of the same sort. For example, here are the
results from two different executions of the above command:

{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn" }

{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn" }
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan" }
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan" }
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens"
}
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn" }
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn" }
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan" }
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan" }
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens"
}
While the values for borough are still sorted in alphabetical order, the order
of the documents containing duplicate values for borough (i.e. Manhattan
and Brooklyn) is not the same.

To achieve a consistent sort, add a field which contains exclusively

unique values to the sort. The following command uses the
$sort
stage to sort on both the borough field and the _id field:

db.restaurants.aggregate(
[
{ $sort : { borough : 1, _id: 1 } }
]
)

Mr. Rajesh S Asst Professor MUSE Page 55

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Since the _id field is always guaranteed to contain exclusively unique
values, the returned sort order will always be the same across multiple
executions of the same sort.

Examples
Ascending/Descending Sort
For the field or fields to sort by, set the sort order to 1 or -1 to specify an
ascending or descending sort respectively, as in the following example:

db.users.aggregate(
[
{ $sort : { age : -1, posts: 1 } }
]
)

This operation sorts the documents in the users collection, in descending

order according by the age field and then in ascending order according to
the value in the posts field.

3. LIMIT

$sort
Sorts all input documents and returns them to the pipeline in sorted order.
The $sort stage has the following prototype form:
{ $sort: { <field1>: <sort order>, <field2>: <sort order> ... } }
$sort takes a document that specifies the field(s) to sort by and the
respective sort order. <sort order> can have one of the following values:
Value Description
1 Sort ascending.
-1 Sort descending.
{ $meta: Sort by the computed textScore metadata in
"textScore" } descending order. See Text Score Metadata Sort
for an example.
If sorting on multiple fields, sort order is evaluated from left to right. For
example, in the form above, documents are first sorted by <field1>. Then
documents with the same <field1> values are further sorted by <field2>.
Behavior
Limits
You can sort on a maximum of 32 keys.
Sort Consistency
MongoDB does not store documents in a collection in a particular order.
When sorting on a field which contains duplicate values, documents
containing those values may be returned in any order.
Mr. Rajesh S Asst Professor MUSE Page 56
BIG DATA ANALYTICS (21AD62) LAB 2025-2026
If consistent sort order is desired, include at least one field in your sort
that contains unique values. The easiest way to guarantee this is to include
the _id field in your sort query.
Consider the following restaurant collection:
db.restaurants.insertMany( [
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan"},
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens"},
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn"},
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan"},
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn"},
])
The following command uses the $sort stage to sort on the borough field:
db.restaurants.aggregate(
[
{ $sort : { borough : 1 } }
]
)
In this example, sort order may be inconsistent, since the borough field
contains duplicate values for both Manhattan and Brooklyn. Documents are
returned in alphabetical order by borough, but the order of those documents
with duplicate values for borough might not the be the same across multiple
executions of the same sort. For example, here are the results from two
different executions of the above command:
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn" }
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn" }
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan" }
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan" }
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens" }
{ "_id" : 5, "name" : "Jane's Deli", "borough" : "Brooklyn" }
{ "_id" : 3, "name" : "Empire State Pub", "borough" : "Brooklyn" }
{ "_id" : 4, "name" : "Stan's Pizzaria", "borough" : "Manhattan" }
{ "_id" : 1, "name" : "Central Park Cafe", "borough" : "Manhattan" }
{ "_id" : 2, "name" : "Rock A Feller Bar and Grill", "borough" : "Queens" }
While the values for borough are still sorted in alphabetical order, the order
of the documents containing duplicate values for borough
(i.e. Manhattan and Brooklyn) is not the same.
To achieve a consistent sort, add a field which contains exclusively unique
values to the sort. The following command uses the $sort stage to sort on
both the borough field and the _id field:
db.restaurants.aggregate(
[
{ $sort : { borough : 1, _id: 1 } }
]
)
Since the _id field is always guaranteed to contain exclusively unique

Mr. Rajesh S Asst Professor MUSE Page 57

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

values, the returned sort order will always be the same across multiple
executions of the same sort.
Examples
Ascendin
g/Descen
ding Sort
For the field or fields to sort by, set the sort order to 1 or -1
to specify an ascending or descending sort respectively, as
in the following example:
db.users.aggregate(
[
{ $sort : { age : -1, posts: 1 } }
]
)

4. SKIP
$skip
Skips over the specified number of documents that pass into
the stage and passes the remaining documents to the next
stage in the pipeline.

The
$skip
stage has the following prototype form:

{ $skip: <positive 64-bit integer> }

Mr. Rajesh S Asst Professor MUSE Page 58

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 59

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 60

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

EXPERIMENT: 6
Implement Word Count/ Frequency Programs Using Map Reduce.
PROGRAM:
AIM: To count a given number using map reduce functions.

Hadoop Streaming API for helping us passing data between our Map and Reduce code
via STDIN (standard input) and STDOUT (standard output).
Note : Change the file has execution permission (chmod +x /home/hduser/mapper.py)
Change the file has execution permission (chmod +x /home/hduser/reducer.py
Mapper program
mapper.py
import sys
# input comes from STDIN (standard input)
for line in sys.stdin:
line = line.strip() # remove leading and trailing whitespace
words = line.split()# split the line into words
# increase counters
for word in words:
# write the results to STDOUT (standard output);
# what we output here will be the input for the
# Reduce step, i.e. the input for reducer.py
# tab-delimited; the trivial word count is 1
print '%s\t%s' % (word, 1)
Reducer program
"""reducer.py"""
from operator import itemgetter
import sys
current_word = None
current_count = 0
word = None

# input comes from STDIN

for line in sys.stdin:
line = line.strip() # remove leading and trailing whitespace
# parse the input we got from mapper.py
word, count = line.split('\t', 1)
# convert count (currently a string) to int
try:
count = int(count)
except ValueError:
# count was not a number, so silently
# ignore/discard this line
continue

# this IF-switch only works because Hadoop sorts map output

# by key (here: word) before it is passed to the reducer
if current_word == word:
current_count += count
else:

Mr. Rajesh S Asst Professor MUSE Page 61

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
if current_word:
# write result to STDOUT
print '%s\t%s' % (current_word, current_count)
current_count = count
current_word = word

# do not forget to output the last word if needed!

if current_word == word:
print '%s\t%s' % (current_word, current_count)

Test the code (cat data | map | sort | reduce)

hduser@ubuntu:~$ echo "foo foo quux labs foo bar quux" | /home/hduser/mapper.py
foo 1
foo 1
quux 1
labs 1
foo 1
bar 1
quux 1

hduser@ubuntu:~$ echo "foo foo quux labs foo bar quux" | /home/hduser/mapper.py | sort -k1,1 |
/home/hduser/reducer.py
bar 1
foo 3
labs 1
quux 2

hduser@ubuntu:~$ cat /tmp/gutenberg/20417-8.txt | /home/hduser/mapper.py

The 1
Project 1
Gutenberg 1
EBook 1
of 1

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 62

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 63

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
EXPERIMENT: 7
Implement a MapReduce Program that process a dataset.
AIM:
To create process dataset using map reduce functions.

PROGRAM:

The python program reads the data from a dataset ( stored in the file data.csv- wine quality).
The data mapped is stored in shuffled.pkl using mapper.py.
The contents of shuffled.pkl are reduced using reducer.py
Mapper Program

import pandas as pd
import pickle

data = pd.read_csv('data.csv')

#Slicing Data
slice1 = data.iloc[0:399,:]
slice2 = data.iloc[400:800,:]
slice3 = data.iloc[801:1200,:]
slice4 = data.iloc[1201:,:]

def mapper(data):
mapped = []

for index,row in data.iterrows():

mapped.append((row['quality'],row['volatile acidity']))
return mapped

map1 = mapper(slice1)
map2 = mapper(slice2)
map3 = mapper(slice3)
map4 = mapper(slice4)

shuffled = {
3.0: [],
4.0: [],
5.0: [],
6.0: [],
7.0: [],
8.0: [],
}
for i in [map1,map2,map3,map4]:
for j in i:

Mr. Rajesh S Asst Professor MUSE Page 64

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
shuffled[j[0]].append(j[1])
file= open('shuffled.pkl','ab')
pickle.dump(shuffled,file)
file.close()

print("Data has been mapped. Now, run reducer.py to reduce the contents in
shuffled.pkl file.")

Reducer Program

import
pickle
file= open('shuffled.pkl','rb')
shuffled = pickle.load(file)
def reduce(shuffled_dict):
reduced = {}

for i in shuffled_dict:

reduced[i] = sum(shuffled_dict[i])/len(shuffled_dict[i])

return reduced
final = reduce(shuffled)
print("Average volatile acidity in different classes of wine: ")
for i in final:
print(i,':',final[i])

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 65

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 66

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

EXPERIMENT: 8

Implement Clustering Techniques Using SPARK.

AIM: To create a clustering using SPARK.

PROGRAM:

# Loads data.
dataset = spark.read.format("libsvm").load("data/mllib/sample_kmeans_data.txt")

# Trains a k-means model.

kmeans = KMeans().setK(2).setSeed(1)
model = kmeans.fit(dataset)

# Evaluate clustering by computing Within Set Sum of Squared Errors.

wssse = model.computeCost(dataset)
print("Within Set Sum of Squared Errors = " + str(wssse))

# Shows the result.

centers = model.clusterCenters()
print("Cluster Centers: ")
for center in centers:
print(center)

OUTPUT:

Mr. Rajesh S Asst Professor MUSE Page 67

BIG DATA ANALYTICS (21AD62) LAB 2025-2026

Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 68

BIG DATA ANALYTICS (21AD62) LAB 2025-2026
EXPERIMENT : 9
Implement an Application that Stores Big Data in MONGODB / PIG Using Hadoop /
R.
AIM: To design a application to stores data in mongdob usinghadoop.

PROGRAM:

R Shiny Tutorial: How to Make Interactive Web Applications in R

Introduction
In this modern technological era, various apps are available for all of us –from tracking our fitness
level, sleep to giving us the latest information about the stock markets. Apps like Robinhood, Google
Fit and Workit seem so amazingly useful because they use real-time data and statistics. As R is a
frontrunner in the field of statistical computing and programming, developers need a system to use its
power to build apps.
This is where R Shiny comes to save the day. In this, R Shiny tutorial, you will come to know the
basics.
What is R Shiny?
Shiny is an R package that was developed for building interactive web applications in R. Using this,
you can create web applications utilizing native HTML and CSS code along with R Shiny code. You
can build standalone web apps on a website that will make data visualization easy. These applications
made through R Shiny can seamlessly display R objects such as tables and plots.
Let us look at some of the features of R Shiny:
• Build web applications with fewer lines of code, without JavaScript.
• These applications are live and are accessible to users like spreadsheets. The outputs may
alter in real-time if the users change the input.
• Developers with little knowledge of web tools can also build apps using R Shiny.
• You get in-built widgets to display tables, outputs of R objects and plots.
• You can add live visualizations and reports to the web application using this package.
• The user interfaces can be coded in R or can be prepared using HTML, CSS or JavaScript.
• The default user interface is built using Bootstrap.
• It comes with a WebSocket package that enables fast communication between the web server
and R.

Components of an R Shiny app

A Shiny app has two primary components – a user interface object and a server function. These
are the arguments passed on to the shinyApp method. This method creates an application object
using the arguments.

Let us understand the basic parts of an R Shiny app in detail:

User interface function

This function defines the appearance of the web application. It makes the application interactive
by obtaining input from the user and displaying it on the screen. HTML and CSS tags can be
used for making the application look better. So, while building the ui.R file you create an
HTML file with R functions.

If you type fluidPage() in the R console, you will see that the method returns a tag <div
class=‖container-fluid‖></div>.

The different input functions are:

Mr. Rajesh S Asst Professor MUSE

Page 69
BIG DATA ANALYTICS (21AD62) LAB 2025-2026

• selectInput() – This method is used for creating a dropdown HTML that has various
choices to select.
• numericInput() – This method creates an input area for writing text or numbers.
• radioButtons() – This provides radio buttons for the user to select an input.

Layout methods
The various layout features available in Bootstrap are implemented by R Shiny. The components are:

Panels
These are methods that group elements together into a single panel. These include:

• absolutePanel()
• inputPanel()
• conditionalPanel()
• headerPanel()
• fixedPanel()

Layout functions
These organize the panels for a particular layout. These include:

• fluidRow()
• verticalLayout()
• flowLayout()
• splitLayout()
• sidebarLayout()

Output methods
These methods are used for displaying R output components images, tables and plots. They are:

• tableOutput() – This method is used for displaying an R table

• plotOutput() – This method is used for displaying an R plot object

Server function
After you have created the appearance of the application and the ways to take input values from the
user, it is time to set up the server. The server functions help you to write the server-side code for the
Shiny app. You can create functions that map the user inputs to the corresponding outputs. This
function is called by the web browser when the application is loaded.

It takes an input and output parameter, and return values are ignored. An optional session parameter is
also taken by this method.

R Shiny tutorial: How to get started with R Shiny?

Steps to start working with the R Shiny package are as follows:

• Go to the R console and type in the command – install.packages(―shiny‖)

• The package comes with 11 built-in application examples for you to understand how Shiny
works

Mr. Rajesh S Asst Professor MUSE

Page 70
BIG DATA ANALYTICS (21AD62) LAB 2025-2026
You can start with the Hello Shiny example to understand the basic structure. Type this code to run
Hello Shiny:

library(shiny)
runExample(“01_hello”)

The steps to create a new Shiny app are:

• Open RStudio and go to the File option

• Select New Project in a directory and click on the ―Shiny Web‖ Application
• You will get a histogram and a slider to test the changes in output with respect to the input
• You will get two scripts ui.R and server.R for coding and customizing the application

Tips for Shiny app development

• Test the app in the browser to see how it looks before sending it for production
• Run the entire script while debugging the app
• Be careful about common error such as commas

OUTPUT:

Mr. Rajesh S Asst Professor MUSE

Page 71
BIG DATA ANALYTICS (21AD62) LAB 2025-2026
Record Notes

Signature of the Faculty

Mr. Rajesh S Asst Professor MUSE Page 72