Assignment 14: Multiprocessing - Utkarsh Gaikwad
Assignment pdf link
Question 1
Question 1: What is multiprocessing in python? Why is it useful?
Answer :
Multiprocessing in Python refers to the ability to execute multiple processes or tasks
simultaneously on multiple processors or cores of a computer, as opposed to executing
them in a sequential manner.
In simpler terms, multiprocessing allows you to take advantage of the multi-core
processors in modern computers to perform CPU-intensive tasks more efficiently, by
splitting them into smaller sub-tasks that can be executed simultaneously on different
processors. This can significantly reduce the time taken to complete the tasks and improve
the overall performance of the program.
In Python, the multiprocessing module provides a way to create and manage multiple
processes. It offers several classes and functions to create and control processes, to
communicate and share data between them, and to handle exceptions and errors that may
occur during their execution.
Multiprocessing is useful in various scenarios, such as:
1. CPU-intensive tasks: Multiprocessing can speed up the execution of tasks that require a lot of CPU time,
such as image processing, machine learning, and scientific computing.
2. Parallel programming: Multiprocessing can be used to implement parallel algorithms, where multiple
processes work together to solve a problem.
3. Scalability: Multiprocessing can help to scale up the performance of a program as the size of the data or the
complexity of the problem increases.
4. Fault-tolerance: Multiprocessing can improve the reliability of a program by isolating the processes from
each other and preventing errors in one process from affecting the others.
Below is small example of multiprocessing
In [1]:
import multiprocessing
def square(n):
"""
This function returns square of a given number
"""
return n**2
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=5)
# Map the worker function to the numbers 0-3
# The map function will divide the work and distribute it among the processes
# Each process will execute the worker function with its assigned number
results = pool.map(square, range(10))
print(results)
# Close the pool and wait for the worker processes to finish
pool.close()
pool.join()
print('All squares calculated with multiprocessing')
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
All squares calculated with multiprocessing
Question 2
Question 2: What are the differences between multiprocessing and
multithreading?
Answer : Below Table describes differences between Multiprocessing
and Multithreading
Feature Multiprocessing Multithreading
Execution Multiple processes Seperate processor core used for each
Multiple threads within a single process
model process
Parallelism True parallelism Concurrency due to GIL
Resource usage Higher resource usage, slower startup times Lower resource usage, faster startup times
Communication IPC mechanisms (pipes, queues, shared memory) Shared memory, synchronization primitives
Easier (shared memory, single memory
Debugging More complex (multiple memory spaces)
space)
Best for CPU-bound tasks I/O-bound tasks
Question 3
Question 3: Write a python code to create a process using the
multiprocessing module.
Answer :
1. Let's say you have a Python program that performs some time-consuming task, such as downloading large
files from the internet.
2. You want to be able to run this program in parallel, so that multiple downloads can happen at the same time,
speeding up the overall process.
3. To achieve this, you can use the multiprocessing module to create a separate process for each download,
allowing the downloads to happen in parallel.
In [2]:
# Example 1
import multiprocessing
import requests
def download_file(url, filename):
"""Download a file from a given URL and save it to disk"""
response = requests.get(url)
with open(filename, 'wb') as f:
f.write(response.content)
print(f"Downloaded {url} to {filename}")
if __name__ == '__main__':
# URLs of files to download
urls = [
'https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignment
%2013%20-%2014%20February%202023/data1.txt',
'https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignment
%2013%20-%2014%20February%202023/data2.txt',
'https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignment
%2013%20-%2014%20February%202023/data3.txt'
]
# Create a new process for each download
processes = []
for i, url in enumerate(urls):
filename = f"file{i+1}.txt"
p = multiprocessing.Process(target=download_file, args=(url, filename))
processes.append(p)
p.start()
# Wait for all processes to finish
for p in processes:
p.join()
print("All downloads completed")
Downloaded https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignme
nt%2013%20-%2014%20February%202023/data1.txt to file1.txt
Downloaded https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignme
nt%2013%20-%2014%20February%202023/data3.txt to file3.txt
Downloaded https://raw.githubusercontent.com/utkarshg1/PWSkills-Assignments/main/Assignme
nt%2013%20-%2014%20February%202023/data2.txt to file2.txt
All downloads completed
In [3]:
#Example 2 Calculating cubes
import multiprocessing
def cube(index, value):
"""
This function returns a cube of given index inside a list
"""
value[index] = value[index]**3
if __name__ == '__main__':
# Creating an example array list to get cubes
arr = multiprocessing.Array('i',[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16])
# Creating processes list for appeding the processes
process = []
for i in range(15):
m = multiprocessing.Process(target=cube,args=(i,arr))
process.append(m)
m.start()
# Wait for processes to finish
for j in process:
j.join()
print(list(arr))
[8, 27, 64, 125, 216, 343, 512, 729, 1000, 1331, 1728, 2197, 2744, 3375, 4096]
Question 4
Question 4: What is a multiprocessing pool in python? Why is it
used?
Answer :
In Python, a multiprocessing pool is a class in the multiprocessing module that provides a way to distribute
tasks across multiple CPU cores. The idea is to create a pool of worker processes that can execute tasks in
parallel, thereby reducing the time it takes to complete a large number of tasks.
Here's how it works:
1. You create a Pool object with a specified number of worker processes.
2. You submit tasks to the pool using the apply(), apply_async(), map(), or map_async() methods.
3. The pool distributes the tasks among the worker processes and runs them in parallel.
4. The results of each task are collected and returned to the main process.
The advantage of using a multiprocessing pool is that it allows you to take advantage of multiple CPU cores to
perform computations in parallel. This can lead to significant speedups for CPU-bound tasks, such as numerical
computations, image processing, or machine learning.
The Pool class in Python also provides various methods for controlling the number of worker processes, waiting
for tasks to complete, and handling errors. Overall, it's a powerful tool for scaling up your Python programs to
take advantage of modern hardware.
Question 5
Question 5: How can we create a pool of worker processes in python
using multiprocessing module?
Answer : We can create pool with help of multiprocessing.Pool
Function
1. Example Calculating Gamma Function Values for various elements in a list
∞
2. Γ(z) = ∫ tz−1 ⋅ e−t ,dt
2. Γ(z) = ∫0 tz−1 ⋅ e−t ,dt
3. this can be calculated with math.gamma() for single value
Wikipedia Link for Gamma Function Explaination
In [4]:
import multiprocessing
import math
def calc_gamma(x):
"""Function to be executed by worker processes"""
result = math.gamma(x)
return result
if __name__ == '__main__':
# Create a list of values for which to calculate the gamma function
values = [0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Create a pool of worker processes
pool = multiprocessing.Pool(processes=4)
# Map the gamma function calculation function to the values
results = pool.map(calc_gamma, values)
# Close the pool and wait for the worker processes to finish
pool.close()
pool.join()
# Print the results
for i in range(len(values)):
print(f"Gamma({values[i]}) = {results[i]} ")
Gamma(0.5) = 1.7724538509055159
Gamma(1) = 1.0
Gamma(2) = 1.0
Gamma(3) = 2.0
Gamma(4) = 6.0
Gamma(5) = 24.0
Gamma(6) = 120.0
Gamma(7) = 720.0
Gamma(8) = 5040.0
Gamma(9) = 40320.0
Gamma(10) = 362880.0
Question 6
Question 6 : Write a python program to create 4 processes, each process
should print a different number using the multiprocessing module in
python.
Answer :
In [7]:
import multiprocessing
import random
def generate_random_number(num):
"""
This function generates random numbers between 1 to 100
"""
random_number = random.randint(1, 100)
print(f"Process number {num}, random number generated : {random_number}")
if __name__ == '__main__':
# Creating a processess list
processes = []
for i in range(4):
p = multiprocessing.Process(target=generate_random_number, args=(i,))
processes.append(p)
p.start()
# Waiting for processess to complete
for p in processes:
p.join()
Process number 0, random number generated : 19
Process number 1, random number generated : 17
Process number 2, random number generated : 1
Process number 3, random number generated : 49
