Linear regression
LINEAR REGRESSION
DEFINITION
KEY CONCEPTS AND TYPES
APPLICATIONS
HOW IT WORKS
KEY ADVANTAGES AND LIMITATIONS
2
Linear regression Definition
Imagine you want to predict how much a house will sell for.
3
Linear regression Definition
Imagine you want to predict how much a house will sell for.
Linear regression could help you find a relationship between the house's size (square
footage) and its selling price.
4
Linear regression Definition
Imagine you want to predict how much a house will sell for.
By analyzing past data, it could determine a line that shows how the price generally increases as the size
increases.
5
Linear regression Definition
Imagine you want to predict how much a house will sell for.
By analyzing past data, it could determine a line that shows how the price generally increases as the size
increases.
6
Linear regression Applications
• Predicting stock prices
• Forecasting sales
• Estimating crop yields
• Analyzing the relationship between income and education
• Many other fields where you want to understand or predict the relationship
between variables
7
Linear regression types
When there is only one independent feature, it is known as Simple Linear Regression
when there are more than one feature, it is known as Multiple Linear Regression
when there is only one dependent variable, it is considered Univariate Linear Regression
while when there are more than one dependent variables, it is known as Multivariate Regression.
8
“
How does it Work?
”
Simple Linear Regression
9
Simple Linear Regression
Model representation
Cost Function
Gradient Descent
10
Simple Linear Regression
Simple linear regression is used to estimate the relationship between two quantitative variables.
You can use simple linear regression when you want to know:
How strong the relationship is between two variables (e.g., the relationship between rainfall and
soil erosion).
The value of the dependent variable at a certain value of the independent variable (e.g., the
amount of soil erosion at a certain level of rainfall).
Regression models describe the relationship between variables by fitting a line to the observed data.
Regression allows you to estimate how a dependent variable changes as the independent variable(s) change.
Response variables Explanatory variables
Outcome variables Predictor variables
Simple Linear Regression 11
Simple linear regression example
You are a social researcher interested in the relationship between income and happiness. You survey 500 people
whose incomes range from 15k to 75k and ask them to rank their happiness on a scale from 1 to 10.
What is the independent variable ?
What is dependent variable?
Is both of them quantitative?
so you can do a regression analysis to see if there is a linear relationship between them.
If you have more than one independent variable, use multiple linear regression instead.
Simple Linear Regression 12
How to perform a simple linear regression?
Features = Input x 𝒔𝒊𝒛𝒆 𝒇 price
Training set Targets = Output y
𝒆𝒔𝒕𝒊𝒎𝒂𝒕𝒆
predict
The job of learning algorithm
is to output a function usually
Learning algorithm called h: hypothesis
The job of the hypothesis function is to take the
𝒙 𝒇 𝒚 value of x and output the estimated value of y
𝒚 -- predicted y
𝑴𝒐𝒅𝒆𝒍 Y --- actual y (from the datased)
Simple Linear Regression 13
How to perform a simple linear regression?
The formula for a simple linear regression is:
y is the predicted value of the dependent variable (y) for any given value of the
independent variable (x).
B0 is the intercept, the predicted value of y when the x is 0.
B1 is the regression coefficient (Slope) – how much we expect y to change
as x increases.
x is the independent (explanatory) variable ( the variable we expect is influencing y).
e is the error of the estimate, or how much variation there is in our estimate of the
regression coefficient.
Simple Linear Regression 14
How to perform a simple linear regression?
Linear regression finds the best fit line through your data by searching for the
regression coefficient (B1) that minimizes the total error (e) of the model.
Simple Linear Regression 15
How to perform a simple linear regression?
y → the response/target variable
“score”.
x → the explanatory variable
“hours”.
m → Slope, in ML : weight
c → The intercept, in ML : bias.
Simple Linear Regression 16
How do we know how well our regression line fits
the data points?
Residual (error): The difference between the
predicted value ŷ and the observed value y.
Residual = y – ŷ
The measure of the fit of a model to predict all
values of the response variable close to the
observed values is called the residual sum of
squares (RSS/SSres)
RSS = Σ (y - ŷ)²
where:
• Σ represents the sum of all data points
• y is the actual value
• ŷ is the predicted value
Simple Linear Regression 17
How do we know how well our regression line fits
the data points?
Defining and measuring the error of a
model is called loss function
or cost function sometimes also
called error function
to minimize the error in the model prediction to find the best line
Simple Linear Regression 18
Cost function
A function that assesses a machine learning model’s performance also referred to as a loss function
or objective function.
A cost function is a mathematical function that calculates the difference between the target actual
values (ground truth) and the values predicted by the model.
Usually, the objective of a machine learning algorithm
is to reduce the error or output of cost function.
Common Cost Functions
Mean Squared Error (MSE): Commonly used in regression problems.
Mean Absolute Error (MAE).
Binary Cross-Entropy.
Categorical Cross-Entropy.
Simple Linear Regression 19
Cost function
In Linear Regression, the conventional Cost Function employed is the Mean Squared Error.
The cost function (J) for m training samples can be written as:
Simple Linear Regression 20
Cost function
Simple Linear Regression 21
Cost function
Simple Linear Regression 22
Cost function
Simple Linear Regression 23
Cost function
Simple Linear Regression 24
Gradient Descent
In machine learning, gradient descent is an iterative optimization algorithm used to minimize the cost
function of a model.
By iteratively adjusting the model's parameters in the direction of steepest descent, it helps the model find
the best possible fit to the data
Steps Required in Gradient Descent Algorithm
Step 1 we first initialize the parameters of the model randomly
Step 2 Compute the gradient of the cost function with respect to each parameter. It involves making partial
differentiation of cost function with respect to the parameters.
Step 3 Update the parameters of the model by taking steps in the opposite direction of the model. Here we
choose a hyperparameter learning rate which is denoted by alpha. It helps in deciding the step size of the
gradient.
Step 4 Repeat steps 2 and 3 iteratively to get the best parameter for the defined model
Gradient Descent 25
How Does Gradient Descent Work?
Gradient descent works by
moving downward toward the
pits or valleys in the graph to find
the minimum value.
During each iteration, gradient
descent step-downs the cost
function in the direction of the
steepest descent.
The size of each step is determined
by parameter α known
as Learning Rate.
Gradient Descent 26
How Does Gradient Descent Work?
Gradient descent works by
moving downward toward the
pits or valleys in the graph to find
the minimum value.
During each iteration, gradient
descent step-downs the cost
function in the direction of the
steepest descent.
The size of each step is determined
by parameter α known
as Learning Rate.
Gradient Descent 27
How Does Gradient Descent Work? Visualiser
How To Choose Learning Rate?
The choice of correct learning rate is very
important as it ensures that Gradient
Descent converges in a reasonable time.
Gradient Descent 28
How Does Gradient Descent Work? Visualiser
How To Choose Learning Rate?
The choice of correct learning rate is very
important as it ensures that Gradient
Descent converges in a reasonable time.
Choose it when the curve decreases
rapidly and continuously
Gradient Descent algorithm 29
How Does Gradient Descent Work?
