Kapil Kumar Nagwanshi

PhD (CSE), Sr. Member IEEE, LMCSI, MIAENG

Associate Professor (CSE), ASET

Amity University Rajasthan, Jaipur
▪ Machine learning models are basically mathematical functions that represent the relationship between different
aspects of data. For instance, a linear regression model uses a line to represent the relationship between
“features” and “target.” The formula looks like this:
𝑦 = 𝑤 𝑇𝑥
▪ where x is a vector that represents features of the data and y is a scalar variable that represents the target (some
numeric quantity that we wish to learn to predict).
▪ This model assumes that the relationship between x and y is linear.
▪ The variable w is a weight vector that represents the normal vector for the line; it specifies the slope of the line.
▪ This is what’s known as a model parameter, which is learned during the training phase.

▪ “Training a model” involves using an optimization procedure to determine the best model parameter that “fits”
the data. 𝑛

𝑦 = ෍ 𝑤𝑖 𝑥𝑖
𝑖=0
▪ So a model parameter is a configuration variable that is internal to the model and whose value can be estimated
from the given data.
▪ They are required by the model when making predictions.
▪ Their values define the skill of the model on your problem.
▪ They are estimated or learned from data.
▪ They are often not set manually by the practitioner.
▪ They are often saved as part of the learned model.
▪ In statistics, hyperparameter is a parameter from a prior distribution; it captures the prior belief before
data is observed.
▪ In any machine learning algorithm, these parameters need to be initialized before training a model.
▪ These are values that must be specified outside of the training procedure.
▪ Vanilla linear regression doesn’t have any hyperparameters.
▪ But variants of linear regression do.
▪ Ridge regression and lasso both add a regularization term to linear regression; the weight for the
regularization term is called the regularization parameter.
▪ Decision trees have hyperparameters such as the desired depth and number of leaves in the tree.
Support vector machines (SVMs) require setting a misclassification penalty term.
▪ Kernelized SVMs require setting kernel parameters like the width for radial basis function (RBF)
kernels.
▪ Model Hyperparameters are the properties that govern the entire training process. The below are the
variables usually configure before training a model.
• Learning Rate
• Number of Epochs
• Hidden Layers
• Hidden Units
• Activations Functions
▪ Hyperparameters are important because they directly control the behaviour of the training
algorithm and have a significant impact on the performance of the model is being trained.
▪ “A good choice of hyperparameters can really make an algorithm shine”.
▪ Choosing appropriate hyperparameters plays a crucial role in the success of our neural network
architecture. Since it makes a huge impact on the learned model.
▪ For example,
▪ if the learning rate is too low, the model will miss the important patterns in the data.
▪ If it is high, it may have collisions.

▪ Choosing good hyperparameters gives two benefits:

▪ Efficiently search the space of possible hyperparameters
▪ Easy to manage a large set of experiments for hyperparameter tuning.
The process of finding most
optimal hyperparameters in
machine learning is called Common algorithms include:
hyperparameter
optimisation.
• Grid Search
• Random Search
• Bayesian Optimisation
 ▪ Grid search is a very traditional technique for implementing
hyperparameters. It brute force all combinations. Grid search
requires to create two set of hyperparameters.
▪ Learning Rate
▪ Number of Layers

▪ Grid search trains the algorithm for all combinations by using the two
set of hyperparameters (learning rate and number of layers) and
measures the performance using “Cross Validation” technique.
▪ This validation technique gives assurance that our trained model got
most of the patterns from the dataset.
▪ One of the best methods to do validation by using “K-Fold Cross
Validation” which helps to provide ample data for training the model
and ample data for validations.
▪ The Grid search method is a simpler algorithm to use but it suffers if
data have high dimensional space called the curse of dimensionality.

https://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf
▪ Randomly samples the search space and evaluates sets from a
specified probability distribution.
▪ For example, Instead of trying to check all 100,000 samples, we can
check 1000 random parameters.
▪ Drawback
▪ However, it doesn’t use information from prior experiments to select the next set
and also it is very difficult to predict the next of experiments.
▪ Hyperparameter setting maximizes the performance of the model on
a validation set.
▪ Machine learning algorithms frequently require to fine-tuning of
model hyperparameters.
▪ Unfortunately, that tuning is often called as ‘black function’ because it
cannot be written into a formula since the derivates of the function are
unknown.
▪ Much more appealing way to optimize and fine-tune
hyperparameters are enabling automated model tuning approach by
using Bayesian optimization algorithm.
▪ The model used for approximating the objective function is called
surrogate model. A popular surrogate model for Bayesian
optimization is Gaussian process (GP).
▪ Bayesian optimization typically works by assuming the unknown
function was sampled from a Gaussian Process (GP) and maintains a
posterior distribution for this function as observations are made.
▪ There are two major choices must be made when performing Bayesian
optimization.
▪ Select prior over functions that will express assumptions about the function being
optimized. For this, we choose Gaussian Process prior
▪ Next, we must choose an acquisition function which is used to construct a utility function
from the model posterior, allowing us to determine the next point to evaluate.
▪ A Gaussian process defines the prior distribution over functions which can be
converted into a posterior over functions once we have seen some data.
▪ The Gaussian process uses Covariance matrix to ensure that values that are close
together.
▪ The covariance matrix along with a mean µ function to output the expected value
ƒ(x) defines a Gaussian process.
▪ 1. Gaussian process will be used as a prior for Bayesian inference
▪ 2. To computing the posterior is that it can be used to make predictions for unseen test cases.
▪ Acquisition Function
▪ Introducing sampling data into the search space is done by acquisition functions. It helps to
maximize the acquisition function to determine the next sampling point. Popular acquisition
functions are
• Maximum Probability of Improvement (MPI)
• Expected Improvement (EI)
• Upper Confidence Bound (UCB)
▪ The Expected Improvement (EI) function seems to be a popular one. It is defined as
▪ EI(x)=𝔼[max{0,ƒ(x)−ƒ(x̂ )}]

▪ where ƒ(x̂ ) is the current optimal set of hyperparameters. Maximising the hyperparameters will
improve upon ƒ.
1. EI is high when the posterior expected value of the loss µ(x) is higher than the current best value ƒ(x̂)
2. EI is high when the uncertainty σ(x)σ(x) around the point xx is high.