HYPERPARAMET

ER TUNING
Kapil Kumar Nagwanshi
PhD (CSE), Sr. Member IEEE, LMCSI, MIAENG

Associate Professor (CSE), SoS E&T

 PARAMETER IN A ML MODEL?
 Machine learning models are basically mathematical functions that represent the relationship between
different aspects of data. For instance, a linear regression model uses a line to represent the relationship
between “features” and “target.” The formula looks like this:

 where x is a vector that represents features of the data and y is a scalar variable that represents the
target (some numeric quantity that we wish to learn to predict).
 This model assumes that the relationship between x and y is linear.
 The variable w is a weight vector that represents the normal vector for the line; it specifies the slope of
the line.
 This is what’s known as a model parameter, which is learned during the training phase.
 “Training a model” involves using an optimization procedure to determine the best model parameter that
“fits” the data.

 So a model parameter is a configuration variable that is internal to the model and whose value can be
estimated from the given data.
 They are required by the model when making predictions.
 Their values define the skill of the model on your problem.
 They are estimated or learned from data.
 They are often not set manually by the practitioner.
 They are often saved as part of the learned model.
 HYPERPARAMETERS: “NUISANCE PARAMETERS”
 In statistics, hyperparameter is a parameter from a prior distribution; it captures the prior belief before
data is observed.
 In any machine learning algorithm, these parameters need to be initialized before training a model.
 These are values that must be specified outside of the training procedure.
 Vanilla linear regression doesn’t have any hyperparameters.
 But variants of linear regression do.
 Ridge regression and lasso both add a regularization term to linear regression; the weight for the
regularization term is called the regularization parameter.
 Decision trees have hyperparameters such as the desired depth and number of leaves in the tree.
Support vector machines (SVMs) require setting a misclassification penalty term.
 Kernelized SVMs require setting kernel parameters like the width for radial basis function (RBF) kernels.

 Model Hyperparameters are the properties that govern the entire training process. The below are
the variables usually configure before training a model.
• Learning Rate
• Number of Epochs
• Hidden Layers
• Hidden Units
• Activations Functions
 WHY ARE HYPERPARAMETERS ESSENTIAL?
 Hyperparameters are important because they directly control the behavior of the
training algorithm and have a significant impact on the performance of the model is
being trained.
 “A good choice of hyperparameters can really make an algorithm shine”.
 Choosing appropriate hyperparameters plays a crucial role in the success of our neural
network architecture. Since it makes a huge impact on the learned model.
 For example,
 if the learning rate is too low, the model will miss the important patterns in the data.
 If it is high, it may have collisions.

 Choosing good hyperparameters gives two benefits:

 Efficiently search the space of possible hyperparameters
 Easy to manage a large set of experiments for hyperparameter tuning.
 HYPERPARAMETERS OPTIMISATION TECHNIQUES

The process of finding

most optimal
hyperparameters in Common algorithms
machine learning is called include:
hyperparameter
optimisation.
• Grid Search
• Random Search
• Bayesian Optimisation
 GRID SEARCH
 Grid search is a very traditional technique for implementing
hyperparameters. It brute force all combinations. Grid search
requires to create two set of hyperparameters.
 Learning Rate
 Number of Layers

 Grid search trains the algorithm for all combinations by using

the two set of hyperparameters (learning rate and number of
layers) and measures the performance using “Cross Validation”
technique.
 This validation technique gives assurance that our trained
model got most of the patterns from the dataset.
 One of the best methods to do validation by using “K-Fold Cross
Validation” which helps to provide ample data for training the
model and ample data for validations.
 The Grid search method is a simpler algorithm to use but it
suffers if data have high dimensional space called the curse of
dimensionality.

https://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf
GRIDSEARCHCV
 It is difficult to manually change the hyperparameters and fit them on my training
data every time. Here’s why:
• it is time-consuming
• it is hard to keep track of hyperparameters we tried and we still have to try

 So, I quickly asked Google if there was any solution to my problem and Google
showed me something called GridSearchCV from Sklearn. Let me share how I
took advantage of this GridSearchCV to solve my problem with a simple example.
 GridSearchCV is a library function that is a member of sklearn’s model_selection
package. It helps to loop through predefined hyperparameters and fit your
estimator (model) on your training set. So, in the end, you can select the best
parameters from the listed hyperparameters.
 In addition to that, you can specify the number of times for the cross-validation
for each set of hyperparameters.
 1. estimator: estimator object you created
2. params_grid: the dictionary object that holds the
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier hyperparameters you want to try
kn = KNeighborsClassifier() 3. scoring: evaluation metric that you want to use, you can
simply pass a valid string/ object of evaluation metric
params = { 4. cv: number of cross-validation you have to try for each
'n_neighbors' : [5, 25], selected set of hyperparameters
'weights': ['uniform', 'distance’],
5. verbose: you can set it to 1 to get the detailed print out
'algorithm': ['auto', 'ball_tree', 'kd_tree’,
'brute’] while you fit the data to GridSearchCV
} 6. n_jobs: number of processes you wish to run in parallel for
this task if it -1 it will use all available processors.
grid_kn = GridSearchCV(estimator = kn,
param_grid = params,
coring = 'accuracy’,
That is all pretty much you need to define. Then you have to
cv = 5, fit your training data as you do normally. You will get the first
verbose = 1, line printed
Fitting like this:
5 folds for each of 16 candidates, totalling 80 fits
n_jobs = -1) ...
... •
Are you confused what it means?
grid_kn.fit(X_train, y_train) ... •
Simple! Since we have to try two options for
n_neighbors, two for weights and four for
algorithms, altogether there are 16 different
combinations we should try out.
• And for each combination, we have 5 CV fits, so 80
different fits will be tested by our GridSearcCV
The time for this fit depends on the number of object.
hyperparameters you are trying out. Once everything
is finished, you will get [Parallel(n_jobs=1)]:
an output like this: Done 80 out of 80 | elapsed: 74.1min finished]
Then to know what are the best parameters you can simply print it
with

# extract best estimator

print(grid_kn.best_estimator_)

Output:
KNeighborsClassifier(algorithm='auto',
leaf_size=30, metric='minkowski',metric_params=None, n_jobs=-1,
n_neighbors=25, p=2, weights='distance’)

# to test the bestfit

print(grid_kn.score(X_test, y_test))

Output:
0.9524753
 RANDOM SEARCH
 Randomly samples the search space and evaluates sets from a
specified probability distribution.
 For example, Instead of trying to check all 100,000 samples,
we can check 1000 random parameters.
 Drawback
 However, it doesn’t use information from prior experiments to select
the next set and also it is very difficult to predict the next of
experiments.
 BAYESIAN OPTIMISATION
 Hyperparameter setting maximizes the performance of the model on a validation set.
 Machine learning algorithms frequently require to fine-tuning of model
hyperparameters.
 Unfortunately, that tuning is often called as ‘black function’ because it cannot be
written into a formula since the derivates of the function are unknown.
 Much more appealing way to optimize and fine-tune hyperparameters are enabling
automated model tuning approach by using Bayesian optimization algorithm.
 The model used for approximating the objective function is called surrogate model.
 A popular surrogate model for Bayesian optimization is Gaussian process (GP).
 Bayesian optimization typically works by assuming the unknown function was
sampled from a Gaussian Process (GP) and maintains a posterior distribution for this
function as observations are made.
BAYESIAN OPTIMIZATION
 There are two major choices must be made when performing Bayesian
optimization.
 Select prior over functions that will express assumptions about the function being
optimized. For this, we choose Gaussian Process prior
 Next, we must choose an acquisition function which is used to construct a
utility function from the model posterior, allowing us to determine the next point
to evaluate.
GAUSSIAN PROCESS
 A Gaussian process defines the prior distribution over functions which can
be converted into a posterior over functions once we have seen some data.
 The Gaussian process uses Covariance matrix to ensure that values that
are close together.
 The covariance matrix along with a mean µ function to output the expected
value ƒ(x) defines a Gaussian process.
 1. Gaussian process will be used as a prior for Bayesian inference
 2. To computing the posterior is that it can be used to make predictions for
unseen test cases.
GAUSSIAN PROCESS
 Acquisition Function
 Introducing sampling data into the search space is done by acquisition functions.
It helps to maximize the acquisition function to determine the next sampling
point. Popular acquisition functions are
• Maximum Probability of Improvement (MPI)
• Expected Improvement (EI)
• Upper Confidence Bound (UCB)
 The Expected Improvement (EI) function seems to be a popular one. It is defined as
 EI(x)=𝔼[max{0,ƒ(x)−ƒ(x̂ )}]

 where ƒ(x̂ ) is the current optimal set of hyperparameters. Maximising the

hyperparameters will improve upon ƒ.
1. EI is high when the posterior expected value of the loss µ(x) is higher than the current
best value ƒ(x̂)
2. EI is high when the uncertainty σ(x)σ(x) around the point xx is high.
[email protected]
• “Random Search for Hyper-Parameter Optimization.” James Bergstra and Yoshua
Bengio. Journal of Machine Learning Research, 2012.
• “Algorithms for Hyper-Parameter Optimization.” James Bergstra, Rémi Bardenet, Yoshua
Bengio, and Balázs Kégl.” Neural Information Processing Systems, 2011. See also
a SciPy 2013 talk by the authors .
• “Practical Bayesian Optimization of Machine Learning Algorithms.” Jasper Snoek, Hugo
Larochelle, and Ryan P. Adams. Neural Information Processing Systems, 2012.
• “Sequential Model-Based Optimization for General Algorithm Configuration.” Frank Hutter,
Holger H. Hoos, and Kevin Leyton-Brown. Learning and Intelligent Optimization, 2011.
• “Lazy Paired Hyper-Parameter Tuning.” Alice Zheng and Mikhail Bilenko. International Joint
Conference on Artificial Intelligence, 2013.
• Introduction to Derivative-Free Optimization (MPS-SIAM Series on Optimization) . Andrew R.
Conn, Katya Scheinberg, and Luis N. Vincente, 2009.
• Gradient-Based Hyperparameter Optimization Through Reversible Learning. Dougal Maclaurin,
David Duvenaud, and Ryan P. Adams. ArXiv, 2015.