REGULARIZATION

Regularization is a technique used in machine learning and deep learning to prevent a model
from overfitting to the training data. Overfitting occurs when a model learns the noise or
random fluctuations in the training data instead of capturing the underlying patterns, leading
to poor generalization to new, unseen data.
Regularization methods add extra constraints or penalties to the model's learning process,
encouraging it to be simpler and more generalized. By preventing the model from becoming
too complex or fitting the noise, regularization helps ensure that the model performs well not
only on the training set but also on test data or new data.
Types of Regularization
1. L1 Regularization (Lasso): L1 regularization adds a penalty proportional to the
absolute value of the weights of the model. The regularization term in the loss
function is the sum of the absolute values of the model’s parameters (weights):
LL1=λ∑i∣wi∣\mathcal{L}_{L1} = \lambda \sum_{i} |w_i|LL1=λi∑∣wi∣
Where:
o λ\lambdaλ is a hyperparameter controlling the strength of the regularization.
o wiw_iwi represents the model’s weights.
The primary effect of L1 regularization is that it can drive some of the weights to exactly
zero, effectively performing feature selection. This makes L1 useful when we want to identify
a sparse set of important features and remove irrelevant ones.
Advantages:
o Promotes sparsity, i.e., it forces some weights to be zero, leading to simpler,
more interpretable models.
o Useful when working with high-dimensional data (e.g., in sparse settings).
2. L2 Regularization (Ridge): L2 regularization adds a penalty proportional to the
squared value of the weights. The regularization term in the loss function is the sum
of the squares of the weights:
LL2=λ∑iwi2\mathcal{L}_{L2} = \lambda \sum_{i} w_i^2LL2=λi∑wi2
Where:
o λ\lambdaλ is again a hyperparameter controlling the strength of the
regularization.
o wiw_iwi represents the model’s weights.
L2 regularization prevents the model from assigning excessively large weights to any feature.
It encourages the weights to be small and evenly distributed, which can lead to better
generalization.
Advantages:
 o Helps to avoid overfitting by shrinking large weights, thereby simplifying the
model.
o Works well when many features contribute to the model, and no one feature is
overwhelmingly important.
3. Elastic Net Regularization: Elastic Net regularization combines both L1 and L2
regularization. The loss function is a linear combination of the L1 and L2 penalties:
LElasticNet=λ1∑i∣wi∣+λ2∑iwi2\mathcal{L}_{ElasticNet} = \lambda_1 \sum_{i} |w_i| + \
lambda_2 \sum_{i} w_i^2LElasticNet=λ1i∑∣wi∣+λ2i∑wi2
Where:
o λ1\lambda_1λ1 and λ2\lambda_2λ2 control the strength of L1 and L2
regularization, respectively.
Elastic Net is useful when there are many correlated features in the data. It inherits the
advantages of both L1 and L2 regularization: L1 can perform feature selection (leading to
sparse solutions), and L2 helps reduce the risk of overfitting.
4. Dropout: Dropout is a regularization technique used in deep learning, where during
training, randomly selected neurons (along with their connections) are "dropped" or
set to zero. This forces the model to rely on multiple paths and learn more robust
features.
o During training, for each forward pass, dropout randomly disables a fraction
of neurons (say 50%).
o During testing, dropout is turned off, and the full network is used, but the
weights are scaled down to account for the fact that some neurons were
dropped during training.
Advantages:
o Prevents the network from becoming too reliant on specific neurons, thus
avoiding overfitting.
o Helps to create a more generalized model by forcing the network to learn
redundant representations.
5. Early Stopping: Early stopping is a technique that halts the training process when the
model’s performance on a validation set stops improving. Typically, the training
continues until the validation error starts to increase, signaling that the model is
starting to overfit.
Advantages:
o Helps prevent overfitting by stopping training at the point where the model has
learned the most generalizable features.
o Doesn't require adding extra terms to the loss function.
 6. Data Augmentation: Data augmentation is a technique used to artificially increase
the size of the training dataset by applying transformations to the existing data. These
transformations might include random rotations, flips, shifts, and scalings of images
or adding noise to data.
Advantages:
o Helps to generalize the model by exposing it to a wider variety of input
variations.
o Prevents overfitting by providing more diverse examples for the model to
learn from.
7. Weight Regularization (or Weight Decay): Weight regularization, often referred to
as weight decay, involves adding a penalty on the weights during training (similar to
L2 regularization). The idea is to penalize large weights in the model by adding a term
to the loss function that discourages large parameter values.
The loss function becomes:
L=Loriginal+λ∑iwi2\mathcal{L} = \mathcal{L}_{original} + \lambda \sum_{i}
w_i^2L=Loriginal+λi∑wi2
Where Loriginal\mathcal{L}_{original}Loriginal is the original loss function, and the
additional λ∑iwi2\lambda \sum_{i} w_i^2λ∑iwi2 is the regularization term.
Advantages:
o Helps prevent the model from overfitting by discouraging overly complex
solutions.
o Encourages the model to learn more general, simpler patterns.
Summary of Regularization Techniques:
 L1 Regularization: Adds a penalty on the absolute values of weights, promoting
sparsity (some weights may be zero).
 L2 Regularization: Adds a penalty on the square of weights, preventing large
weights and improving generalization.
 Elastic Net Regularization: A combination of L1 and L2 regularization, useful when
features are highly correlated.
 Dropout: Randomly disables neurons during training to prevent the model from over-
relying on specific units.
 Early Stopping: Stops training when validation performance stops improving,
preventing overfitting.
 Data Augmentation: Increases training data variety to help the model generalize
better.
 Weight Decay: A specific form of L2 regularization applied to the weights of the
model.