Université d’Alger 1
Benyouçef benkhedda
Data Mining
Dr . BOUFENAR Chaouki
Master 1
Ingénierie des Systèmes
Informatiques Intelligents
2018/2019
18/05/2019 Cours de Data Mining 1
What Motivated Data Mining?
Natural evolution of information technology
Wide availability of huge amounts of data
Imminent need for turning data into useful information
18/05/2019 Cours Data mining 2
What is Data Mining ?
“ Data mining refers to extracting or “mining” knowledge from large amounts
of data ” [1]
The term “ Data Mining ” is misnomer ?!
More appropriate term is “ Knowledge Mining ”
Data mining = Knowledge Discovery from Data (KDD)
18/05/2019 Cours Data mining 3
What is Data Mining ?
Data mining is the core of KDD process
Cleaning Preprocessing Transformation Data mining Evaluation
18/05/2019 Cours Data mining 4
Data mining as a confluence of
multiple disciplines
Data Mining
Algorithms Visualisation
18/05/2019 Cours Data mining 5
Goals of Data Mining
Predictions
Earthquakes Sales Volumes
18/05/2019 Cours Data mining 6
Goals of Data Mining
Identification
Security & Crime Detection
Mining Gene Expression on
data for Drug Discovery
18/05/2019 Cours Data mining 7
Goals of Data Mining
Classification
18/05/2019 Cours Data mining 8
Goals of Data Mining
Optimisation
Time Optimisation
Space Optimisation
Sales maximisation
18/05/2019 Cours Data mining 9
Data Mining Techniques
18/05/2019 Cours Data mining 10
Classification Vs Prediction
Target attributes
Categorical/Discret Numerical/Continuous
Classification Prediction
learn which loan applicants are “safe” and
which are “risky” for the bank marketing manager would like to predict
how much a given customer will spend
analyze breast cancer data in order to
during a sale at AllElectronics
predict which one of three specific
treatments a patient should receive
18/05/2019 Cours Data mining 11
Classification process
Learning
Training data
Classification
Algorithm
Classification rules
IF age = youth THEN
loan_decision = risky
IF income = high THEN
loan_decision = safe
IF age = middle_aged AND income = low
THEN loan_decision = risky
18/05/2019 Cours Data mining 12
Classification process
Classification
Test data
Classification
rules
New data
(Dnnnn , Middle-age , Low )
loan_decision = ?
risky
18/05/2019 Cours Data mining 13
Data Cleaning and Preprocessing
Real-world data
Missing values Noisy inconsistent
Ignore the tuple
Fill in the missing value manually
Use a global constant (“Unknown” or −∞)
Use the attribute mean
Use the most probable value
18/05/2019 Cours Data mining 14
Data Cleaning and Preprocessing
Real-world data
Missing values Noisy inconsistent
Noise is a random error or variance in a measured variable
Example
Sorted data for price (in dollars): 4, 8, 15, 21, 21, 24, 25, 28, 34
Irregular data Smoothing regular data
Remove irregularities
Binning Regression Clustering
18/05/2019 Cours Data mining 15
Data Cleaning and Preprocessing
4 8 15 21 21 24 25 28 34
Binning
Partition into (equal-frequency) bins
Bin 1 Bin 2 Bin 3
4 8 15 21 21 24 25 28 34
Smoothing by bin means Smoothing by bin boundaries
9 9 9 22 22 22 29 29 29 4 4 15 21 21 24 25 25 34
Bin 1 Bin 2 Bin 3 Bin 1 Bin 2 Bin 3
18/05/2019 Cours Data mining 16
Data Cleaning and Preprocessing
Regression
Regression is a set of statistical methods for estimating the relationships among variables
Linear regression quantifies the relationship between one or more predictor variables
(independent or explanatory variables) and one outcome variable (dependent variable)
18/05/2019 Cours Data mining 17
Data Cleaning and Preprocessing
Clustering
similar values are organized into groups, or clusters. Values that fall outside of the set of
clusters may be considered outliers
18/05/2019 Cours Data mining 18
Data Transformation
Data are transformed or consolidated in to forms in appropriate mining.
Smoothing Aggregation Generalisation Normalisation Attribute construction
18/05/2019 Cours Data mining 19
Data Transformation
Data are transformed or consolidated in to forms in appropriate mining.
Aggregation Generalisation Normalisation Attribute construction
The daily sales data may be aggregated so as to compute monthly and annual total amounts.
Data cubes store multidimensional aggregated information.
18/05/2019 Cours Data mining 20
Data Transformation
Data are transformed or consolidated in to forms in appropriate mining.
Generalisation Normalisation Attribute construction
Low-level Low-level
Generalisation
data concepts
Categorical attributes, like street, can be generalized to higher-level concepts, like
city or country
values for numerical attributes, like age, may be mapped to higher-level concepts, like
youth, middle-aged, and senior
18/05/2019 Cours Data mining 21
Data Transformation
Data are transformed or consolidated in to forms in appropriate mining.
Normalisation Attribute construction
the attribute data are scaled so as to fall within a small specified range, such as -1.0 to 1.0
or 0.0 to 1.0
Min-max Normalisation
Let income range $12,000 to $98,000 normalized to [0.0, 1.0].
73,000 − 12,000
• Then $73,000 is mapped to : 1.0 − 0 + 0 = 0,716
98,000 − 12,000
z-score normalization
Let Ā= 54,000, σA= 16,000, for the attribute income
73,600 − 54,000
• a value of $73,600 for income is transformed to : = 1,225
16,000
18/05/2019 Cours Data mining 22
Data Transformation
Data are transformed or consolidated in to forms in appropriate mining.
Attribute construction
new attributes are constructed and added from the given set of attributes to help the mining
process
we may wish to add the attribute area based on the attributes height and width.
By attribute construction can discover missing values.
18/05/2019 Cours Data mining 23
Attributes Subset Selection
Data sets for analysis may contain hundreds of attributes, many of which may be
irrelevant to the mining task or redundant.
if the task is to classify customers as to whether or not they are likely to purchase a popular
new CD at AllElectronics when notified of a sale, attributes such as the customer’s
telephone number are likely to be irrelevant, unlike attributes such as age or music_taste.
Speed-up the mining processes by removing the irrelevent or redundant attributes
Reduces the number of attributes appearing in the discovered patterns
Make the patterns easier to understand.
18/05/2019 Cours Data mining 24
Data Mining
Techniques
Supervised Unsupervised
Classification Clustering
o Data are labeled with pre-defined classes o Class labels are Unknown
o Test data are classified into these classes o Establish the existence of classes (Clusters)
in the data
18/05/2019 Cours Data mining 25
Performance Measure
Confusion Matrix
Error Rate =
Precision =
FP Rate =
Specificity = 1-FP Rate
Sensibility = Rappel =
18/05/2019 Cours Data mining 26
Spliting Dataset
Data set
Train Validation Test
Training Dataset : The sample of data used to fit the model.
Validation Dataset: The sample of data used to provide an unbiased evaluation of
a model fit on the training dataset while tuning model hyper-parameters.
Test Dataset: The sample of data used to provide an unbiased evaluation of a final
model fit on the training dataset.
18/05/2019 Cours Data mining 27
Overfitting
Overfitting : a model that is too specialized on Training Set data and that will not
generalize well
• Properties
• Generalisable correlations
• Fluctuations
• Random variations
lead to Overfitting Badly predictions on Test data
• Noise
• Outlier
18/05/2019 Cours Data mining 28
Overfitting
• Blue line : a prediction function
• Green points : Training Set data
• Red points : Testing Set data
18/05/2019 Cours Data mining 29
Overfitting
Underfitting Appropriate fitting Overfitting
High bias High Variance
18/05/2019 Cours Data mining 30
How to avoid Overfitting?
Gather more data
Data augmentation
Simplify the model
Early termination
L1 and L2 regularization
For Deep Learning : Dropout and Dropconnect
18/05/2019 Cours Data mining 31
How to avoid Overfitting?
Gather more data The more data you get, the less likely the model is to overfit.
Adding more data
Data augmentation
Simplify the model the model becomes unable to overfit all the samples
Early termination forced the model to generalize
L1 and L2 regularization
For Deep Learning : Dropout and Dropconnect
18/05/2019 Cours Data mining 32
How to avoid Overfitting?
Gather more data
Data augmentation The more data you get, the less likely the model is to overfit.
18/05/2019 Cours Data mining 33
How to avoid Overfitting?
Gather more data
Data augmentation Collecting more data is a tedious and expensive process
Dilatation, Rotation, Adding noise, …
Simplify the model
Early termination
L1 and L2 regularization
For Deep Learning : Dropout and Dropconnect
18/05/2019 Cours Data mining 34
How to avoid Overfitting?
Gather more data
Reducing its complexity
Data augmentation # of estimators in a random forest,
# of parameters in a neural network
Simplify the model
Early termination
Model lighter, train faster and run faster.
L1 and L2 regularization
For Deep Learning : Dropout and Dropconnect
18/05/2019 Cours Data mining 35
How to avoid Overfitting?
Gather more data
Data augmentation
Simplify the model
Early termination
L1 and L2 regularization When the testing error starts to increase, it’s time to stop!
For Deep Learning : Dropout and Dropconnect
18/05/2019 Cours Data mining 36
How to avoid Overfitting?
add a penalty to the loss function
Gather more data
The L1 penalty The L2 penalty
Data augmentation
Simplify the model
minimize the squared magnitude
Early termination minimize the squared magnitude
L1 and L2 regularization • The model is forced to make compromises on its weights,
as it can no longer make them arbitrarily large.
For Deep Learning : Dropout and Dropconnect
• This makes the model more general, which helps combat
overfitting.
18/05/2019 Cours Data mining 37
How to avoid Overfitting?
Gather more data
Data augmentation
Simplify the model
Early termination
L1 and L2 regularization
For Deep Learning : Dropout and Dropconnect
Randomly deactivate either neurons (dropout) or connections (dropconnect) during the
training.
18/05/2019 Cours Data mining 38
Data Mining
Supervised
Step 01 : Learning
Better generalisable
Learning
Training data Model
Algorithm
Step 02 : Testing
Test Model Evaluation
data
18/05/2019 Cours Data mining 39
Data Mining
Supervised Learning
18/05/2019 Cours Data mining 40
Data Mining
Unsupervised
The data have no target attribute
We want to explore the data to find some intrinsic structures in them
Methods
Hierarchical Not Hierarchical
Hierarchical Cluster Analysis K-Means self organizing maps Clustering
Classification Ascendante centres mobiles Cartes topologiques de
Hiérarchique Kohonen
18/05/2019 Cours Data mining 41
Data Mining
Unsupervised
Hierarchical method
The optimal number of classes is determined by reading the tree
Very expensive in computation time
Not Hierarchical method
Allow a classification of a huge sets of data
We initially impose the number of classes
18/05/2019 Cours Data mining 42
Data Mining
Hierarchical method
18/05/2019 Cours Data mining 43
Data Mining
Hierarchical method
Principe
Repeatdely combine two nearest objects
Data structures
Data matrix (Object-by-Variable) Dissimilarity matrix (Object-by-Object)
n objets (persons)
p variables (age, height, weight, gender, …)
18/05/2019 Cours Data mining 44
Data Mining
Hierarchical method
Classification criteria
Similarity measurement between objects : Euclidean distance
, = −
Similarity measurement between groups of objects
• Single linkage (smaller distance)
• Complete linkage (Large distance)
18/05/2019 Cours Data mining 45
Data Mining
Hierarchical method
How partition is good?
Intra cluster distance for each cluster is Min
Two criterias ?????
Inter cluster distance for each cluster is Max
Intra cluster distance = SSE (Sum Square Error ) intra class or cluster = Intra inertia
− : The mean in cluster q
Inter cluster distance = SSE (Sum Square Error ) inter class or cluster = Inter inertia
− : The mean in all
18/05/2019 Cours Data mining 46
Data Mining
Hierarchical method
How partition is good?
Huygens theory
− = − + −
One criteria
18/05/2019 Cours Data mining 47
References
J. Han, M. Kamber, Data Mining: Concepts and Techniques, Elsevier
Inc. (2006).
18/05/2019 Cours Data mining 48
Cours de Compilation L3 SI - Analyse
18/05/2019 49
Syntaxique
