BIG DATA ANALYTICS
Lecture 9 --- Week 10
Content
Classification versus Regression
Supervised vs. Unsupervised Learning
Evaluating Predictive Models
Supervised Learning Algorithms
Model Evaluation
Classification vs Regression
Classification:
predicts categorical class labels
classifies data (constructs a model) based on the training set and the
values (class labels) in a classifying attribute and uses it in classifying new
data.
Regression:
models continuous-valued functions, i.e., predicts unknown or missing
values.
Typical Applications
credit approval
target marketing
medical diagnosis
treatment effectiveness analysis
Classification – A Motivating
Application
Credit approval
A bank wants to classify its customers based on whether they
are expected to pay back their approved loans
The history of past customers is used to train the classifier
The classifier provides rules, which identify potentially reliable
future customers
Classification rule:
If age = “31...40” and income = high then credit_rating = excellent
Future customers
Paul: age = 35, income = high excellent credit rating
John: age = 20, income = medium fair credit rating
Classification—A Two-Step Process
Model construction: describing a set of predetermined classes
Each tuple/sample is assumed to belong to a predefined class, as
determined by the class label attribute
The set of tuples used for model construction: training set
The model is represented as classification rules, decision trees, or
mathematical formulae
Model usage: for classifying future or unknown objects
Estimate accuracy of the model
The known label of test samples is compared with the classified result
from the model
Accuracy rate is the percentage of test set samples that are correctly
classified by the model
Test set is independent of training set, otherwise over-fitting will occur
Classification Process (1): Model
Construction
Classification
Algorithms
Training
Data
NAME RANK YEARS TENURED Classifier
Mike Assistant Prof 3 no (Model)
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes IF rank = ‘professor’
Dave Assistant Prof 6 no
OR years > 6
Anne Associate Prof 3 no
THEN tenured = ‘yes’
Classification Process (2): Use the
Model in Prediction
Accuracy=?
Classifier
Testing
Data Unseen Data
(Jeff, Professor, 4)
NAME RANK YEARS TENURED
Tom Assistant Prof 2 no Tenured?
Mellisa Associate Prof 7 no
George Professor 5 yes
Joseph Assistant Prof 7 yes
Classification (Training Phase)
In the first step, a classifier is built describing a predetermined set of data
classes or concepts.
This is the learning step (or training phase), where a classification
algorithm builds the classifier by analyzing or “learning from” a training set
made up of database tuples and their associated class labels.
A tuple, X, is represented by an n-dimensional attribute vector, X = (x1,
x2, …. , xn), depicting n measurements made on the tuple from n database
attributes, respectively, A1, A2, ….. , An.
Each tuple, X, is assumed to belong to a predefined class as determined by
another database attribute called the class label attribute.
The class label attribute is discrete-valued and unordered.
It is categorical (or nominal) in that each value serves as a category or class.
The individual tuples making up the training set are referred to as training
tuples and are randomly sampled from the database under analysis. In the
context of classification, data tuples can be referred to as samples, examples,
instances, data points, or objects.
This first step of the classification process can also be viewed as the learning of a
mapping or function, y = f (X), that can predict the associated class label y of a
given tuple X.
In this view, we wish to learn a mapping or function that separates the data
classes.
This mapping is represented in the form of classification rules, decision trees, or
mathematical formulae.
The mapping is represented as classification rules that identify loan
applications as being either safe or risky.
The rules can be used to categorize future data tuples, as well as
provide deeper insight into the data contents.
They also provide a compressed data representation.
Classification (Testing Phase)
In the second step, the model issued for classification.
First, the predictive accuracy of the classifier is estimated.
If we were to use the training set to measure the classifier’s accuracy,
this estimate would likely be optimistic, because the classifier tends
to overfit the data (i.e., during learning it may incorporate some
particular anomalies of the training data that are not present in the
general data set overall).
Supervised vs. Unsupervised
Learning
Supervised learning (classification)
Supervision: The training data (observations,
measurements, etc.) are accompanied by labels indicating
the class of the observations
New data is classified based on the training set
Unsupervised learning (clustering)
The class labels of training data is unknown
Given a set of measurements, observations, etc. with the
aim of establishing the existence of classes or clusters in
the data
Evaluating Predictive Models
Predictive Accuracy
Speed
time to construct the model
time to use the model
Robustness
handling noise and missing values
Scalability
efficiency in disk-resident databases
Interpretability:
understanding and insight provided by the model
Goodness of rules (quality)
True Positives, True Negatives, False Negatives, False Positives
compactness of classification rules
Supervised Learning Algorithms
Artificial Neural Network
Linear Regression
Support Vector Machine
Artificial Neural Networks
Perceptron
Developed by Frank Rosenblatt by using McCulloch and Pitts model,
perceptron is the basic operational unit of artificial neural networks. It
employs supervised learning rule and is able to classify the data into
two classes.
Operational characteristics of the perceptron: It consists of a single
neuron with an arbitrary number of inputs along with adjustable
weights, but the output of the neuron is 1 or 0 depending upon the
threshold. It also consists of a bias whose weight is always 1.
Following figure gives a schematic representation of the perceptron.
Perceptron thus has the following three basic elements −
Links − It would have a set of connection links, which carries a
weight including a bias always having weight 1.
Adder − It adds the input after they are multiplied with their
respective weights.
Activation function − It limits the output of neuron. The most basic
activation function is a Heaviside step function that has two possible
outputs. This function returns 1, if the input is positive, and 0 for any
negative input.
Linear Regression
Linear regression may be defined as the statistical model that
analyzes the linear relationship between a dependent variable with
given set of independent variables. Linear relationship between
variables means that when the value of one or more independent
variables will change (increase or decrease), the value of dependent
variable will also change accordingly (increase or decrease).
Mathematically the relationship can be represented with the help of
following equation −
Y = mX + b
Here, Y is the dependent variable we are trying to predict
X is the dependent variable we are using to make predictions.
m is the slop of the regression line which represents the effect X
has on Y
b is a constant, known as the Y-intercept. If X = 0,Y would be
equal to b.
Positive Linear Relationship
A linear relationship will be called positive if both independent and
dependent variable increases. It can be understood with the help of
following graph −
Negative Linear relationship
A linear relationship will be called negative if independent increases
and dependent variable decreases. It can be understood with the help
of following graph −
Support Vector Machines
An SVM model is basically a representation of different classes in a
hyper-plane in multidimensional space. The hyper-plane will be
generated in an iterative manner by SVM so that the error can be
minimized. The goal of SVM is to divide the datasets into classes to
find a maximum marginal hyper-plane (MMH).
The followings are important concepts in SVM −
Support Vectors − Data-points that are closest to the hyper-plane is
called support vectors. Separating line will be defined with the help of
these data points.
Hyper-plane − As we can see in the above diagram, it is a decision
plane or space which is divided between a set of objects having
different classes.
Margin − It may be defined as the gap between two lines on the
closet data points of different classes. It can be calculated as the
perpendicular distance from the line to the support vectors. Large
margin is considered as a good margin and small margin is
considered as a bad margin.
The main goal of SVM is to divide the datasets into classes to find a
maximum marginal hyper-plane (MMH) and it can be done in the
following two steps −
First, SVM will generate hyper-planes iteratively that segregates
the classes in best way.
Then, it will choose the hyper-plane that separates the classes
correctly.
Model Evaluation
Metrics for Performance Evaluation
How to evaluate the performance of a model?
Methods for Performance Evaluation
How to obtain reliable estimates?
Methods for Model Comparison
How to compare the relative performance among competing models?
Metrics for Performance Evaluation
Focus on the predictive capability of a model
Rather than how fast it takes to classify or build models, scalability, etc.
Confusion Matrix:
PREDICTED CLASS
Class=Yes Class=No
a: TP (true positive)
ACTUAL Class=Yes a b
b: FN (false
CLASS negative)
Class=No c d
c: FP (false
positive)
d: TN (true
Metrics for Performance
Evaluation… PREDICTED CLASS
Class=Yes Class=No
ACTUAL Class=Yes a b
(TP) (FN)
CLASS
Class=No c d
(FP) (TN)
Most widely-used metric:
ad TP TN
Accuracy
a b c d TP TN FP FN
Methods of Estimation
Holdout
Reserve 2/3 for training and 1/3 for testing
Random subsampling
One sample may be biased -- Repeated holdout
Cross validation
Partition data into k disjoint subsets
k-fold: train on k-1 partitions, test on the
remaining one
Leave-one-out: k=n
Guarantees that each record is used the same number of times for training
and testing
Bootstrap
Sampling with replacement
~63% of records used for training, ~27% for testing
ROC (Receiver Operating Characteristic)
Developed in 1950s for signal detection theory to
analyze noisy signals
Characterize the trade-off between positive hits and false
alarms
ROC curve plotsTP TPR (on the y-axis) against FPR
TPR
(on the x-axis) PREDICTED CLASS
TP FN
Fraction of positive Yes No
instances predicted as
positive Yes a b
FP Actual (TP) (FN)
FPR No c d
FP TN
(FP) (TN)
Fraction of negative
instances predicted as
positive
