EIE522

Pattern Recognition:
Theory & Applications
Prof. Kenneth K.M. Lam
Department of Electronic & Information Engineering
The Hong Kong Polytechnic University
Room DE632
Email: [email protected]
 Intended Learning Outcomes
Upon completion of the subject, students shall be able to
a. Understand and analyze methods for automatic training of
classification systems based on typical statistical, syntactic
and neural network approaches;
b. Understand common feature extraction methods for pattern
recognition;
c. Design systems and algorithms for pattern recognition;
d. Implement typical pattern recognition algorithms in
MATLAB/Python;
e. Present ideas and findings effectively; and
f. Think critically and learn independently.

January 2021 Pattern Recognition P.2

Assessments:

Assignments 2 (5% @) 10%

Continuous
Assessment Lab Reports 2 (9%/6%) 15%
Tests 2 (25%) 25%
Examination 50%

January 2021 Pattern Recognition P.3

Tutors:

Mr Tianshan Liu
Room: DE503
email: [email protected]

Mr Jun Xiao
Room: DE503
email: [email protected]

January 2021 Pattern Recognition P.4

 References:
❑ Richard O. Duda, Peter E. Hart & David G. Stork,
Pattern Classification, 2nd Edition, John Wiley, 2001.
❑ Rafael C. Gonzalez and Richard E. Wood, Digital
Image Processing, 2nd Edition, Addison-Wesley, 2001.
❑ Robert Schalkoff, Pattern Recognition - Statistical,
Structural and Neural Approaches, John Wiley, 1992.
❑ Andrew Webb & Keith Copsey, Statistical Pattern
Recognition, 3rd Edition Wiley, 2011 (online access)
❑ J.M. Zurada, Introduction to Artificial Neural Systems,
West Publishing, 1992.
❑ Ian Goodfellow, Y. Bengio, and A. Courville, Deep
Learning, MIT Press, 2016 (www.deeplearningbook.org)
January 2021 Pattern Recognition P.5
Pattern Recognition P.6
January 2021 Pattern Recognition P.7
 Pattern Recognition
❑ Pattern recognition is the science that concerns the
description or classification (recognition) of
measurements
– Similar structures can be grouped together to form
a single class called a pattern

– Taking in raw data and making an action based on

the category of the pattern
January 2021 Pattern Recognition P.8
❑ Computer implementation of pattern recognition
algorithms requires discrete digital representations of
the patterns
❑ Three approaches to pattern recognition
– Statistical (decision theoretic)
• Focus on the statistical properties of the patterns
– Machine Learning
• Support Vector Machine, Random Forest,
AdaBoosting, …
– Computational Intelligence Techniques (Artificial
Neural Networks, Evolutionary Computation, Fuzzy
Systems)
January 2021 Pattern Recognition P.9
 Applications of Pattern Recognition
❑ Facial Expression Recognition

ANGER DISGUST FEAR

HAPPINESS SADNESS SURPRISE

Feature Vector 1

Feature Vector 2 Classifier Expression

January 2021 Pattern Recognition P.10

 ❑ Face Recognition Matching

Face Facial Feature

Detection Detection &
Extraction

Input Image

❑ To develop a face recognition system: Face Database

– face detection, facial feature detection, facial feature
extraction, face recognition, face database and indexing
January 2021 Pattern Recognition P.11
 48

••••

❑ Human Detection

• •
• •
• • 96
• •

••••

HOG:
Histogram of A cell of
size 88
Gradients
pixels


January 2021 Pattern Recognition P.12
❑ Human Action Recognition

Step Run Hop Jump

Skip Swivel turn Crawl

January 2021 Pattern Recognition P.13
❑ Optical Character Recognition (OCR)

Printing
Handwriting
Handwritten

January 2021 Pattern Recognition P.14

❑ Identification and Authentication
Minutia Extraction：
Determine location
and orientation of
ridge bifurcations
and ridge
terminations

January 2021 Pattern Recognition P.15

 Structure of a Typical PR System
❑ Choose a discretization method in order to obtain a
measurement vector (feature vector, descriptor) for
each sample pattern
❑ Usually the dimension of the measurement space is
very large
 Reduce the dimension by mapping from Z-space
into X-space, where dim(X) << dim(Z),
while retaining as many properties or features of
the original samples as possible
 Feature Selection and Extraction

January 2021 Pattern Recognition P.16

❑ Suppose L patterns and pattern classes denoted by
ci, 1  i  L
xi,j  ci, 1  i  L, 1  j  ni
A pattern recognition system is a system which
takes a new sample x* of unknown classification and
assigns it to some pattern class ci (1  i  L) on the
basis of some decision rule
❑ The decision rule is often obtained by partitioning the
pattern space into disjoint regions corresponding to
the classes ci

January 2021 Pattern Recognition P.17

 C10 C1
C9

C4 C2

C8
C3

C7 C6
C5

Possible algorithm feedback or interaction

observed
world pattern
feature/
data pi sensor/ preprocessing classification
primitive classification
&
transducer extraction
enhancement algorithm
algorithm

measurement, mi
January 2021 Pattern Recognition P.18
 Feature Selection & Extraction

❑ It is important to choose and to extract features that

1. are computationally feasible
2. lead to “good” PR system
3. reduce the problem data into a manageable
amount of information without discarding
valuable information
❑ Feature selection is the process of choosing input to
the PR system and involves judgement

January 2021 Pattern Recognition P.19

❑ There are many different types of features that we
can select; this depends on the objects to be
classified and the applications
– e.g. color, shape, texture, ….

color texture shape

– Haar like feature (for face detection), histogram of
gradients (HOG) (for human and vehicle
detection), Gabor wavelets & local binary pattern
(for texture and face recognition), …
January 2021 Pattern Recognition P.20
❑ In many situations, a set of patterns from the same
class may exhibit wide variations from a single
exemplar of the class
 invariant features: e.g. translation, rotation,
scale

❑ The features are then passed to a classifier that

evaluates the evidence presented and makes a final
decision as to the classes
– Bayes classifier, AdaBoost, Support Vector
Machine, Neural Network, …
January 2021 Pattern Recognition P.21
 Feature Vector & Feature Space
❑ Features are arranged in a d-dimensional feature
vector, denoted as x, which yields a multi-
dimensional measurement space or feature space
𝑥1
12.00 a 2-D space 𝑥
2
8.00

4.00 class w1

x2 0.00
class w2
-4.00

-8.00
Scatter Plot
-12.00
-6.00 -4.00 -2.00 0.00 2.00 4.00 6.00
x1
January 2021 Pattern Recognition P.22
 Classifiers, Decision Regions &
Boundaries, & Discriminant Functions

❑ A classifier partitions feature space into class-labeled

decision regions
❑ The border of each decision region is a decision
boundary
❑ In a c-class case, discriminant functions, denoted as
gi(x), i = 1,2,…,c, are used to partition d as follows:
Decision Rule :
Assign x to class m (region Rm), where
gm(x) > gi(x)  i = 1, 2, .., c & i  m

January 2021 Pattern Recognition P.23

 R2 R1 R2
R1
R3
R2
R1
R2 R3
R1
R3 R4

Linear (piecewise) Quadratic (hyperbolic) (Relatively) general

❑ Linear discriminant function:

x vector
𝑔𝑖 𝒙 = 𝒘𝑇𝑖 𝒙 + 𝑤0𝑖 x scalar
❑ For 2-d case: X matrix
𝑥1
𝑔𝑖 𝒙 = 𝑤𝑖1 𝑤𝑖2 𝑥2 + 𝑤0𝑖
= 𝑤𝑖1 𝑥1 + 𝑤𝑖2 𝑥2 +𝑤0𝑖
January 2021 Pattern Recognition P.24
 Training & Learning in PR Systems

❑ Training set, H – a set of “typical” patterns, where

typical attributes or the class or structure of each is
known
 Provide significant information on how to
associate input data with output decision

❑ H = {x1, x2, . . ., xn} → H = {H1, H2, . . ., Hc}

If a set of linear discriminant functions can be used to
correctly classify every element of H, then a linearly
separable problem results

January 2021 Pattern Recognition P.25

❑ Supervised learning
– A system trained on the basis of labeled samples

❑ Unsupervised learning
– An adaptive pattern classifier trained with
unlabeled samples
– Find subsets of the samples, called clusters,
whose elements are mutually 'close' but far away
from members of other clusters

January 2021 Pattern Recognition P.26

 n+1 n+2 n+3

Cluster 2
Cluster 1

Cluster 3 New cluster

Reference: R. Schalkoff, Chapter 1

January 2021 Pattern Recognition P.27