Neural Networks

Marcello Pelillo
University of Venice, Italy

Artificial Intelligence
a.y. 2017/18
 DARPA Neural Network Study (1989)

“Over the history of computing science, two advances have matured: High
speed numerical processing and knowledge processing (Artificial Intelligence).
Neural networks seem to offer the next necessary ingredient for intelligent
machines − namely, knowledge formation and organization.”
DARPA Neural Network Study (1989)

Two key features which, it is widely believed, distinguish neural

networks from any other sort of computing developed thus far:

Neural networks are adaptive, or trainable. Neural networks are

not so much programmed as they are trained with data − thus
many believe that the use of neural networks can relieve today’s
computer programmers of a significant portion of their present
programming load. Moreover, neural networks are said to
improve with experience − the more data they are fed, the more
accurate or complete their response.

Neural networks are naturally massively parallel. This suggests

they should be able to make decisions at high-speed and be fault
tolerant.
 History
Early work (1940-1960)
• McCulloch & Pitts (Boolean logic)
• Rosenblatt (Learning)
• Hebb (Learning)

Transition (1960-1980)
• Widrow – Hoff (LMS rule)
• Anderson (Associative memories)
• Amari

Resurgence (1980-1990’s)
• Hopfield (Ass. mem. / Optimization)
• Rumelhart et al. (Back-prop)
• Kohonen (Self-organizing maps)
• Hinton , Sejnowski (Boltzmann machine)

New resurgence (2012 -)

• CNNs, Deep learning, GAN’s ….
 A Few Figures

The human cerebral cortex is composed of about

100 billion (1011) neurons

of many different types.

Each neuron is connected to other 1000 / 10000 neurons, wich yields

1014/1015 connections

The cortex covers about 0.15 m²

and is 2-5 mm thick
 The Neuron
Cell Body (Soma): 5-10 microns in diameter

Axon: Output mechanism for a neuron; one axon/cell, but thousands of

branches and cells possible for a single axon

Dendrites: Receive incoming signals from other nerve axons via synapse
 Neural Dynamics
The transmission of signal in the cerebral cortex is a complex process:

electrical  chemical  electrical

Simplifying :

1) The cellular body performs a “weighted sum” of the incoming signals

2) If the result exceeds a certain threshold value, then it produces an

“action potential” which is sent down the axon (cell has “fired”),
otherwise it remains in a rest state

3) When the electrical signal reaches the synapse, it allows the

“neuro-transmitter” to be released and these combine with the
“receptors” in the post-synaptic membrane

4) The post-synaptic receptors provoke the diffusion of an electrical

signal in the post-synaptic neuron
 Synapses

SYNAPSE is the relay point where information is conveyed by chemical transmitters from neuron
to neuron. A synapse consists of two parts: the knowblike tip of an axon terminal and the receptor
region on the surface of another neuron. The membranes are separated by a synaptic cleft some
200 nanometers across. Molecules of chemical transmitter, stored in vesicles in the axon terminal,
are released into the cleft by arriving nerve impulses. Transmitter changes electrical state of the
receiving neuron, making it either more likely or less likely to fire an impulse.
 Synaptic Efficacy

It’s the amount of current that enters into the post-synaptic neuron,
compared to the action potential of the pre-synaptic neuron.

Learning takes place by modifying the synaptic efficacy.

Two types of synapses:

• Excitatory : favor the generation of action potential

in the post-synaptic neuron

• Inhibitory : hinder the generation of action potential

 The McCulloch and Pitts Model (1943)
The McCulloch-Pitts (MP) Neuron is modeled as a binary threshold unit

The unit “fires” if the net input åwIj j j

reaches (or exceeds) the unit’s threshold T:
æ ö
y = g ççå w j I j - T ÷÷
è j ø
If neuron is firing, then its output y is 1, otherwise it is 0.
ì 0 if x < 0
g is the unit step function: g(x) = í
î 1 if x ³ 0

Weights wij represent the strength of the synapse between neuron j and neuron i
 Properties of McCulloch-Pitts Networks
By properly combining MP neurons one can simulate the behavior of any
Boolean circuit.

Three elementary logical operations (a) negation, (b) and, (c) or. In each diagram
the states of the neurons on the left are at time t and those on the right at time t +1.

The construction for the exclusive or

Network Topologies and Architectures
• Feedforward only vs. Feedback loop (Recurrent networks)
• Fully connected vs. sparsely connected
• Single layer vs. multilayer

Multilayer perceptrons, Hopfield networks,

Boltzman machines, Kohonen networks, …

(a) A feedforward network and (b) a recurrent network

 Classification Problems

Given :
1) some “features”: f1 , f 2 ,...., f n
2) some “classes”: c1 ,....,c m

Problem :
To classify an “object” according to its features
 Example #1
To classify an “object” as :
c1 = “ watermelon ”
c2 = “ apple ”
c3 = “ orange ”

According to the following features :

f1 = “ weight ”
f2 = “ color ”
f3 = “ size ”

Example :
weight = 80 g
color = green “apple”
size = 10 cm³
 Example #2

Problem: Establish whether a patient got the flu

• Classes : { “ flu ” , “ non-flu ” }

• (Potential) Features :

f1 : Body temperature
f2 : Headache ? (yes / no)
f3 : Throat is red ? (yes / no / medium)
f4 :
 Example #3
Hand-written digit recognition
 Example #4:
Face Detection
 Example #5:
Spam Detection
 Geometric Interpretation
Example:
Classes = { 0 , 1 }
Features = x , y : both taking value in [ 0 , +∞ [

Idea: Objects are represented as “point” in a geometric space

 Neural Networks for Classification
A neural network can be used as a classification device .

Input ≡ features values

Output ≡ class labels

Example : 3 features , 2 classes

 Thresholds
We can get rid of the thresholds associated to neurons by adding an
extra unit permanently clamped at -1.

In so doing, thresholds become weights and can be adaptively adjusted

during learning.
 The Perceptron
A network consisting of one layer of M&P neurons connected in a
feedforward way (i.e. no lateral or feedback connections).

• Discrete output (+1 / -1)

• Capable of “learning” from examples (Rosenblatt)
• They suffer from serious computational limitations
The Perceptron Learning Algorithm
The Perceptron Learning Algorithm
 Decision Regions
It’s an area wherein all examples of one class fall.

Examples:
 Linear Separability
A classification problem is said to be linearly separable if the decision regions
can be separated by a hyperplane.

Example: AND
X Y X AND Y
0 0 0
0 1 0
1 0 0
1 1 1
 Limitations of Perceptrons
It has been shown that perceptrons can only solve linearly separable
problems.

Example: XOR (exclusive OR)

X Y X XOR Y
0 0 0
0 1 1
1 0 1
1 1 0
 The Perceptron Convergence Theorem

Theorem (Rosenblatt, 1960)

If the training set is linearly separable, the perceptron learning algorithm

always converges to a consistent hypothesis after a finite number of
epochs, for any η > 0.

If the training set is not linearly separable, after a certain number of

epochs the weights start oscillating.
A View of the Role of Units
 Multi–Layer Feedforward Networks
• Limitation of simple perceptron: can implement only linearly separable
functions

• Add “ hidden” layers between the input and output layer. A network
with just one hidden layer can represent any Boolean functions including
XOR

• Power of multilayer networks was known long ago, but algorithms for
training or learning, e.g. back-propagation method, became available
only recently (invented several times, popularized in 1986)

• Universal approximation power: Two-layer network can approximate

any smooth function (Cybenko, 1989; Funahashi, 1989; Hornik, et al..,
1989)

• Static (no feedback)

 Continuous-Valued Units

Sigmoid (or logistic)

 Continuous-Valued Units

Hyperbolic tangent
 Back-propagation Learning Algorithm
• An algorithm for learning the weights in a feed-forward network,
given a training set of input-output pairs
• The algorithm is based on gradient descent method.
 Supervised Learning
Supervised learning algorithms require the presence of a “teacher” who
provides the right answers to the input questions.

Technically, this means that we need a training set of the form

L= { (x , y ) ,
1 1
..... (x , y ) }
p p

where :

xm ( m =1… p) is the network input vector

ym ( m =1… p) is the desired network output

vector
 Supervised Learning

The learning (or training) phase consists of determining a configuration of

weights in such a way that the network output be as close as possible to the
desired output, for all the examples in the training set.

Formally, this amounts to minimizing an error function such as (not only

possible one):

E = åå (y )
1 m m 2
k - Ok
2 m k

where Okμ is the output provided by the output unit k when the network is
given example μ as input.
 Back-Propagation
To minimize the error function E we can use the classic gradient-
descent algorithm:

η = “learning rate”

To compute the partial derivates we use the error back propagation

algorithm.

It consists of two stages:

Forward pass : the input to the network is propagated

layer after layer in forward direction

Backward pass : the “error” made by the network is

propagated backward, and weights
are updated properly
 Notations

Given pattern µ, hidden unit j receives a net input

h j   w jk xk
k
and produces as output :

 
V j  g h j  g   w jk xk 
k 
 Back-Prop:
Updating Hidden-to-Output Weights

åå ( )
1 2
E= ykm - Okm
2 m k

¶E
DWij = - h
¶Wij
¶ é1 ù
ê åå (y )
2
= -h m
k - Ok m
ú
¶Wij êë 2 m k úû
¶Okm
= h å å ( yk - Ok m m
)
m k ¶Wij
¶Oim
= h å ( yi - Oi m m
)
m ¶Wij
= h å ( yim - Oim ) g '( h ) V m
i
m
j
m

= h å dim Vjm where :  i   yi  Oi  g '  hi 

m
 Back-Prop:
Updating Input-to-Hidden Weights (1)

E = åå (y )
1 m m 2

E k - Ok
w jk   2 m k
w jk
Oi

    yi  Oi
 i
 
 w jk
hi hi Vl 
 i

    yi  Oi  
 g' h  
i
w jk w jk
  Wil
l w jk
V j
 Wij
w jk

 Wij
 
g h j
w jk
h j
 Wij g' h j   w


jk
 Back-Prop:
Updating Input-to-Hidden Weights (2)
h j  
  w jm xm
w jk w jk m
 xk

Hence, we get:

w jk    y   O   g ' h  W
i i i ij  
g ' h j xk
,i

   W i ij  
g ' h j xk
,i

 

ˆ  j xk

where : ˆ j  g ' h j    i Wij

i
Error Back-Propagation
Locality of Back-Prop
The Back-Propagation Algorithm
The Back-Propagation Algorithm
The Role of the Learning Rate
 The Momentum Term
Gradient descent may:

• Converge too slowly if η is too small

• Oscillate if η is too large

Simple remedy:

The momentum term allows us to use large values of η thereby avoiding

oscillatory phenomena

Typical choice: α = 0.9, η = 0.5

The Momentum Term
 The Problem of Local Minima

Back-prop cannot avoid local minima.

Choice of initial weights is important.

If they are too large the nonlinearities

tend to saturate since the beginning of
the learning process.
NETtalk
NETtalk
 NETtalk

• A network to pronounce English text

• 7 x 29 (=203) input units

• 1 hidden layer with 80 units

• 26 output units encoding phonemes

• Trained by 1024 words in context

• Produce intelligible speech after 10 training epochs

• Functionally equivalent to DEC-talk

• Rule-based DEC-talk was the result of a decade effort by many linguists

• NETtalk learns from examples and, require no linguistic knowledge

 Theoretical / Practical Questions

▪ How many layers are needed for a given task?

▪ How many units per layer?

▪ To what extent does representation matter?

▪ What do we mean by generalization?

▪ What can we expect a network to generalize?

• Generalization: performance of the network on data not
included in the training set
• Size of the training set: how large a training set should be for
“good” generalization?
• Size of the network: too many weights in a network result in
poor generalization
 True vs Sample Error

The true error is unknown (and will remain so forever…).

On which sample should I compute the sample error?
Training vs Test Set
 Cross-validation

Leave-one-out: using as many test folds as there are examples (size of test fold = 1)
 Overfitting

(a) A good fit to noisy data.(b) Overfitting of the same data: the fit is perfect on the
“training set” (x’s), but is likely to be poor on “test set” represented by the circle.
Early Stopping
 Size Matters

• The size (i.e. the number of hidden units) of an artificial

neural network affects both its functional capabilities and
its generalization performance

• Small networks could not be able to realize the desired

input / output mapping

• Large networks lead to poor generalization performance

 The Pruning Approach
Train an over-dimensioned net and then remove redundant nodes
and / or connections:

• Sietsma & Dow (1988, 1991)

• Mozer & Smolensky (1989)
• Burkitt (1991)

Adavantages:

• arbitrarily complex decision regions

• faster training
• independence of the training algorithm
 An Iterative Pruning Algorithm
Consider (for simplicity) a net with one hidden layer:

Suppose that unit h is to be removed:

IDEA: Remove unit h (and its in/out connections) and adjust the
remaining weights so that the I/O behavior is the same

G. Castellano, A. M. Fanelli, and M. Pelillo, An iterative pruning algorithm for feedforward neural networks, IEEE
Transactions on Neural Networks 8(3):519-531, 1997.
 An Iterative Pruning Algorithm
This is equivalent to solving the system:

nh nh

å ij j å( ij ij ) j
w y (m )
= w + d y (m )
i =1… nO , m =1… P
j=1 j=1
j¹h

before after

which is equivalent to the following linear system (in the unknown δ’s):


jh
ij y ( )
j  wih y ( )
h i =1… nO , m =1… P
 An Iterative Pruning Algorithm

In a more compact notation:

Ax = b
Pno ´ no ( nh -1)
where AÎÂ

But solution does not always exists.

Least-square solution :

min Ax - b
x
 Detecting Excessive Units

• Residual-reducing methods for LLSPs start with an initial solution

x0 and produces a sequences of points {xk} so that the residuals

Axk  b  rk
decrease: rk  rk 1

• Starting point: x0  0  r0  b 

• Excessive units can be detected so that b is minimum

 The Pruning Algorithm

1) Start with an over-sized trained network

2) Repeat

2.1) find the hidden unit h for which b is minimum

2.2) solve the corresponding system
2.3) remove unit h

Until Perf(pruned) – Perf(original) < epsilon

3) Reject the last reduced network

 Example: 4-bit parity
Ten initial 4-10-1 networks

nine 4-5-1
Pruned nets 5 hidden nodes (average)
one 4-4-1

100 0,25

90

80 0,2

70
recognition rate (%)

recognition rate
60 MSE 0,15

MSE
50 MINIMUM
NET
40 0,1

30

20 0,05

10

0 0
10 9 8 7 6 5 4 3 2 1
number of hidden units
 Example: 4-bit simmetry

Ten initial 4-10-1 networks

Pruned nets 4.6 hidden nodes (average)

100 0,25

90

80 0,2

70
MINIMUM
recognition rate (%)

recognition rate NET

60 MSE 0,15

MSE
50

40 0,1

30

20 0,05

10

0 0
10 9 8 7 6 5 4 3 2 1
number of hidden units
Deep Neural Networks
The Age of “Deep Learning”
 The Deep Learning “Philosophy”

• Learn a feature hierarchy all the way from pixels to classifier

• Each layer extracts features from the output of previous layer

• Train all layers jointly

 Shallow vs Deep Networks

Shallow architectures are inefficient at representing deep functions

From. R. E. Turner
Performance Improves with More Data
 Old Idea… Why Now?

1. We have more data - from Lena to ImageNet.

1. We have more computing power, GPUs are

really good at this.

1. Last but not least, we have new ideas

 Image Classification

Predict a single label (or a distribution over labels as shown here to indicate our confidence)
for a given image. Images are 3-dimensional arrays of integers from 0 to 255, of size Width x
Height x 3. The 3 represents the three color channels Red, Green, Blue.

From: A. Karpathy
Challenges

From: A. Karpathy
 The Data-Driven Approach

An example training set for four visual categories.

In practice we may have thousands of categories and

hundreds of thousands of images for each category. From: A. Karpathy
 Inspiration from Biology
Biological vision is hierachically organized

From. R. E. Turner
Hierarchy of Visual Areas

From. D. Zoccolan
The Retina
The Retina
 Receptive Fields

“The region of the visual field in which light stimuli evoke responses of
a given neuron.”
 Cellular Recordings
Kuffler, Hubel, Wiesel, …

1953: Discharge patterns

and functional
organization of
mammalian retina

1959: Receptive fields of

single neurones in the
cat's striate cortex

1962: Receptive fields,

binocular interaction and
functional architecture in
the cat's visual cortex

1968 ..
Retinal Ganglion Cell Response
Beyond the Retina
 Simple Cells

Orientation selectivity: Most V1 neurons are orientation selective meaning that they
respond strongly to lines, bars, or edges of a particular orientation (e.g., vertical) but
not to the orthogonal orientation (e.g., horizontal).
Complex Cells
Hypercomplex Cells (end-stopping)
 Take-Home Message:
Visual System as a Hierarchy of Feature Detectors
Convolution
Convolution
Mean Filters
Gaussian Filters
Gaussian Filters
The Effect of Gaussian Filters
The Effect of Gaussian Filters
Kernel Width Affects Scale
Edge detection
Edge detection
Using Convolution for Edge Detection
 A Variety of Image Filters

Laplacian of Gaussians (LoG) (Marr 1982)

 A Variety of Image Filters

Gabor filters (directional) (Daugman 1985)

A Variety of Image Filters

From: M. Sebag
Traditional vs Deep Learning Approach

From: M. Sebag
Convolutional Neural Networks (CNNs)

(LeCun 1998)

(Krizhevsky et al. 2012)

 Fully- vs Locally-Connected Networks
Fully-connected: 400,000 hidden units = 16 billion parameters
Locally-connected: 400,000 hidden units 10 x 10 fields = 40 million parameters
Local connections capture local dependencies

From. M. A. Ranzato
 Using Several Trainable Filters

Normally, several filters are packed together and learnt automatically

during training
 Pooling
Max pooling is a way to simplify the network architecture, by
downsampling the number of neurons resulting from filtering operations.
Combining Feature Extraction and Classification
 AlexNet (2012)

• 8 layers total

• Trained on Imagenet Dataset (1000

categories, 1.2M training images, 150k
test images)
 AlexNet Architecture
• 1st layer: 96 kernels (11 x 11 x 3)
• Normalized, pooled
• 2nd layer: 256 kernels (5 x 5 x 48)
• Normalized, pooled 650,000 neurons
• 3rd layer: 384 kernels (3 x 3 x 256) 60 million parameters
• 4th layer: 384 kernels (3 x 3 x 192)
• 5th layer: 256 kernels (3 x 3 x 192)
• Followed by 2 fully connected layers, 4096 neurons each
• Followed by a 1000-way SoftMax layer
Training on Multiple GPU’s
 Rectified Linear Units (ReLU’s)
Problem: Sigmoid activation takes on values in (0,1). Propagating the
gradient back to the initial layers, it tends to become 0 (vanishing
gradient problem).

From a practical perspective, this slows down the training procedure of

the initial layers of the network.
Rectified Linear Units (ReLU’s)
 Mini-batch Stochastic Gradient Descent

Loop:
1. Sample a batch of data
2. Forward prop it through the graph, get loss
3. Backprop to calculate the gradients
4. Update the parameters using the gradient
 Data Augmentation

The easiest and most common method to reduce overfitting on

image data is to artificially enlarge the dataset using label-preserving
transformations

AlexNet uses two forms of this data augmentation.

• The first form consists of generating image translations and

horizontal reflections.

• The second form consists of altering the intensities of the RGB

channels in training images.
 Dropout
Set to zero the output of each hidden neuron with probability 0.5.

The neurons which are “dropped out” in this way do not contribute to
the forward pass and do not participate in backpropagation.

So every time an input is presented, the neural network samples a

different architecture, but all these architectures share weights.

Reduces complex co-

adaptations of neurons,
since a neuron cannot
rely on the presence of
particular other neurons.
ImageNet
ImageNet Challenges
ImageNet Challenge 2012
Revolution of Depth
A Hierarchy of Features

From: B. Biggio
 Layer 1

Each 3x3 block shows

the top 9 patches for
one filter
Layer 2
Layer 3
Layer 3
 Feature Analysis

• A well-trained ConvNet is an excellent feature extractor.

• Chop the network at desired layer and use the output as a feature
representation to train an SVM on some other dataset (Zeiler-Fergus 2013):

• Improve further by taking a pre-trained ConvNet and re-training it on a

different dataset (Fine tuning).
 Other Success Stories of Deep Learning

Today deep learning, in its several manifestations, is being applied

in a variety of different domains besides computer vision, such as:

• Speech recognition

• Optical character recognition

• Natural language processing

• Autonomous driving

• Game playing (e.g., Google’s AlphaGo)

• …
 References

• http://neuralnetworksanddeeplearning.com

• http://deeplearning.stanford.edu/tutorial/

• http://www.deeplearningbook.org/

• http://deeplearning.net/

Platforms:
• Theano
• Torch
• TensorFlow
• …