Digital Image Processing 2 Contents

In this lecture we will look at spatial filtering

techniques:
– Neighbourhood operations
Spatial Filtering – What is spatial filtering?
– Smoothing operations
– What happens at the edges?
Christophoros Nikou – Correlation and convolution
[email protected] – Sharpening filters
– Combining filtering techniques

University of Ioannina - Department of Computer Science C. Nikou – Digital Image Processing (E12)

3 Neighbourhood Operations 4 Simple Neighbourhood Operations

Neighbourhood operations simply operate Some simple neighbourhood operations
on a larger neighbourhood of pixels than include:
point operations Origin x – Min: Set the pixel value to the minimum in
Neighbourhoods are the neighbourhood
mostly a rectangle – Max: Set the pixel value to the maximum in
around a central pixel the neighbourhood
(x, y) – Median: The median value of a set of
Any size rectangle Neighbourhood
numbers is the midpoint value in that set (e.g.
and any shape filter from the set [1, 7, 15, 18, 24] 15 is the
are possible median). Sometimes the median works better
y Image f (x, y)
than the average
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

Simple Neighbourhood Operations

5 6 The Spatial Filtering Process
Example
Origin x
a b c r s t
Original Image
123 127 128 119 115 130
x Enhanced Image x d
g
e
h
f
i
* u
x
v
y
w
z
140 145 148 153 167 172 Original Image Filter
Simple 3*3 Pi l
Pixels
e 3*3 Filter
Neighbourhood
133 154 183 192 194 191
eprocessed = v*e +
194 199 207 210 198 195 r*a + s*b + t*c +
u*d + w*f +
164 170 175 162 173 151 y Image f (x, y) x*g + y*h + z*i
y y The above is repeated for every pixel in the
original image to generate the filtered image
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

1
 7 Spatial Filtering: Equation Form 8 Smoothing Spatial Filters
a b • One of the simplest spatial filtering
∑ ∑ w(s, t ) f ( x + s, y + t )
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

g ( x, y ) = operations we can perform is a smoothing

s = − at = − b
operation
Filtering can be given – Simply average all of the pixels in a
i equation
in ti fform as neighbourhood around a central value
shown above – Especially useful
1/ 1/ 1/
Notations are based in removing noise 9 9 9
from images Simple
on the image shown 1/ 1/ 1/
– Also useful for 9 9 9 averaging
to the left
highlighting gross filter
1/ 1/ 1/
detail 9 9 9
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

9 Smoothing Spatial Filtering 10 Image Smoothing Example

Origin x
104 100 108 1/
9
1/
9
1/
9 • The image at the top left
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

is an original image of
99 106 98

95 90 85
* 1/

1/
9
1/

1/
9
1/

1/
9
size 500*500 pixels
9 9 9

Simple 3*3
33
1/ 100
104
9
1/ 108
9
1/
9
3 3 Smoothing
3*3
Original Image Filter • The subsequent images
Pi l
Pixels
show the image after
/9 1106
199 /9 198
/9
Neighbourhood /9 190
195 /9 /9
185 Filter
e = 1/9*106 +
1/ *104 + 1/ *100 + 1/ *108 + filtering with an averaging
9 9 9
1/ *99 + 1/ *98 +
9 9
filter of increasing sizes
y Image f (x, y) 1/ *95 + 1/ *90 + 1/ *85
9 9 9 − 3, 5, 9, 15 and 35
= 98.3333
The above is repeated for every pixel in the original image to
• Notice how detail begins
generate the smoothed image. to disappear
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

11 Weighted Smoothing Filters 12 Another Smoothing Example

• More effective smoothing filters can be • By smoothing the original image we get rid
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

generated by allowing different pixels in of lots of the finer detail which leaves only
the neighbourhood different weights in the the gross features for thresholding
averaging function
1/ 2/ 1/
– Pixels closer to the 16 16 16
central pixel are more
2/ 4/ 2/
important 16 16 16

– Often referred to as a 1/ 2/ 1/
weighted averaging 16 16 16
Original Image Smoothed Image Thresholded Image
Weighted
averaging filter
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

2
 13
Averaging Filter Vs. Median Filter 14
Spatial smoothing and image
Example approximation
• Spatial smoothing may be viewed as a
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)
process for estimating the value of a pixel
from its neighbours.

Original Image Image After Image After • What is the value that “best” approximates
With Noise Averaging Filter Median Filter
the intensity of a given pixel given the
• Filtering is often used to remove noise intensities of its neighbours?
from images
• Sometimes a median filter works better • We have to define “best” by establishing a
than an averaging filter criterion.
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

15
Spatial smoothing and image 16
Spatial smoothing and image
approximation (cont...) approximation (cont...)
A standard criterion is the the sum of Another criterion is the the sum of absolute
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

squares differences. differences.

N N
⎧ N
⎫ ⎧N ⎫
E = ∑ [ x(i ) − m ] ⇔ m = arg min ⎨∑ [ x(i) − m] ⎬ E = ∑ x(i ) − m ⇔ m = arg min ⎨∑ x(i ) − m ⎬
2 2

i =1
m ⎩ i =1 ⎭ i =1
m ⎩ i =1 ⎭
⎧ 1 x>0
∂E N N N
∂E N
⎪
= 0 ⇔ −2∑ ( x(i ) − m ) = 0 ⇔ ∑ x(i ) =∑ m = 0 ⇔ −∑ sgn ( x(i ) − m ) = 0, sign( x) = ⎨ 0 x = 0
∂m i =1 i =1 i =1 ∂m i =1 ⎪−1 x < 0
⎩
N N
There must be equal in quantity positive and negative values.
⇔ ∑ x(i ) =Nm ⇔ m = 1 ∑ x (i ) The average value
i =1 N i =1 m = median{x(i )}
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

17
Spatial smoothing and image 18
Spatial smoothing and image
approximation (cont...) approximation (cont...)
– The median filter is non linear: Example x[n] 1 1 1 1 1 2 2 2 2 2
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

median{x + y} ≠ median{x} + median{ y} edge

– It works well for impulse noise (e.g. salt and Impulse
x[n] 1 3 1 1 1 2 3 2 2 3
pepper). noise
– It requires
i sorting
ti off the
th image
i values.
l
Median
– It preserves the edges better than an average (N=3) x[n] - 1 1 1 1 2 2 2 2 -
filter in the case of impulse noise.
– It is robust to impulse noise at 50%.
Average x[n] - 1.7 1.7 1 1.3 2 2.3 2.3 2.2 -
(N=3)
The edge is smoothed
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

3
 Strange Things Happen At The Edges!
19 Strange Things Happen At The Edges! 20
(cont…)
At the edges of an image we are missing There are a few approaches to dealing with
pixels to form a neighbourhood missing edge pixels:
Origin x – Omit missing pixels
e e
• Only works with some filters
• Can add extra code and slow down processing
– Pad the image
e
• Typically with either all white or all black pixels
e
– Replicate border pixels
– Truncate the image
e e e
– Allow pixels wrap around the image
y Image f (x, y) • Can cause some strange image artefacts
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

Strange Things Happen At The Edges!

21 22 Correlation & Convolution
(cont…)
• The filtering we have been talking about so far is
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Filtered Image:
referred to as correlation with the filter itself
Zero Padding referred to as the correlation kernel
• Convolution is a similar operation, with just one
subtle difference
Original Filtered Image:
Image Replicate Edge Pixels
a b c r s t eprocessed = v*e +
z*a + y*b + x*c +
d
f
e
g h
e
* u
x
v
y
w
z
w*d + u*e +
Filtered Image: t*f + s*g + r*h
Wrap Around Edge Pixels Original Image Filter
Pixels

C. Nikou – Digital Image Processing (E12)

• For symmetricC. Nikou
filters it makes no difference.
– Digital Image Processing (E12)

23 Correlation & Convolution (cont.) 24 Correlation & Convolution (cont.)

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

4
25
Effect of Low Pass Filtering on White 26
Effect of Low Pass Filtering on White
Noise Noise (cont...)
Let f be an observed instance of the image f0 Applying a low pass filter h (e.g. an average
corrupted by noise w: filter) by convolution to the degraded image:

f = f0 + w g = h * f = h *( f 0 + w) = h * f 0 + h * w
with noise samples having mean value The expected value of the output is:
E[w(n)]=0 and being uncorrelated with respect
to location:
E[ g ] = E[h * f 0 ] + E[h * w] = h * f 0 + h * E[ w]
⎧σ 2 , m = n = h * f 0 + h *0 = h * f 0
E[ w(m) w(n)] = ⎨
⎩ 0, m ≠ n The noise is removed in average.
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

27
Effect of Low Pass Filtering on White 28
Effect of Low Pass Filtering on White
Noise (cont...) Noise (cont...)
What happens to the standard deviation of g? Considering that h is an average filter, we have
Let g = h * f 0 + h * w = f 0 + w at pixel n:
1
where the bar represents filtered versions of
the signals, then
w(n) = (h * w)(n) = ∑ w(k )
N k∈Γ ( n )
σ g2 = E[ g 2 ] − ( E[ g ]) = Ε[( f 0 + w) 2 ] − ( f 0 )2
2 Therefore,

⎡⎛ 1 ⎞
2
⎤
= Ε[( f 0 ) 2 + ( w) 2 + 2 f 0 w] − ( f 0 ) 2 E[( w(n)) 2 ] = E ⎢⎜
⎢⎣⎝ N
∑ w(k ) ⎟ ⎥
⎥⎦
k∈Γ ( n ) ⎠
= E[( w) 2 ] + 2 E[ f 0 ]E[ w] = E[( w) 2 ]
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

29
Effect of Low Pass Filtering on White 30
Effect of Low Pass Filtering on White
Noise (cont...) Noise (cont...)
⎡⎛ 1 ⎞
2
⎤ Sum of squares
E ⎢⎜ ∑ w(k ) ⎟ ⎥ 1 1
⎢⎣⎝ N ⎠ ⎥⎦ ∑ E ⎡{w(k )} ⎤ = 2 ∑σ
2 2
k∈Γ ( n )
N2 k ∈Γ ( n )
⎣ ⎦ N k ∈Γ ( n )

1
= ∑ E ⎡{w(k )} ⎤
2
⎣ ⎦ Sum of squares Cross products (uncorrelated as m∫l)
N2 k∈Γ ( n )
2
2
+
N2
∑ ∑ E [ w(n − l )w(n − m)] = 0
+ ∑ ∑ E [ w(n − l ) w(n − m) ] l∈Γ ( n ) m∈Γ ( n )
m ≠l
N2 l∈Γ ( n ) m∈Γ ( n )
m ≠l
Cross products
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

5
 Effect of Low Pass Filtering on White
31 32 Sharpening Spatial Filters
Noise (cont...)
Finally, substituting the partial results: • Previously we have looked at smoothing
filters which remove fine detail
⎡⎛ 1 ⎞
2
⎤ 1 • Sharpening spatial filters seek to highlight
σ = E ⎢⎜
2
g
⎢⎣⎝ N
∑ w(k ) ⎟ ⎥= 2
⎥⎦ N
∑σ 2
fine detail
k ∈Γ ( n ) ⎠ k∈Γ
Γ(n)
– Remove blurring from images
1 σ 2 – Highlight edges
= Nσ 2 = • Sharpening filters are based on spatial
N 2
N
differentiation
The effect of the noise is reduced.
This processing is not optimal as it also smoothes image
edges.
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

33 Spatial Differentiation 34 Spatial Differentiation

• Differentiation measures the rate of change
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

of a function
• Let’s consider a simple 1 dimensional A B

example

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

35 Derivative Filters Requirements 36 1st Derivative

• First derivative filter output • Discrete approximation of the 1st derivative
– Zero at constant intensities
– Non zero at the onset of a step or ramp ∂f
= f ( x + 1) − f ( x)
– Non zero along ramps ∂x
• It is just the difference between subsequent
• Second derivative filter output values and measures the rate of change of
– Zero at constant intensities the function
– Non zero at the onset and end of a step or ramp
– Zero along ramps of constant slope

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

6
37 1st Derivative (cont…) 38 1st Derivative (cont.)
8 • The gradient of an image:
7
6
5
4
3
2
1 •
0

5 5 4 3 2 1 0 0 0 6 0 0 0 0 1 3 1 0 0 0 0 7 7 7 7
The gradient points in the direction of most rapid increase
in intensity.
0 -1 -1 -1 -1 -1 0 0 6 -6 0 0 0 1 2 -2 -1 0 0 0 7 0 0 0
8 Gradient direction
6
4
2
0 The edge strength is given by the gradient magnitude
-2
-4
-6
-8

Source: Steve Seitz

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

39 1st Derivative (cont.) 40 2nd Derivative

• Discrete approximation of the 2nd
derivative:
∇f

∂2 f
= f ( x + 1) + f ( x − 1) − 2 f ( x)
∂2x

∂f ∂f
∂x ∂y

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

Using Second Derivatives For Image

41 2nd Derivative (cont…) 42
Enhancement
8
7 • Edges in images are often ramp-like
6
5 transitions
4
3 – 1st derivative is constant and produces thick
2
1 edges
– 2nd derivative zero crosses the edge (double
0

5 5 4 3 2 1 0 0 0 6 0 0 0 0 1 3 1 0 0 0 0 7 7 7 7 response at the onset and end with opposite

-1 0 0 0 0 1 0 6 -12 6 0 0 1 1 -4 1 1 0 0 7 -7 0 0 signs)
10

-5

-10

-15

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

7
 Using Second Derivatives For Image
43 Derivatives 44
Enhancement
• A common sharpening filter is the Laplacian
– Isotropic
• Rotation invariant: Rotating the image and applying
the filter is the same as applying the filter and then
rotating the image.
• In other words, the Laplacian of a rotated image is the
rotated Laplacian of the original image.
– One of the simplest sharpening filters
– We will look at a digital implementation
∂2 f ∂2 f
∇2 f = +
∂2 x ∂2 y
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

45 The Laplacian 46 The Laplacian (cont…)

∂2 f ∂2 f ∇ 2 f = −4 f ( x , y )
∇ f = 2 + 2
2

∂ x ∂ y + f ( x + 1, y ) + f ( x − 1, y )
∂2 f + f ( x, y + 1)) + f ( x, y − 1))
= f ( x + 1, y ) + f ( x − 1, y ) − 2 f ( x, y )
∂2 x
0 1 0
∂2 f
= f ( x, y + 1) + f ( x, y − 1) − 2 f ( x, y ) 1 -4 1
∂2 y
0 1 0

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

47 The Laplacian (cont…) 48 The Laplacian (cont…)

• Applying the Laplacian to an image we get • The result of a Laplacian
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

a new image that highlights edges and filtering is not an enhanced

other discontinuities image
• We have to do more work
• Subtract the Laplacian result
Laplacian
from the original image to Filtered Image
generate our final sharpened Scaled for Display

enhanced image
Original Laplacian Laplacian
Image Filtered Image Filtered Image
Scaled for Display
g ( x, y ) = f ( x, y ) − ∇ 2 f
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

8
 49 Laplacian Image Enhancement 50 Laplacian Image Enhancement
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)
- =
Original Laplacian Sharpened
Image Filtered Image Image

• In the final, sharpened image, edges and

fine detail are much more obvious

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

51 Simplified Image Enhancement 52 Simplified Image Enhancement (cont…)

• The entire enhancement can be combined • This gives us a new filter which does the
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

into a single filtering operation: whole job in one step

g ( x, y ) = f ( x, y ) − ∇ 2 f
= 5 f ( x, y ) − f ( x + 11, y ) − f ( x − 11, y ) 0 -1
1 0

-1 5 -1
− f ( x, y + 1) − f ( x, y − 1)
0 -1 0

C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

53 Simplified Image Enhancement (cont…) 54 Variants On The Simple Laplacian

• There are lots of slightly different versions
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

of the Laplacian that can be used:

0 1 0 1 1 1
Standard Variant of
1 -4
4 1 1 -8
8 1
L l i
Laplacian L l i
Laplacian
0 1 0 1 1 1

-1 -1 -1

-1 9 -1

-1 -1 -1
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

9
 55 Unsharp masking 56 Unsharp masking (cont...)
• Used by the printing industry
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)
Sharpening mechanism
• Subtracts an unsharped (smooth) image
from the original image f (x,y).
–Blur the image
g
b(x,y)=Blur{f (x,y)} If k>1, the process is referred to
–Subtract the blurred image from the original as highboost filtering
(the result is called the mask)
gmask(x,y)=f (x,y)-b(x,y)
–Add the mask to the original
g(x,y)=f (x,y)+k gmask(x,y), k being non negative
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

Using First Derivatives For Image

57 Unsharp masking (cont...) 58
Enhancement
T
Original image ⎡ ∂f ∂f ⎤
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

T
∇f = ⎡⎣Gx G y ⎤⎦ = ⎢
⎣ ∂x ∂y ⎥⎦
Blurred image
(Gaussian 5x5, σ=3)
• Although
g the derivatives are linear
Mask operators, the gradient magnitude is not.
• Also, the partial derivatives are not rotation
Unsharp masking (k=1) invariant (isotropic).
• The magnitude of the gradient vector is
Highboost filtering (k=4.5) isotropic.
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

Using First Derivatives For Image

59 60 Sobel Operators
Enhancement (cont…)
• In some applications it is more • Sobel operators introduce the idea of
computationally efficient to approximate: smoothing by giving more importance to the
center point:
∇f ≈ Gx + G y
-1 -2 -1 -1 0 1
• This expression preserves relative changes
in intensity but it is not isotropic. 0 0 0 -2 0 2

• Isotropy is preserved only for a limited 1 2 1 -1 0 1

number of rotational increments which
depend on the filter masks (e.g. 90 deg.). • Note that the coefficients sum to 0 to give a 0
response at areas of constant intensity.
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

10
 61 Sobel operator Example 62 1st & 2nd Derivatives
Comparing the 1st and 2nd derivatives we
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

An image of a
contact lens which can conclude the following:
is enhanced in
order to make – 1st order derivatives generally produce thicker
defects (at four
and five o’clock in edges (if thresholded at ramp edges)
th image)
the i
obvious
) more
– 2nd order derivatives have a stronger
response to fine detail e.g. thin lines
• Sobel gradient aids to eliminate constant or slowly – 1st order derivatives have stronger response
varying shades of gray and assist automatic to gray level step
inspection. – 2nd order derivatives produce a double
• It also enhances small discontinuities in a flat gray response at step changes in grey level (which
filed. helps in detecting zero crossings)
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

63
Combining Spatial Enhancement 64
Combining Spatial Enhancement
Methods Methods (cont…)
• Successful image
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

enhancement is typically not

achieved using a single
operation
• Rather we combine a range of
techniques in order to achieve (a)
a final result
Laplacian filter of
• This example will focus on bone scan (a)
(b)
enhancing the bone scan to Sharpened version of
the right bone scan achieved (c)
by subtracting (a)
and (b) Sobel filter of bone
scan (a) (d)
C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

65
Combining Spatial Enhancement 66
Combining Spatial Enhancement
Methods (cont…) Methods (cont…)
Result of applying a (h)
power-law trans. to Compare the original and final images
Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Images taken from Gonzalez & Woods, Digital Image Processing (2002)

Sharpened image (g)

which is sum of (a)
and (f) (g)
The product of (c)
and (e) which will be (f)
used as a mask
(e)

Image (d) smoothed with

a 5*5 averaging filter C. Nikou – Digital Image Processing (E12) C. Nikou – Digital Image Processing (E12)

11
67 Summary
In this lecture we have looked at the idea of
spatial filtering and in particular:
– Neighbourhood operations
– The filtering process
– Smoothing filters
– Dealing with problems at image edges when
using filtering
– Correlation and convolution
– Sharpening filters
– Combining filtering techniques

C. Nikou – Digital Image Processing (E12)

12