The blue and green colors are actually the same
http://blogs.discovermagazine.com/badastronomy/2009/06/24/the-blue-and-the-green/
Hybrid Images
• A. Oliva, A. Torralba, P.G. Schyns,
“Hybrid Images,” SIGGRAPH 2006
Why do we get different, distance-dependent
interpretations of hybrid images?
Slide: Hoiem
Other filters
1 0 -1
2 0 -2
1 0 -1
Sobel
Vertical Edge
(absolute value)
Other filters
1 2 1
0 0 0
-1 -2 -1
Sobel
Horizontal Edge
(absolute value)
Filtering vs. Convolution
f=filter, size k x l
I=image, size m x n
• 2d filtering
h[m, n] f [k , l ] I [m k , n l ]
k ,l
• 2d convolution
h[m, n] f [k , l ] I [m k , n l ]
k ,l
In Python you can use https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html
Important filter: Gaussian
• Weight contributions of neighboring pixels by nearness
0.003 0.013 0.022 0.013 0.003
0.013 0.059 0.097 0.059 0.013
0.022 0.097 0.159 0.097 0.022
0.013 0.059 0.097 0.059 0.013
0.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Slide credit: Christopher Rasmussen
Smoothing with Gaussian filter
Smoothing with box filter
Gaussian filters
• Remove “high-frequency” components from the image (low-
pass filter)
– Images become more smooth
• Convolution with self is another Gaussian
– So can smooth with small-width kernel, repeat, and get same result
as larger-width kernel would have
– Convolving two times with Gaussian kernel of width σ is same as
convolving once with kernel of width σ√2
• Separable kernel
– Factors into product of two 1D Gaussians
Source: K. Grauman
Separability of the Gaussian filter
Source: D. Lowe
Separability example
2D convolution
(center location only)
The filter factors
into a product of 1D
filters:
Perform convolution =
along rows: *
Followed by convolution =
along the remaining column: *
Source: K. Grauman
Separability
• Why is separability useful in practice?
Some practical matters
Practical matters
How big should the filter be?
• Values at edges should be near zero
• Rule of thumb for Gaussian: set filter half-width to
about 3 σ
Practical matters
• What about near the edge?
– the filter window falls off the edge of the image
– need to extrapolate
– methods:
• clip filter (black)
• wrap around
• copy edge
• reflect across edge
Source: S. Marschner
Recap of Filtering
• Linear filtering is dot product at each
1 0 -1
position 2 0 -2
– Not a matrix multiplication 1 0 -1
– Can smooth, sharpen, translate (among
many other uses)
– Linear filters “look for” features that
resemble the filter itself
• Be aware of details for filter size,
extrapolation, cropping
Why do we care so much about
filtering/convolution?
• Pixels are individually weak and
noisy signals. Reasoning over
neighborhoods helps.
Surely there are other ways to
extract information from images?
• Yes, but they may be more
brittle, slower to compute, or
less easy to plug into machine
learning tools.
Alternative to Filtering - Viola Jones Face Detection
Viola, Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
Median filters
• A Median Filter operates over a window by
selecting the median intensity in the window.
• What advantage does a median filter have over
a mean filter?
• Is a median filter a kind of convolution?
© 2006 Steve Marschner • 23 Slide by Steve Seitz
Comparison: salt and pepper noise
© 2006 Steve Marschner • 24 Slide by Steve Seitz
Review: questions
1. Write down a 3x3 filter that returns a positive value if the
average value of the 4-adjacent neighbors is less than the
center and a negative value otherwise
2. Write down a filter that will compute the gradient in the x-
direction:
gradx(y,x) = im(y,x+1)-im(y,x) for each x, y
Slide: Hoiem
Thinking in Frequency
Slides: Hoiem, Efros, and others
This lecture
• Fourier transform and frequency domain
– Frequency view of filtering
• Reminder: Read your textbook
– Today’s lecture covers material in 3.4
Slide: Hoiem
Why does the Gaussian give a nice smooth
image, but the square filter give edgy artifacts?
Gaussian Box filter
Why does a lower resolution image still make
sense to us? What do we lose?
Image: http://www.flickr.com/photos/igorms/136916757/ Slide: Hoiem
Thinking in terms of frequency
Background: Change of Basis
Background: Change of Basis
For vectors and for image patches
Related concept: Image Compression
How is it that a 4MP image can be compressed
to a few hundred KB without a noticeable
change?
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
https://en.wikipedia.org/wiki/JPEG
Using DCT in JPEG
• The first coefficient B(0,0) is the DC component, the average
intensity
• The top-left coeffs represent low frequencies, the bottom right
– high frequencies
Lossy Image Compression (JPEG)
8x8 image patch Patch representation after
projecting on to DCT bases
DCT bases
Image compression using DCT
• Quantize
– More coarsely for high frequencies (which also tend to have smaller values)
– Many quantized high frequency values will be zero
• Encode
– Can decode with inverse dct
Filter responses
Quantization table
Quantized values
JPEG Compression Summary
1. Convert image to YCrCb
2. Subsample color by factor of 2
– People have bad spatial sensitivity for color
3. Split into blocks (8x8, typically), subtract 128
4. For each block
a. Compute DCT coefficients
b. Coarsely quantize
• Many high frequency components will become zero
c. Encode (e.g., with Huffman coding)
http://en.wikipedia.org/wiki/YCbCr
http://en.wikipedia.org/wiki/JPEG
Jean Baptiste Joseph Fourier (1768-1830)
...the manner in which the author arrives at these
had crazy idea (1807): equations is not exempt of difficulties and...his
Any univariate function can beanalysis to integrate them still leaves something to be
rewritten as a weighted sum of desired on the score of generality and even rigour.
sines and cosines of different
frequencies.
• Don’t believe it?
– Neither did Lagrange,
Laplace
Laplace, Poisson and
other big wigs
– Not translated into
English until 1878!
• But it’s (mostly) true!
– called Fourier Series Legendre
Lagrange
– there are some subtle
restrictions
How would math
have changed if the
Slanket or Snuggie
had been invented?
A sum of sines
Our building block:
Asin(x
Add enough of them to get
any signal g(x) you want!
Frequency Spectra
• example : g(t) = sin(2πf t) + (1/3)sin(2π(3f) t)
= +
Slides: Efros
Frequency Spectra
Frequency Spectra
= +
=
Frequency Spectra
= +
=
Frequency Spectra
= +
=
Frequency Spectra
= +
=
Frequency Spectra
= +
=
Frequency Spectra
1
= A sin(2 kt )
k 1 k
Example: Music
• We think of music in terms of frequencies at different
magnitudes
Slide: Hoiem
Other signals
• We can also think of all kinds of other signals the same way
xkcd.com
Fourier analysis in images
Intensity Image
Fourier Image
http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
