Inverse Problems in Image Processing
Erosyni Kokiopoulou,
Martin Ple
singer
{effrosyni.kokiopoulou; martin.plesinger}@sam.math.ethz.ch
Welcome to the world of inverse problems!
Outline
Seminar organization
Manipulating images in MATLAB
Motivation & A brief introduction to image deblurring
Reading assignments
Seminar organization
General info
Time:
Building:
Room:
First meeting:
Topic selection:
First talk:
Last talk:
Prerequisities:
Oce hours:
Thursday 15:0017:00
ML
ML H 34.3
September 23, 2010
September 30, 2010
October 21, 2010
December 16, 2010
Numerical Linear Algebra
Matrix Computations
MATLAB
Monday 14:0015:00
What do I have to do?
Regularly attend the seminar.
Give a successful talk about the given topic in English.
Please, meet us for consultation before the talk,
1st approximately 2 weeks before,
2nd approximately 3 days before.
Send us the presentation slides after the talk.
Write a short report (summary) about the topic (3 pages).
Optionally
MATLAB task: play with simple 1D problems as well as
image deblurring.
Implement in MATLAB some experiments from the reading material.
Send us the slides and report via e-mail.
Organization
2 presentations per week = 9 weeks
Timetable (Tentative)
1
2
3
4
5
6
7
8
9
week
week
week
week
week
week
week
week
week
October 21
the rst two talks
October 28
November 04
November 11 an important date?
November 18
November 25
December 02
December 09
December 16 the last two talks
(recommendation only)
bachelor st.
bachelor st.
bachelor st.
BSc/MSc st.
master st.
master st.
master st.
master st.
master st.
Books
Hansen, Nagy, OLeary: Deblurring Images, Spectra, Matrices, and
Filtering, SIAM, FA03, 2006 ... recommended for bachelor students
Hansen: Discrete Inverse Problems, Insight and Algorithms, SIAM,
FA07, 2010
... recommended for master students
Books availability
Legal possibilities:
Amazon.com
ETHZ library
Privat
($130 together, for seriously interested only ;))
(the second book only)
Other possibilities:
Electronic
Copy
(pre-version of the rst book only)
MATLAB code
We recommend:
Regularization Tools
http://www2.imm.dtu.dk/~pch/Regutools
by P. C. Hansen,
HNO package
http://www2.imm.dtu.dk/~pch/HNO
by P. C. Hansen, J. Nagy, D. OLeary.
Google Per Christian Hansen > go to Stu to Download > open
Regularization Tools and Debluring Images .
Manipulating images in MATLAB
What is an image?
An image is a vector (matrix or tensor) from a real vector space
X=
Rmnd,
where m, n are numbers of rows and columns of the image, i.e. the
height and width in pixels, resp., d is the dimension of a color space.
image color scheme
grayscale
RGB
color space
[0, 1]3 or [0, 255]
[0, 1]3 or [0, 255]3
dimension
d=1
d=3
Image Basics
Images can be color, grayscale or binary.
Grayscale intensity image: 2D array where each entry
contains the intensity value of the corresponding pixel.
There are many types of image le formats:
GIF (Graphics Interchange Format)
JPEG (Joint Photographic Experts Group)
PNG (Portable Network Graphics)
TIFF (Tagged Image File Format)
Images can be also stored using the MAT-le format.
Reading, Displaying and Writing Images
The command imfinfo displays information about the images stored
in the data le:
info = imfinfo(cameraman.tif);
The command imread loads an image in MATLAB:
I = imread(cameraman.tif);
Images can be displayed by three commands:
imshow,
imagesc,
and
image.
Display of Images
50
50
100
100
150
150
200
200
250
250
50
imshow
100
150
200
250
50
imagesc
100
150
200
250
200
250
image
50
50
100
100
150
150
200
200
250
250
50
100
150
200
250
imagesc, colormap(gray)
50
100
150
image, colormap(gray)
Display of Images (contd)
imshow renders images more accurately in terms of size and color
image and imagesc display images with a false colormap
image does not provide a proper scaling of the pixel values
imagesc should be combined with axis image
Writing of Images
The imwrite command writes an image to a le:
imwrite(I, image.jpg);
Arithmetic on Images
Integer representation of images can be limiting.
Common practice: convert the image to double precision,
process it, and convert it back to the original format.
The function double does the conversion: Id = double(I);
When the input image has double entries, imshow expects values
in [0,1].
Use imshow(Id,[]) to avoid unexpected results!
The function rgb2gray can be used to convert color images
to grayscale intensity images.
Summary: A very short guide to manipulating images in MATLAB
X = imread(file.bmp);
X = double(X);
% opens an image in MATLAB
% converts X to doubles
% X is a standard matrix (2D array) in grayscale case
% X is a 3D array in color case, three 2D arrays (for R, G, and B)
colormap gray;
imshow(X);
imagesc(X);
% changes the color scheme in MATLAB
% plots the (real) matrix X as an image
% ditto
% the first function is better, but available only with the
% Image Processing Toolbox.
% Convention: min(min(X)) is black, max(max(X)) is white.
Further information
MATLAB (online) help,
Image Processing Toolbox help,
Chapter 2 of the rst book [H., N., OL., Deblurring Images],
Regularization Tools Manual (available online),
consultations.
Motivation
A gentle start
[Kjller, Master Thesis, DTU Lyngby, 2007]
Another examples
Computer tomography (CT) maps a 3D object to X-ray pictures,
A()
: RM N K
Rmn
j=1
is a mapping (not opeartor) of M N K voxels, to -times mn pixels.
Boundary problem, from real analysis we know that it is (under some
assumptions) uniquely solvable. In numerical computations (rounding
errors, other sources of noise, e.g. electronic noise on semiconductors
PN-junctions in transistors) it is dicult to solve (Medicine).
We lose some important information.
Transmision tomography in crystalographics is used, e.g., for reconstruction of orientation-distribution-function (ODF) of grains in a
polycrystalline material,
the observation (right-hand side) is a vector of difractograms (analogous to the previous example).
Seismic tomography in geology is used for reconstruction of boundaries or cracks in tectonic plates
in localities with high tectonic activity and frequent
earthquakes. A model example of situation in San
Francisco craton [Hansen, AIRtools, DTU Lyngby]:
Reading bar codes:
A brief introduction to image
deblurring
Linear operators on a vector space Rmnd
Consider for simplicity the grayscale color scheme, thus d = 1.
A simple linear operator
A(X) = B,
X, B Rmn
(real matrices)
is, e.g., a rotation, a reection, a shift operator, etc (all of them are
easily invertible, in principle), but there are ugly operators, e.g.,
the blurring operator.
The blurring operator is linear and in purely mathematical sense still
invertible, but there are fundamental diculties with the real practical
computations of this inverse problem.
Naive solution of an equation with the blurring operator
Since the operator is linear we can rewrite the problem as a system
of linear algebraic equations
Ax = b,
A Rnmnm,
x, b Rnm
and solve it: True x and blured b image, and naive solution A1b:
x = true image
b = blurred, noisy image
x = inverse solution
,
,
(example by James Nagy, Emory University, Atlanta).
Question:
What did it happen ???
Blurring operator Gauian blur
The simplest case is the Gauian blurring operator (used in the previous example).
First recall the Gau function in 1D and 2D:
1.5
f1D(k) = ek
2
2
f2D(h, w) = e(h +w )
2
1.5
1
1
0.5
0.5
0
4
4
2
2
0
0
0
3
2
4
Note that the 2D Gau function is separable, i.e. can be written as a product of
two functions of one variable f2D (h, w) = e(h +w ) = eh ew = f1D (h)f1D (w).
2
PointSpreadFunction
The point-spread-function (PSF) is a characteristic of a (linear) blurring operator and illustrates how the opearator acts on a single pixel
P SFA (h, w) R(2+1)(2+1),
where h, w is from height and width. Examples of PSFs:
horizontal
motion blur
vertical
motion blur
out-of-focus
blur
Gauian
blur
,
,
,
.
Gauian blur is given by the Gauisan PSF which is nothing else
than the 2D Gau function (black 0, white 1).
Relatioship between A and P SFA
The PSF construction from the operator A() = P SFA (w, h) is clear
from the previous slide.
The (grayscale) image can be represented as a 2D function X =
X(w, h) = xw,h (color of the pixel at position (w, h)), the action of A
can be obtained by the 2D convolution of X with the P SFA .
In (our) discrete and nite case, recall that
X Rmn,
P SFA R(2+1)(2+1)
are matrices, it is
A(X) =
X(w , h )P SFA ( + 1 + , + 1 + ).
= =
Problem:
Entries xw,h are not dened for w = + 1, . . . , 0, and
m + 1, . . . , m + and w = + 1, . . . , 0, and n + 1, . . . , n + !!!
Question:
Is it possible to construct the matrix A from the PSF ???
Boundary conditions
The action of the (inverse) operator is not dened close to boundary
of the image. There are several possibilities, typically:
zero boundary
periodic boundary
reexive boundary
which involve structure of the matrix in the linear system.
Boundary conditions 1D problem
PSF is a T
oplitz matrix
b1
b2
b3
b4
b5
p4
= p
p2
p3
p4
p5
p1
p2
p3
p4
p5
p1
p2
p3
p4
x
1
x2
p1 x3
AP SF x,
p2
x4
p3
x5
with the boundary condition
b1
b2
b3
b4
b5
p5 p4 p3 p2
p5 p4 p3
p5 p4
p5
p1
p2
p3
p4
p5
p1
p2 p1
p3 p2 p1
p4 p3 p2 p1
x .
Note: The row [p1, p2, p3, p4, p5] is, e.g., the discretized 1D Gau function f1D (k).
Boundary conditions matrix structure
Clearly,
zero boundary:
periodic boundary:
reexive boundary:
[0, 0|x1, x2, x3, x4, x5|0, 0]T ,
[x4, x5|x1, x2, x3, x4, x5|x1, x2]T ,
[x2, x1|x1, x2, x3, x4, x5|x5, x4]T .
The linear system b = A x, where A = AP SF + M , and for
zero boundary
M 0,
periodic boundary
p
1
p2 p1
p5 p4
p5
reexive boundary
p p5
4
p5
, M
p1
p1 p2
Back to the naive solution
There are fundamental diculties with solving the linear problem:
Action of the blurring operator A is realized by convolution with very
smooth 2D Gau function, the operator has a smoothing property.
The (example) right-hand side B is represented by smooth function.
While solving the linear system, i.e. evaluation A1(B), we invert a
smoothing operator, apply it on a smooth function, and we want to
obtain an image X which is typically discontiuous function. Recall
b = blurred, noisy image
B=
x = true image
X=
Inverse problems Troubles!
The problem is typically very sensitive to small perturbations (e.g.,
smooth B, smooth A, nonsmooth solution).
But B is always corrupted by rounding errors (noise). Thus we have
system of equations
Ax = b + eexact,
where
e Rmn
is unknown,
and we want to nd xtrue A1b. The naive solution ilustrates the
catastophical impact of noise
x = inverse solution
X naive A1(B exact + E) =
Instead of the solution we see the amplied noise only.
Condition number of A is very large, e.g. (A) 10100, i.e. A is
close to singular, we can easily lose some important information.
The eect of inverted noise
We have seen that
xnaive = A1b = A1bexact + A1e.
Using the SVD of A = U V T , U = [u1, . . . , uN ], U 1 = U T , V =
[v1, . . . , vN ], V 1 = V T , = diag(1, . . . , N ), 1 . . . N 0,
A
b=V
N uT b
i v,
U e=
i
i=1 i
T
N uT e
i v,
e=
i
i
i=1
N = mn.
uT
e
Consider the quantities i : when i N this quantity is divided by
i
i and the contribution of vi in xnaive gets magnied!
The singular vectors corresponding to smallest i represent high frequency information.
The eect of inverted noise SVD
0
10
10
10
20
10
30
10
|u*jb |
e
|uj b |
s
40
10
10
20
30
40
j
50
60
Dont panic!
We have a plan B!
Using clever methods (regularization, spectral ltering) survive the
problem
b = blurred, noisy image
B=
659 iterations
X approx =
Main idea of regularization methods: Filter out the components corresponding to the small i.
Spectral ltering
Instead of the naive solution use
N
N
Tb
(bexact + e)
u
uT
approx
i
i
x
=
(i)
vi =
(i)
vi
i
i
i=1
i=1
where (i) is a lter function
1
TSVD
(j) = 1 (for < )
i
0.5
(j) = 0 (for i > )
0
100
200
300
400
1
Tikhonov
2
2
(j) = ( + 2)1
0.5
0
100
200
300
400
Fredholm integral equation of the rst kind
Generic form (1D):
1
0
K(s, t)f (t)dt = g(s),
0 s 1.
Inverse problem: Given the kernel K and g, compute f
(typically g is smooth and f has discontinuities!)
Deconvolution problem: special case of the above
1
0
h(s t)f (t)dt = g(s),
0 s 1.
Singular value expansion (SVE) of K: Important analysis tool.
Reading assignments topics
See the list of topics.
