High-Quality Computational Imaging Through Simple Lenses

Felix Heide1 , Mushfiqur Rouf1 , Matthias B. Hullin1 , Björn Labitzke2 , Wolfgang Heidrich1 , Andreas Kolb2
1
University of British Columbia, 2 University of Siegen

Fig. 1. Our system reliably estimates point spread functions of a given optical system, enabling the capture of high-quality imagery through poorly performing
lenses. From left to right: Camera with our lens system containing only a single glass element (the plano-convex lens lying next to the camera in the left image),
unprocessed input image, deblurred result.

Modern imaging optics are highly complex systems consisting of up to two pound lens made from two glass types of different dispersion, i.e.,
dozen individual optical elements. This complexity is required in order to their refractive indices depend on the wavelength of light differ-
compensate for the geometric and chromatic aberrations of a single lens, ently. The result is a lens that is (in the first order) compensated
including geometric distortion, field curvature, wavelength-dependent blur, for chromatic aberration, but still suffers from the other artifacts
and color fringing. mentioned above.
In this paper, we propose a set of computational photography tech- Despite their better geometric imaging properties, modern lens
niques that remove these artifacts, and thus allow for post-capture cor- designs are not without disadvantages, including a significant im-
rection of images captured through uncompensated, simple optics which pact on the cost and weight of camera objectives, as well as in-
are lighter and significantly less expensive. Specifically, we estimate per- creased lens flare.
channel, spatially-varying point spread functions, and perform non-blind In this paper, we propose an alternative approach to high-quality
deconvolution with a novel cross-channel term that is designed to specifi- photography: instead of ever more complex optics, we propose
cally eliminate color fringing. to revisit much simpler optics used for hundreds of years (see,
e.g., [Rashed 1990]), while correcting for the ensuing aberra-
Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Pic-
tions computationally. While this idea is not entirely new (see,
ture/Image Generation—Digitizing and scanning
e.g., [Schuler et al. 2011]), our approach, which exploits cross-
Additional Key Words and Phrases: Computational Photography, Deconvo- channel information, is significantly more robust than other meth-
lution, Optimization, Optics ods that have been proposed. It is therefore able to handle much
larger and more dense blur kernels, such as disk-shaped kernels
with diameters of 50 pixels and more, which occur frequently in
uncorrected optics unless the apertures are stopped down to an im-
1. INTRODUCTION practical size. A key component to achieve this robustness is the
Over the past decades, camera optics have become increasingly development of a convex solver with guaranteed convergence prop-
complex. The lenses of modern single lens reflex (SLR) cameras erties that can minimize the resulting cross-channel deconvolution
may contain a dozen or more individual lens elements, which are problem. In fact, our experiments show that a failure of current
used to optimize light efficiency of the optical system while min- methods to arrive at a global optimum of the objective function
imizing aberrations, i.e., non-linear deviations from an idealized limits the quality of state-of-the-art deconvolution methods. The
thin lens model. specific technical contributions that enable the use of simple lens
Optical aberrations include effects such as geometric distortions, designs for high-quality photography are
chromatic aberration (wavelength-dependent focal length), spheri-
cal aberration (focal length depends on the distance from the op- —a new cross-channel prior for color images,
tical axis), and coma (angular dependence on focus) [Mahajan —a deconvolution method and convex solver that can efficiently
1991]. All single lens elements with spherical surfaces suffer from incorporate this prior and is guaranteed to converge to a global
these artifacts, and as a result cannot be used in high-resolution, optimum, and
high-quality photography. Instead, modern optical systems feature —a robust approach for per-channel spatially-varying PSF estima-
a combination of different lens elements with the intent of cancel- tion using a total variation (TV) prior based on the same opti-
ing out aberrations. For example, an achromatic doublet is a com- mization framework.
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
2 •

We demonstrate our method by producing high-quality pho- ment of the Richardson-Lucy algorithm. More recently, there has
tographs on modern 12 megapixel digital SLRs using only sin- been renewed interest in solving such problems with modern tech-
gle lens elements such as plano-convex or biconvex lenses, and niques, including deconvolution for spatially varying PSFs [Kee
achromatic doublets. We achieve high quality results compara- et al. 2011] and the removal of chromatic aberrations [Chung et al.
ble to conventional lenses, with apertures around f /4.5. Finally, 2009; Joshi et al. 2009], both for images taken with complex opti-
we show that our formulation outperforms competing deconvolu- cal systems. We note that in both cases the point spread functions
tion approaches in terms of both quality and computational per- are significantly smaller than with the simple lenses we use. Joshi
formance. Using our techniques, simplified camera optics become et al.’s color prior [2009] encourages color gradients that are lin-
viable alternatives to conventional camera objectives without the ear blends of two base colors. This is not a good prior for remov-
disadvantages outlined above. ing color fringing from chromatic aberration since large-area color
fringing is typically composed of several segments that are linear
mixtures of different base colors.
2. RELATED WORK
Kang [2007] proposes a method specifically for removing color
Only a few years after Gauss introduced the linear model of op- fringing in images. However, this method is based on edge detec-
tics [1841] that gives rise to the thin lens model, Seidel [1857] in- tion, which is feasible for images taken with partially corrected op-
troduced aberration theory, i.e., the study of non-linear effects in tics, but is impossible in the presence of very large blurs that result
optical systems. The common way to correct for these effects in op- from the use of uncorrected optics.
tical systems is to design increasingly complex systems with larger Cho et al. [2010; 2012] propose a deconvolution method that
and larger numbers of individual lens elements [Mahajan 1991]. An uses locally learned gradient statistics for deconvolution. Although
alternative way to deal with aberrations is to remove the resulting this has not been attempted in the literature, one could imagine us-
blur after the capture in a deconvolution step. ing this method to transfer gradient information from one channel
to another. Instead, our approach is to directly match point-wise
Image Deconvolution. gradients in the different channels, which provides better localiza-
The most basic deconvolution approaches include frequency-space tion and therefore suppression of color fringing than an approach
division and the Wiener filter [Wiener 1964], which however deal based on area statistics.
poorly with frequencies that are suppressed by the blur kernel. A Schuler et al. [2011] presented the work most similar to ours.
classical iterative method for image deconvolution is Richardson- They solve the demosaicing problem as well as the deconvolution
Lucy [Richardson 1972; Lucy 1974], which was developed for problem at the same time by working in Yuv color space. While
use in astronomy to compensate for blur in optical systems that working in this color space is a strong-enough prior to avoid color
are insufficiently corrected for aberrations. The basic Richardson- artifacts stemming from PSFs with relatively sparse features and
Lucy algorithm has been extended in many ways, including resid- thin structures, this approach is not sufficient to avoid artifacts in
ual deconvolution [Yuan et al. 2007] and Bilateral Richardson- the presence of large disk-like blurs such as the ones produced
Lucy [Yuan et al. 2008]. by spherical and chromatic aberrations (Section 7). Schuler et al.’s
Recently, there has been a lot of work on incorporating image method also suffers from the lack of global convergence stemming
priors into the deconvolution process, as an attempt to better cope from their use of a Hyper-Laplacian prior.
with very broad blur kernels and the complete loss of certain fre- In contrast to all these methods, our approach uses a convex
quencies. Such priors include convex terms such as total variation cross-channel prior that can be implemented efficiently and with
(TV, e.g., [Chan et al. 2011]) and total curvature [Goldluecke and guaranteed global convergence. As a result, it can produce excel-
Cremers 2011], which can be optimized with specialized solvers lent image quality even in the presence of very large blur kernels.
that are guaranteed to converge to the globally optimal solution. We emphasize that our contribution is not only to introduce a color
Other authors have used non-convex regularization terms that are prior, but also to find a convex optimization framework for this
optimized using techniques such as Iteratively Reweighted Least prior, which is a key component in achieving excellent image qual-
Squares (IRLS) [Levin et al. 2007; Joshi et al. 2009]. These meth- ity in the presence of very large blur kernels.
ods are not guaranteed to converge to a global optimum, but pro- Finally, there has been work on using chromatic aberrations to
duce state-of-the-art results in practice. Krishnan and Fergus [2009] increase the depth of field of the imaging system [Cossairt and Na-
introduce a prior based on the Hyper-Laplacian, and an efficient yar 2010; Guichard et al. 2009]. This problem is somewhat orthog-
solver to implement it. Like IRLS, this new solver is not guaran- onal to our goal, and could be added to our method as well. We also
teed to converge to a globally optimal solution. We show in Ap- note that Cossairt and Nayar [2010] still require an optical system
pendix B that this limitation is in fact practically relevant, and that in which all aberrations except for chromatic ones are minimized.
a TV solver with guaranteed global convergence will often produce
sparser gradient distributions than these Hyper-Laplacian solvers. 3. OVERVIEW
PSF Estimation. The goal of our work is to replace complex optical systems used
Although blind deconvolution methods exist, the best image qual- in modern camera objectives with very simple lens designs such
ity is achieved with known blur kernels. Several methods for esti- as plano-convex lenses, while achieving image quality comparable
mating the kernels exist, including spectral methods [Gibson and to modern cameras at the full resolution of current image sensors
Bovik 2000] and estimation from edge-based [Joshi et al. 2008] (Fig. 2).
or noise-based patterns [Brauers et al. 2010] or image pairs [Yuan The primary challenge in achieving these goals is that simple
et al. 2007]. lenses with spherical interfaces exhibit aberrations, i.e., higher-
order deviations from the ideal linear thin lens model [Seidel 1857].
Deconvolution for Aberration Correction. These aberrations cause rays from object points to focus imper-
As mentioned above, the use of deconvolution algorithms for cor- fectly onto a single image point, thus creating complicated point
recting for aberrations dates back at least to the original develop- spread functions that vary over the image plane, which need to be
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 3

overview of this optimization framework. We refer the reader to the

original work by Chambolle and Pock for an in-depth discussion.
The optimization framework considers general problems of the
form
xopt = argmin F (Kx) + G(x). (1)
x
Fig. 2. In this paper we propose to replace complex lens systems by a
single lens element plus a set of computational photography techniques. For example, in an inverse problem with a TV regularizer, the first
term in Eq. (1) is the regularizer (that is K(x) = ∇x, F(y) =
removed through deconvolution. The effect gets more pronounced kyk1 for TV), while the second term is the data fitting term (some
at larger apertures where more off-axis rays contribute to the image residual norm).
formation. Figure 3 shows the variation of the PSF over the image Let X, Y be finite-dimensional real vector spaces for the primal
plane for two simple lens elements. We can make several observa- and dual space, respectively. The operators from Eq. (1) are then
tions from these images: formally defined as:
K :X→Y is a linear operator from X to Y
G : X → [0, +∞) is a proper, convex, (l.s.c.) function. (2)
F : Y → [0, +∞) is a proper, convex, (l.s.c.) function.
where l.s.c. stands for lower-semicontinuous. The dual problem of
Eq. (1) is given as
yopt = argmax − (G∗ (−K∗ y) + F∗ (y)) , (3)
y

where the ∗ denotes the convex conjugate. To solve the above (pri-
Fig. 3. Patch-wise estimated PSFs for two simple lenses. The PSF estima- mal and dual) problem, the following algorithm is proposed by
tion and non-blind deblurring in our method is done in patches to account Chambolle and Pock [2011]:
for the PSFs’ spatial variance. Left: PSF of a single biconvex lens at f /2.0.
Right: PSF of a single plano-convex lens at f /4.5. A LGORITHM 1. (First-order primal-dual algorithm)
—Initialization: Choose τ, σ ∈ R+ , θ = 1 with τ σL2 < 1
Choose initial iterates (x0 , y0 ) ∈ X × Y, x̄0 = x0
—The blur is highly spatially varying, ranging from disc-like struc-
tures (spherical aberration) with diameters of 50 pixels and more —Iterations (n ≥ 0): Update xn , yn , x̄n as following:
to elongated streaks (coma and astigmatism). We can address 
this problem by subdividing the image into tiles over which we yn+1 = proxσF∗ (yn + σKx̄n )

assume a constant point spread function (see, e.g., [Levin et al. xn+1 = proxτ G (xn + τ K∗ yn+1 )
2007]). = xn+1 + θ (xn+1 − xn )

x̄
n+1
—The blur is highly wavelength dependent (chromatic aberration).
The resolvent or proximal operator with respect to G is defined
This results in objectionable color fringing in the image. At the
as:
same time, the PSF of at least one of the color channels often
proxτ G (x̃) := (I + τ ∂G)−1 (x̃)
contains more energy in high spatial frequencies than the others
(one channel is usually focused significantly better than the oth- 1
= argmin kx − x̃k22 + G(x) (4)
ers); note here that we do not require it to be perfectly in focus. x 2τ
This suggests that we may be able to utilize cross-channel infor-
mation and reconstruct spatial frequencies that are preserved in and analogously for proxσF∗ := (I + σ∂F∗ )−1 . It is assumed that
at least one of the channels. these operators are easy to compute, for example in closed form or
using a Newton-like method. The parameter L, which is necessary
The rest of the paper is structured as follows. In Section 5 we to compute valid σ, τ , is defined as the operator norm L = kKk2 .
describe a cross-channel prior, and an efficient convex optimiza- Note that Algorithm 1 never minimizes functions including both
tion algorithm for solving the deconvolution problem. Section 6 de- G and F∗ at the same time. This splitting of the minimizations
scribes a PSF estimation algorithm for recovering tile-based PSFs for G and F from Eq. (1) and the alternation between those mini-
such as the ones shown in Figure 3. Both the deconvolution algo- mizations is the high-level idea behind all forward-backward split-
rithm and the PSF estimation use the convex optimization frame- ting methods [Combettes and Pesquet 2011], [Friedlander 2011]. In
work by Chambolle and Pock [2011], which we summarize in Sec- each iteration, these methods first perform a forward step to mini-
tion 4. We conclude with results and a discussion in Sections 7 mize G in the objective function and then minimize F, where both
and 8. minimizations are coupled by terms that ensure that the solution is
close to the previous solution (note the first term in the minimiza-
4. REVIEW OF OPTIMAL FIRST-ORDER tion in Eq. (4)). A basic forward-backward method that illustrates
PRIMAL-DUAL CONVEX OPTIMIZATION this idea is the projected gradient method. Consider the example of
solving Eq. (1) with K(x) = x, F(y) = δC (y) and arbitrary G,
To solve both the image deconvolution and the PSF estimation where δC (y) is the indicator function for a set C that is 1 if y ∈ C,
problem for working with simple lenses, we derive optimization else ∞. Thus, in this example G is to be minimized with the hard
methods based on the optimal first-order primal-dual framework constraint x ∈ C. The projected gradient method would be the fol-
by Chambolle and Pock [2011]. In this section we present a short lowing iteration over n ≥ 0:
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
4 •

xn+1 = proxσF (xn − σ∇G(xn )), (5) 5.2 Deconvolution with Cross-Channel Information
| {z } | {z }
backward step forward step Real optical systems suffer from dispersion in the lens elements,
leading to a wavelength dependency of the PSF known as chro-
where σ is a step size. The iteration is illustrated in Fig. 4. We matic aberration. While complex modern lens assemblies are de-
can see that in each iteration the projected gradient method first signed to minimize these artifacts through the use of multiple lens
performs a gradient descent step and then a prox-step for F. For elements that compensate for each others’ aberrations, it is worth
the considered problem this is a Euclidean projection on the hard pointing out that even very good lenses still have a residual amount
constraint set C. Consequently the resolvent operators from Eq. (4) of chromatic aberration. For the simple lenses we aim to use, the
can be thought of as a generalization of the projection. chromatic aberrations are very severe – one color channel is fo-
cused significantly better (although never perfectly in focus) than
the other channels, which are blurred beyond recognition (of course
x01 excluding achromatic lenses which compensate for chromatic aber-
) rations).
G(x 0
−σ ∇ x02 Given individual PSFs B{1...3} for each color channel J{1...3} of
x0 proxσF proxσF an image J one might attempt to independently deconvolve each
)
x1 color channel. As Figure 5 demonstrates, this approach does not in
∇G(
−σ general produce acceptable results, since frequencies in some of the
x2 channels may be distorted beyond recovery. Note the severe ringing
∇G C
x1 in the top center and strong residual blur in the bottom center.

Fig. 4. Projected gradient method that illustrates the high-level idea be-
hind forward-backward splitting methods: Alternating sequence of gradient
descent (forward) steps and projection (backward) steps.

Many known sparse optimization methods fall into the class of

forward-backward splitting methods, illustrated by the projected
gradient example above. Other forward-backward splitting meth-
ods include ISTA, FISTA, Split Bregman, ADMM [Friedlander
2011]. A distinguishing property of Algorithm 1 is that the au-
thors show that it converges to a global minimum with the rate
O(1/N ), which is optimal for the general problem from Eq. (1)
(see again [Chambolle and Pock 2011]). Fig. 5. Effect of using the cross-channel priors (with the addition from
Sec. 5.5) for two image patches (full images in results section). Left: Orig-
5. DEBLURRING inal blurred capture. Middle: Independently deconvolved results for each
channel using the method of Levin et al. [2007]. Right: Deconvolution re-
Having summarized the general optimization framework, we now sult with our cross channel prior.
derive a specific and efficient deconvolution algorithm based on
this framework. We start by introducing an image formation model, We propose to share information between the deconvolution pro-
and motivating a novel cross-channel prior for deconvolving multi- cesses of the different channels, so that frequencies preserved in
channel images, then define an optimization function for deconvo- one channel can be used to help the reconstruction in another chan-
lution with this term, and cast it into the framework from the previ- nel. Our cross-channel prior is based on the assumption that edges
ous section. Next, we discuss an improvement of the basic method in the image appear in the same place in all channels, and that hue
to deal with dark image regions, and finally we describe how the de- changes are sparse throughout the image (also see Figure 6). These
veloped core method can be integrated into an efficient scale-space assumptions lead to the following prior for a pair of channels l, k
deconvolution method.
∇ik /ik ≈ ∇il /il
(8)
5.1 Image Formation Model ⇔ ∇ik · il ≈ ∇il · ik ,
Consider a grayscale image tile captured with n × m resolution. which we enforce in a sparse (`1 ) fashion. Note that the division
Let J , I , N ∈ Rn×m be the observed blurred image, the underly- and multiplication /, · are pixel-wise operators.
ing sharp image and additive image noise, respectively. The forma-
tion of the blurred observation with the blur kernel B can then be 5.3 Minimization Problem
formulated as Using this cross-channel prior, we formulate the problem of jointly
J = B ⊗I +N (6) deconvolving all channels as the optimization problem
3 5
j = Bi + n, (7) X X
(i1...3 )opt = argmin kBc ic − jc k22 + λc kHa ic k1
In the second form, B, j, i, and n are the corresponding quanti- i1...3 c=1 a=1
ties in matrix-vector form. As mentioned in the overview, an actual 2
(9)
X X
camera image will be composed of many tiles, each with a PSF that + βcl kHa ic · il − Ha il · ic k1 ,
is assumed constant over the tile. l6=c a=1

ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 5
Intensity Ir,g,b

ỹi
y = proxσF∗ (ỹ) ⇔ yi = (12)
max(1, |ỹi |)
x = proxτ G (x̃)
Position x x1 x2 x0 1
⇔ 0 = (x − x̃) + 2 BTc Bc x − BTc j


τ
Fig. 6. Blurred scanline on the left with different PSFs in each channel τ 2F (Bc )∗ F (jc ) + F (x̃)
 
and a sharp green channel. Reconstructed scanline on the right. Our cross- ⇔ x = F −1 , (13)
channel regularizer enforces gradient consistency between the color chan- τ 2 |F (Bc )|2 + 1
nels and allows sparse hue changes. Constant regions and pure luma gra- where F (.) in the last line denotes here the Fourier transform
dients have a low regularizer energy (regions shaded in light green), but and ỹ, x̃ are the function variables of the proximal operators as
changes in chroma are penalized (red regions). For the region between x1 defined in Eq. (4). The first proximal operator is just a per-pixel
and x2 , the extrapolated intensities of all channels intersect the x-axis in the projection operator (for detailed derivation we refer the reader
same point x0 , which visualizes Equation 8. By using an `1 norm, sparse to [Chambolle and Pock 2011]). The second proximal operator is
chroma changes occurring exactly at edges (right) are preferred over wider the solution to a linear system as shown in the second line. Since
regions (left) that are typically perceived as color fringing. the system matrix is composed of convolution matrices with a large
support we can efficiently solve this linear system in the Fourier do-
where the first term is a standard least-squares data fitting term, and main (last line).
the second term enforces a heavy-tailed distribution for both gradi- What still needs to be determined is the convex conjugate K∗ of
ents and curvatures. The convolution matrices H{1,2} , implement the linear operator K, which is given as follows:
the first derivatives, while H{3...5} correspond to the second deriva-
K∗ = S T = ST1 · · · STt
 
tives. We use the same kernels as Levin et al. [2007] but employ an 
 (14)
`1 norm in our method rather than a fractional norm. This ensures = λc HT1 · · · λc HT5 βcl HT1 Dil − DH1 il · · · ,
that our problem is convex. The last term of Eq. (9) implements
where t is the number of matrices S is composed of. In sum-
our cross-channel prior, again with an `1 norm. λc , βcl ∈ R with
mary, the matrix-vector multiplication K∗ (y) in Algorithm 1 can
c, l ∈ {1 . . . 3} are weights for the image prior and cross-channel
be expressed as the following sum
prior terms, respectively.
X t
K∗ (y) = ST y = STk y[(k−1)·nm,...,k·nm−1] , (15)
5.4 Deconvolution Algorithm k=1
The minimization from Eq. (9) is implemented by alternately min- where each STkis just a sum of filtering operations and point-wise
imizing with respect to one channel while fixing all the other chan- multiplications. Likewise, the resolvent operators given above can
nels. To optimize for this single channel x = ic we derive a first- be implemented using small filtering operators or the FFT for larger
order primal-dual algorithm adopting the framework described in filters.
Sec. 4. First, the optimization is rewritten in matrix-vector form as
Parameter Selection
λc H1
 
Algorithm 1 converges to the global optimum of the convex func-
 ..  tional if θ = 1, τ σL2 < 1 with τ, σ > 0 and L = kKk2
 . 

λc H5
 (see [Chambolle and Pock 2011] for the proof). We use θ = 1, σ =
0.9



βcl Dil H1 − DH1 il  x
 10 and τ = σL 2 for the deconvolution algorithm proposed in this
2
(ic )opt = argmin  + kBc x − jc k2 paper. That only leaves open how to compute the operator norm L.
..

x (10) Since we have K(x) = Sx where S was a matrix, kKk2 is
 
 . 
the square root of the largest eigenvalue of the symmetric matrix


βcl Dil H2 − DH2 il 

..
 ST S (see, e.g., [Rowland 2012]). We find the value L by using the
. power iteration where again all matrix-vector-multiplications with
1
2 ST S can be decomposed into filtering operations (other eigenvalue
= argmin kSxk1 + kBc x − jc k2 methods like the Arnoldi iteration are consequently also suited for
x
computing L).
where here D denotes the diagonal matrix with the diagonal taken
from the subscript vector. See Appendix A for a detailed derivation. 5.5 Regularization for Low-Intensity Areas
S is a matrix consisting of the sequence of all t = 5 + 2(3 − 1)
In this section, we propose a modification of the basic cross-
matrices coming from the `1 minimization terms in Eq. (9). By
channel prior that produces improved results in dark regions, i.e.,
comparison with Eq. (1), we can now define
for pixels where all channel values approach zero. In these regions,
K(x) = Sx the prior from Eq. (8) is not effective, since the hue normalization
reduces the term to zero. As a result, significant color artifacts (such
F(y) = kyk1 (11) as color ringing) can remain in dark regions, see Figure 7. Note that
G(x) = kBc x − jc k22 . by allowing luma gradients in the original cross-prior this is an in-
herent design problem of this prior and not an optimization issue.
Given this structure, the following resolvent operators neces- In these regions, we therefore propose to match absolute (rather
sary to apply Algorithm 1 are already known [Chambolle and Pock than relative) gradient strengths between color channels. The oper-
2011]: ator G from (11) is modified as
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
6 •

J = Jcz , B = Bcz ,
λ0 = λc

k=0

I0 = I0 =
∆J = J − Ik ⊗ B
Icz−1 ↑ Icz ↑

Iterate
Ik+1 = Ik + ∆J ⊗−1
λ B
Fig. 7. Chromatic artifacts in low-intensity areas. Left: Original blurred 1
k

patch. Center: Reconstruction using Eq. (9). Notice the chromatic artifacts λk+1 = 3 λk , k =k+1
in all low-intensity areas and the correct reconstruction in the other areas.
Right: Regularization for low-intensity areas added as discussed below.
Icz = Ik
2
XX z−1 Scale z z+1
G(x) = kBc x − jc k22 + λb kDw (Ha x − Ha il )k22 ,(16) Fig. 8. Iterative deconvolution of the residual image which eliminates
l6=c a=1
ringing artifacts, center. The operator ⊗−1
λk is the deconvolution of the im-
where Dw is a spatial mask that selects dark regions below a age to the left using the kernel to the right and a regularizer-weight λk .
threshold . The mask is blurred slightly with a Gaussian kernel Kσ For performance reasons the algorithm is implemented in scale space. This
to avoid spatial discontinuities at the borders of regions affected by figure shows the deconvolution on a level z in the pyramid. We set Ic−1 = 0.
the additional regularization term:
 P   See again Fig. 8, where ↑ is an upsampling operator to the next
βcl T (il ) 1 ; ik ≤  finer scale. We use nearest neighbor since it preserves edges. The
w = 1 − lP ⊗ Kσ with T (i)k = ,
l βcl
0 ; else pyramids {J z }Z z Z
z=0 , {B }z=0 of the blurred image/kernel pairs are
computed by bicubic downsampling of J , B with the scale fac-
and  = 0.05 and σ = 3 in our implementation.
tor 21 . The reconstruction pyramid {I z }Z
z=0 is progressively recov-
The resolvent operator with respect to G from Eq. (13) is re-
ered from coarse (scale 0) to fine, where at each scale the initial
placed by the following derivation (see Appendix A for details):
iterate is the upsampled result of the next coarser level. Note, that
uopt = proxτ G (ũ) our scale space implementation purely serves as a performance im-
  provement. In particular we do not need to overcome local minima
2
XX in our minimization problem since it is convex.
⇔ 2τ BTc Bc + I + 2τ λb HTa D2w Ha  uopt Contrary to [Yuan et al. 2007], we are also not using any infor-
l6=c a=1 mation from coarser scales in the deblurring at a considered scale.
2
XX Since the reconstructions can contain significantly less detail we
= 2τ BTc jc + ũ + 2τ λb HTa D2w Ha il found that guiding fine scale deblurring with coarser scale infor-
l6=c a=1 mation is problematic in many cases.
⇔ Auopt = b (17)
6. PSF ESTIMATION
This expresses the solution of the resolvent operator as the matrix
inversion problem Auopt = b. Since blur kernel sizes of the order The previous section assumes that the PSF of the optical system is
of magnitude of 102 × 102 can be expected for practical applica- given for each image tile. While the PSF can be obtained by any
tions, A is very large and impractical to invert explicitly. The sys- existing technique, we use a calibration-pattern-based approach. In
tem is solved using the Conjugate Gradient (CG) algorithm. This contrast to methods that directly measure the PSF like Schuler et
allows us to express the matrix-vector multiplication in the CG- al. [2011], no pinhole-light source and a dark-room lab is necessary.
algorithm as a sequence of filtering operations as before. Instead we just use a consumer laser-printer to make our targets. To
estimate the PSFs from the target images, it is natural to apply the
5.6 Residual and Scale-Space Deconvolution same optimization framework that was used for deblurring also for
the PSF estimation step. This method is detailed below. Note again
The basic deconvolution approach described so far can be ac- that any PSF estimation method could be used here and the whole
celerated and further improved by adopting a scale-space ap- estimation process is a calibration procedure, which only needs to
proach [Yuan et al. 2008] as well as residual deconvolution [Yuan be performed once for each lens.
et al. 2007].
The idea behind residual deconvolution is to iteratively decon- 6.1 PSF Estimation as Deconvolution
volve the residual image starting with a large regularization weight
which is progressively decreased. Since the residual image has The PSF estimation problem can be posed as a deconvolution prob-
a significantly reduced amplitude, its deconvolved reconstruction lem, where both a blurred image and a sharp image of the same
contains less ringing which is proportional to the amplitude. The scene are given. The blurred image is simply the scene imaged
iteration is shown in the center of Fig. 8. through the simple lens, with the aperture open, while the sharp
Our method handles saturation in the blurred image by remov- image can be obtained by stopping the lens down to a small, almost
ing the rows where j is saturated from the data fitting term. This pinhole aperture, where the lens aberrations no longer have an ef-
is done by pre-multiplying the residual Bc x − jc with a diagonal fect. By acquiring a sharp image this way (as opposed to a synthetic
weighting matrix whose diagonal is 0 for saturated rows and 1 else; sharp image) we avoid both geometric and radiometric calibration
the derivation from Eq. (17) is changed straightforwardly. issues in the sharp reference image.
To increase the performance of the algorithm by using good start- Let J be an image patch in a considered blurred channel, I the
ing points for Eq. (10), the method is performed in scale space. corresponding sharp pinhole-aperture patch. We estimate a PSF
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 7

Bopt describing the blur in J by solving the minimization problem

rbnd
bopt = argmin kIb − s · jk22 T
+ λk∇bk1 + µk1 b − 1k22 , (18)
b

where the first term

P is a linearPleast-squares data fitting term, and
the scalar s = k,l I (k, l)/ k,l J (k, l) accounts for the differ-
ence in exposure between the blurred and pinhole image. The sec-
ond term represents a standard TV prior on the gradients of the
recovered PSF, P and the third term enforces an energy conservation
constraint, i.e., k,l B (k, l) = 1. Fig. 9. Proposed PSF calibration target consisting of a grid of framed
Note that Eq. (18) is a convex optimization problem. We derive a white noise patches. If the frame width is rbnd ≥ blur-radius, a bound-
first-order primal-dual algorithm adopting the framework described ary condition propagating the boundary pixels outside (e.g., replicate the
in Sec. 4. Specifically, Eq. (18) is expressed using the following pixels) is a physically correct model for the blurred and sharp noise patch.
operators adopting the notation from before:
3 set of neighboring tiles. Although the PSFs may contain high
K(x) = ∇x frequency features, these tend to change smoothly over the image
F(y) = kyk1 plane. Combined with the relatively small tile size, we found that
(19) this spatial filtering does not cause feature loss, but reduces noise
1 µ
G(x) = kIb − s · jk22 + k1T b − 1k22 significantly.
λ λ
The following resolvent operators and convex conjugates necessary
to apply Algorithm 1 are then provided in [Chambolle and Pock 7. RESULTS
2011]: yi In this section we present a number of results captured with simple
y = proxσF∗ (ỹ) ⇔ yi = (20a) lens elements attached to a Canon EOS D40 camera, as well as
max(1, |ỹi |)
detailed comparisons with state-of-the-art deconvolution methods.
K∗ x = ∇∗ x = − div x (20b) For figures where we choose to present cropped regions to highlight
However, the resolvent operator with respect to G(u) has to be detail structure, the full native camera resolution image is provided
derived for our problem. As we show in Appendix A, we can ex- in the supplemental material. We encourage the reader to zoom in
press it as a sequence of frequency domain operations: on the individual images in the electronic version of this paper.

uopt = proxτ G (ũ) Efficiency.

∗ λ
! First we compare the performance of our method to several other
τ sF (I) F (j) + F (ũ) + τ µF (1) (21)
sparse optimization approaches. To this end, we implemented
= F −1 2
2
λ
,
τ |F (I)| + 2
+ τ µF (O) our objective function, including the cross-channel prior in the
following solvers: Split-Bregman [Goldstein and Osher 2009],
where O is a convolution matrix consisting only of ones. FISTA [Beck and Teboulle 2009], and Nesterov [Nesterov 2005],
Using these operators, we can again apply Algorithm 1. The which are all recent, state-of-the-art sparse optimizers with guar-
computation of L has been described in Sec. 5.4 and the same τ anteed convergence to a global optimum. They therefore produce
and σ can be used. identical results to our method. To keep the comparison fair, we
disabled the residual and multiscale parts of our algorithm. All ex-
6.2 Calibration Pattern periments were performed on a single core of an Intel Core2 Quad
To allow for robust PSF estimation, the scene used for this pur- CPU with 2.40GHz and 4GB RAM.
pose should have a broad spectrum. We therefore decide against The test images are standard images used in imaging research,
both natural scenes and edge based patters (e.g., [Trimeche et al. that were convolved with the PSFs shown in Figure 10. All al-
2005; Joshi et al. 2008]), and instead adopt a white noise pattern. gorithms were run until the per-iteration change in the result fell
See [Brauers et al. 2010] for a comparison of a white-noise pattern below a given threshold. Table I shows the timing results for the
to different other calibration patterns. Our specific pattern, shown different algorithms and images. We note that our algorithm consis-
in Figure 9, is subdivided into several noise patches, separated by tently outperforms the alternatives. For comparison we have added
a white border. Each noise tile will produce a PSF for a single im- the last two rows in Table I which show the timings for the method
age tile, while the white frame helps suppress boundary effects in from Levin et al. [2007] both in its original form, and a modified
the PSF estimation process. This is because it enables a physically version with our cross-channel prior added. Our method is signif-
correct constant boundary condition around each patch if the frame icantly faster than the version with the cross prior, but somewhat
width is rbnd ≥ blur-radius (see again Fig. 9). slower than Levin without our prior (as shown later in Figure 18,
that method produces poorer results than our method). We note,
6.3 Full PSF Estimation Framework however, that Levin’s method only performs two iterations of IRLS,
while we run our algorithm to convergence. Also, our use of multi-
To increase the efficiency of the PSF estimation, we apply the al- scale deconvolution significantly speeds up the full algorithm.
gorithm introduced above in scale space. We initialize our iterative
minimization at each scale with the upsampled results from the next Results from different lenses.
coarser scale, which yields a good starting point for the convex ob- Figures 1 and 11 show several results from the most basic refrac-
jective function from Eq. (18) and thus speeds up the minimization. tive optical system, a single plano-convex lens (focal length 130
After the initial PSF estimation, we perform an additional step mm, f/4.5) shown lying on the side in Figure 1. For all results in
of smoothing by computing weighted averages of PSFs for a 3 × this paper, ISO100 and autoexposure settings were used, which we
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
8 •

Fig. 10. Blur kernels used for Table I (all 45 × 45). From left to right:
a) simple lens kernel estimated with our method, b) standard lens kernel
estimated with our method, c) out-of-focus kernel from [Levin et al. 2007],
d) motion blur from [Yuan et al. 2007].

Table I. Runtime comparison using different images and kernels.

All times are in seconds.
Image: Mandrill Cameraman Lena
Mean:
Kernel: a) b) c) d) a) b) c) d) a) b) c) d)
Proposed method 14.5 11.7 22.8 18.2 19.8 12.9 23.6 19.0 20.7 10.2 25.5 21.6 18.4
Split Bregman 34.5 31.1 37.5 37.9 32.6 28.3 39.2 32.3 34.8 29.1 40.0 31.9 34.1
FISTA 320.7 304.8 385.4 341.6 443.9 413.5 422.5 404.4 425.5 409.9 407.5 376.3 388.0
Nesterov 344.8 320.5 378.7 361.9 462.4 440.5 453.2 431.7 451.7 440.8 424.3 405.5 409.7
Levin+cross term 53.5 27.6 72.3 71.6 40.6 26.4 69.4 71.9 40.2 24.3 76.4 74.8 54.1
Levin 11.1 11.4 11.3 11.6 11.8 11.8 11.4 11.0 11.3 11.8 11.2 11.4 11.4

consider as standard settings. The corresponding PSFs can be found

on the right side of Figure 3. We note that all results show good de-
tail down to the native sensor resolution, demonstrating that our
method is indeed capable of producing high quality digital pho-
tographs with very simple lens designs. The parking meter image
and the twigs show that deconvolving with the PSF for one partic-
ular scene depth preserves the deliberate defocus blur (bokeh) for
objects at different depths without introducing artifacts. The man-
hole cover image at the top shows focus change over a tilted plane.
Saturated regions as shown in the second inset of this image are
handled properly by our method. The supplemental material con-
tains even more plano-convex lens results than Fig. 11. It also in-
cludes a result under low-light conditions.
Figures 14 and 12 test the limits of our method using lenses with
a larger aperture of f /2. Figure 14 shows the results for a 100 mm
biconvex lens, with PSFs from Figure 3. The larger aperture in-
creases the depth dependency of the PSF. Therefore our deconvolu-
tion method produces very sharp reconstructions in-plane (see color
checker in Fig. 14), but suffers from some ringing out-out-of-plane
(see text on the foreground object). A similar depth dependency is
visible in Figure 12, where we show results for a 100 mm achro-
matic doublet (PSFs can be found in the supplemental material).
Considering that the PSFs exhibit disk-like structures in all chan-
nels, our deconvolution method still produces detailed reconstruc-
tions. The depth-dependency of the PSFs (that is the ringing in the
images) can be minimized through a stronger weight on the regular- Fig. 11. Images captured with a simple plano-convex lens (left) and re-
ization terms, which, however, introduces cartoon-like artifacts at stored with our method (right). Note recovered spatial detail and the absence
the highest resolution. Figure 13 shows a result of the same achro- of color fringing. Also note the bokeh and graceful handling of out-of-focus
mat stopped down to f /4.5. This significantly reduces the artifacts blur in the top three images.
in results produced by our method. Note the strong chromatic ar-
tifacts in the naively deconvolved result, while our reconstruction
contains significantly reduced ringing that is not chromatic. In the
future one could imagine to improve the depth dependency by esti-
mating depth-depending PSFs (see Section 8).
Results for a commercial camera lens, a Canon 28–105 mm
zoom lens at 105 mm and f/4.5, are shown in Figures 15 and 16
(see supplemental material for full images). While this commercial
lens shows much reduced aberrations compared to the uncorrected
optics used above, there still are some residual blurs that can be
removed with our method. In particular, the vertical edges of the
calibration pattern in Figure 16 reveal a small amount of chromatic Fig. 12. Left: Cropped central region of the image from achromatic lens
aberration that is removed by our method. The PSFs for this lens with f /2.0. Right: Deblurred version.
are around 11 × 11 pixels in diameter.
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 9

Fig. 16. Patches from an image captured with a commercial Canon lens
(105 mm, f/4.5). Left: Captured input patches. Right: our results. The full
Fig. 13. Left: Captured input image from achromatic lens with f /4.5. image can be found in the supplemental material.
Right: Deblurred version. The third inset on the right is the patch recov-
ered with Levin et al. [2007] naively on independent channels.

Fig. 14. Left: Captured input image from biconvex lens at f /2. Right:
Deblurred version. A very low cross-prior weight has been used here to
illustrate the depth-dependency of the PSF.

Fig. 17. Results for a multispectral camera with custom optics. The top
row shows PSF estimation results for 20 nm wavelength increments, rang-
ing from 420 to 720 nm (blue: image center, red: upper left corner). The
bottom set of images and insets shows the captured image (left) and the
deconvolved result (right) after mapping to sRGB space.

stimulus response) will have different blur kernels, so there can al-
ways be situations where the assumption of fixed per-channel PSFs
will fail, such as for example light sources with multiple narrow
spectra or albedos with very narrow color tuning. This will intro-
duce errors in the data-fitting term of our objective function. Since
we have a strong cross-channel and image prior (see comparisons to
Fig. 15. Test scene captured with a commercial Canon lens (105 mm, other priors below), we are still able to reconstruct images with high
f/4.5). Left: Captured input image. Right: Deblurred result. quality. For future research we imagine using a physical model of
the full depth and wavelength-dependent PSF (see Section 8). Such
Multispectral camera. a model may then allow to obtain a plausible spectral reconstruc-
Our aberration correction method cannot only be used for regular tion of the image, similar to what Rump and Klein [2010] show.
RGB cameras but also for multispectral imagers. In this case, the We have compared our method against several state-of-the-
cross-channel prior is applied for all pairs of frequency bands. Fig- art deconvolution algorithms, including the one from Levin et
ure 17 shows results for a multispectral camera with custom multi- al. [2007]. Figure 13 shows an additional inset of per-channel de-
element optics. As with the conventional cameras, our method suc- convolution with their method. Compared to our result, it shows
cessfully removes chromatic aberration and restores lost image de- more prevalent ringing artifacts, which are particularly noticeable
tail in the blurred channels. Considering the wavelength-dependent because they also have chromatic component. Our cross-channel
PSFs here, we want to point out that the assumption of fixed PSFs prior successfully suppresses chromatic artifacts, but also helps to
for each color-channel of an RGB-sensor is often violated. This transfer information between channels for reduced ringing and in-
assumption is made for all the RGB sensor results in this paper creased detail overall. Figure 18 shows another comparison. Here
and is a classical assumption in deconvolution literature. However, we also implemented our cross-channel prior as an additional regu-
one cannot tell from a tri-chromatic sensor the exact wavelength larization term for Levin et al.’s method. While this improves the re-
distribution. Metamers (different spectra that produce the same tri- sult, it does not match the quality of our method. The IRLS solver is
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
10 •

Fig. 18. Top row from left to right: Captured image, deconvolved result
from [Levin et al. 2007] independently on each channel, IRLS method
from [Levin et al. 2007] extended with our prior, our result. Bottom row:
Patches in same order of the methods. Fig. 19. Top row from left to right: Captured image, our deconvolved re-
sult, result from [Schuler et al. 2011]. Patch matrix below from top left to
unstable, not guaranteed to converge, does not minimize the sparse bottom right: Captured image, result from [Schuler et al. 2011], our result
norm close to its non-differentiable origin, and thus cannot achieve with strong regularization (smoother reconstruction), our result with low
the results of our method. We especially found robustness to be a regularization (more detailed reconstruction) .
general issue once the cross-channel prior was added to IRLS, with
the resulting method being very sensitive to parameter selection and parameters documented in their paper have been used. Note that our
often failing completely. prior removes nearly all chromatic aberrations, while Schuler’s Yuv
To further investigate the convergence properties of Hyper- regularization fails with the large aberrations that we can handle.
Laplacian priors, we conducted more detailed experiments with
the method by Krishnan and Fergus [2009] and compared it with 8. CONCLUSIONS AND FUTURE WORK
a plain TV regularized solution (Appendix B) for which there are
many solvers that guarantee convergence to a global optimum. We We have proposed a new method for high-quality imaging using
found that the TV solution often produces sparser gradient dis- simple lens elements rather than complex lens designs. This is
tributions (i.e., lower Hyper-Laplacian residual) than the solvers made possible by a new, robust deconvolution with a novel cross-
designed to optimize for these objectives. We can therefore con- channel gradient prior that enforces sparsity of hue changes across
clude that Hyper-Laplacian regularization often fails at its objective the image. The prior allows detail transfer between different chan-
to produce sparser gradient distributions, at least with the solvers nels and suppresses reconstruction artifacts in individual channels.
available today. Both this deconvolution algorithm and a related PSF estimation
We have also compared our method to Schuler et al. [2011]. method rely on a convex optimization framework [Chambolle and
First, in Fig. 19 we show results using their datasets (images and Pock 2011], which guarantees convergence to a globally optimal
estimated PSFs). We show results with two different weights for solution as well as optimal convergence rates. We demonstrate that
the regularizer. Note that in both cases, our method recovers more this solver is also faster than alternative methods in practice. Fur-
fine-scale structure than their approach. Since Schuler et al. use a ther improvements in terms of both performance and quality are
Hyper-Laplacian objective with the solver analyzed in Appendix B, achieved by adding residual and multiscale deconvolution.
we again achieve a more heavy-tailed gradient distribution in our Overall our method produces image quality comparable to that
result. However, the results for this dataset do not match the quality of commercial point-and-shoot cameras for reasonable apertures
we can achieve with our own datasets – for varying regularization (around f/4.5) even with the most basic refractive imaging optics
weight always a strong residual blur remains, especially around the imaginable: a single plano-convex lens. For much larger apertures
leaves and window edges. of f/2 and more, the quality degrades due to the depth dependency
We believe these artifacts are due to issues with the estimated of the PSF and significantly larger disk-shaped blur kernels, which
PSFs (which we take directly from Schuler et al.) rather than is- destroy much of the frequency content. We point out that, in com-
sues in the actual deconvolution method. We note that the PSFs parison to other deconvolution algorithms, our cross-channel prior
lack a very low-frequency component, which we have observed still manages to transfer significantly more detail between channels
with every single lens element we have tested in our setup. Instead, to produce an acceptable result, although not at the quality level of
their blur kernels exhibit very thin structures akin to motion blur a high-end lens in combination with a good SLR camera. We con-
kernels and contain very little chromatic variation. A missing low- clude that in order to achieve that level of image quality it may still
frequency component in the kernels could easily explain the poor be necessary to optimize the lens design to partially compensate
quality of the deconvolved results. Unfortunately, Schuler et al. do for aberrations. However, we believe that even in this case, future
not specify details about the lens, so we are unable to obtain similar lenses can be simpler than they are today, thus paving the road for
images using our setup. lighter, cheaper, and more compact camera lenses.
We note that Schuler et al.’s Yuv-cross-channel regularization At the moment, we are deconvolving the images with PSFs that
breaks down when the chromatic aberrations increase. It com- are calibrated for a single scene depth and that are fixed per chan-
pletely fails for many of the large aberrations that we can handle. nel. One possible extension of our method is to perform decon-
In Fig. 20 we compare our cross-channel prior against their Yuv- volution with PSFs for different image depths and wavelengths.
cross-channel regularization. How to handle depth-dependency has been demonstrated in, e.g.,
In this case we did not have access to their original implemen- Levin et al. [2007], we have suggested a direction for wavelength-
tation, so we reimplemented their method. To solve Eq.(5) from dependent deconvolution in Section 7. Doing so would require a
their paper, we have adapted the code from the Hyper-Laplacian large wavelength-dependent 3D table of PSFs, which may be dif-
solver [Krishnan and Fergus 2009] as suggested by the authors. All ficult to calibrate and expensive to store. However, since we are
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 11

C HO , T. S., J OSHI , N., Z ITNICK , C. L., K ANG , S. B., S ZELISKI , R.,

AND F REEMAN , W. T. 2010. A content-aware image prior. In Computer
Vision and Pattern Recognition, IEEE Conference on. 169–176.
C HO , T. S., Z ITNICK , C. L., J OSHI , N., K ANG , S. B., S ZELISKI , R., AND
F REEMAN , W. T. 2012. Image restoration by matching gradient distri-
butions. Pattern Analysis and Machine Intelligence, IEEE Transactions
on 34, 4, 683–694.
C HUNG , S.-W., K IM , B.-K., AND S ONG , W.-J. 2009. Detecting and
eliminating chromatic aberration in digital images. In Image Processing
(ICIP), 2009 16th IEEE International Conference on. 3905–3908.
C OMBETTES , P. L. AND P ESQUET, J.-C. 2011. Proximal splitting meth-
ods in signal processing. Fixed-Point Algorithms for Inverse Problems in
Science and Engineering, 185–212.
C OSSAIRT, O. AND NAYAR , S. 2010. Spectral focal sweep: Extended
depth of field from chromatic aberrations sweep. In Computational Pho-
tography (ICCP), 2010 IEEE International Conference on. 1–8.
F RIEDLANDER , M. P. 2011. Sparse optimization minitutorial. http://
www.cs.ubc.ca/~mpf/papers/mpf-tutorial-siopt11.pdf.
G AUSS , C. 1841. Dioptrische Untersuchungen. Dieterich.
G IBSON , J. D. AND B OVIK , A., Eds. 2000. Handbook of Image and Video
Processing. Academic Press, Inc.
G OLDLUECKE , B. AND C REMERS , D. 2011. Introducing total curvature
for image processing. In Proc. ICCV. 1267–1274.
G OLDSTEIN , T. AND O SHER , S. 2009. The Split Bregman method for
L1-regularized problems. SIAM J. Img. Sci. 2, 323–343.
G UICHARD , F., N GUYEN , H. P., T ESSIRES , R., P YANET, M., TAR -
CHOUNA , I., AND C AO , F. 2009. Extended depth-of-field using sharp-
ness transport across color channels. In Proceedings of SPIE, the Inter-
national Society for Optical Engineering. 72500N.
J OSHI , N., L AWRENCE , C., Z ITNICK , R., S ZELISKI , D., AND K RIEG -
MAN , J. 2009. Image deblurring and denoising using color priors. In
Computer Vision and Pattern Recognition, IEEE Conference on. 1550–
1557.
J OSHI , N., S ZELISKI , R., AND K RIEGMAN , D. J. 2008. PSF estimation
using sharp edge prediction. In Computer Vision and Pattern Recogni-
Fig. 20. Images from Fig. 11, 1 reconstructed using our cross-channel tion, IEEE Conference on. 1–8.
prior (left) and restored with the Yuv-regularization from [Schuler et al.
K ANG , S. B. 2007. Automatic removal of chromatic aberration from a
2011] (right).
single image. In Computer Vision and Pattern Recognition, IEEE Con-
ference on. 1–8.
working with very simple optics, an alternative similar to [Shih
K EE , E., PARIS , S., C HEN , S., AND WANG , J. 2011. Modeling and re-
et al. 2012] may be to generate the PSFs procedurally from pa-
moving spatially-varying optical blur. In Proc. IEEE ICCP. 1–8.
rameters of the optical system, such as lens curvature, glass type, as
well as calibrated or measured focus and alignment parameters. We K RISHNAN , D. AND F ERGUS , R. 2009. Fast image deconvolution using
imagine to improve significantly on their results since our simple Hyper-Laplacian priors. Advances in Neural Information Processing Sys-
optical system leads to only a handful of optimization parameters. tems 22, 1–9.
L EVIN , A., F ERGUS , R., D URAND , F., AND F REEMAN , W. T. 2007. Im-
age and depth from a conventional camera with a coded aperture. In ACM
REFERENCES Trans. Graph. (Proc. SIGGRAPH).
B ECK , A. AND T EBOULLE , M. 2009. A fast iterative shrinkage- L UCY, L. 1974. An iterative technique for the rectification of observed
thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2, distributions. The Astronomical Journal 79, 745+.
183–202. M AHAJAN , V. N. 1991. Aberration Theory Made Simple. SPIE.
B RAUERS , J., S EILER , C., AND A ACH , T. 2010. Direct PSF estimation N ESTEROV, Y. 2005. Smooth minimization of non-smooth functions.
using a random noise target. In Proceedings of SPIE, the International Math. Program. 103, 127–152.
Society for Optical Engineering. 75370B. R ASHED , R. 1990. A pioneer in anaclastics: Ibn sahl on burning mirrors
C HAMBOLLE , A. AND P OCK , T. 2011. A first-order primal-dual algo- and lenses. Isis 81, 3, 464–491.
rithm for convex problems with applications to imaging. J. Math. Imag- R ICHARDSON , W. H. 1972. Bayesian-based iterative method of image
ing Vis. 40, 120–145. restoration. J. Opt. Soc. Am. 62, 55–59.
C HAN , S., K HOSHABEH , R., G IBSON , K., G ILL , P., AND N GUYEN , T. ROWLAND , T. 2012. Operator norm. From MathWorld - A Wolfram
2011. An augmented lagrangian method for total variation video restora- Web Resource, created by Eric W. Weisstein. http://mathworld.
tion. IEEE Trans. Image Proc. 20, 11, 3097–3111. wolfram.com/OperatorNorm.html.
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
12 •

RUMP, M. AND K LEIN , R. 2010. Spectralization: Reconstructing spec- P ROOF FOR (17).
tra from sparse data. In SR ’10 Rendering Techniques, J. Lawrence and
M. Stamminger, Eds. Eurographics Association, Saarbruecken, Germany, uopt = proxτ G (ũ)
1347–1354. 1
= argmin ku − ũk22
S CHULER , C. J., H IRSCH , M., H ARMELING , S., AND S CH ÖLKOPF, B. u 2τ
2011. Non-stationary correction of optical aberrations. In Computer Vi- + kBc u − jc k22
sion (ICCV), IEEE International Conference on. IEEE, 659–666.
2
S EIDEL , L. 1857. Über die Theorie der Fehler, mit welchen die durch XX
+λb kDw (Ha u − Ha il )k22
optische Instrumente gesehenen Bilder behaftet sind, und über die
l6=c a=1
mathematischen Bedingungen ihrer Aufhebung. Abhandlungen der | {z }
Naturwissenschaftlich-Technischen Commission bei der Königl. Bay- Φ(u)
erischen Akademie der Wissenschaften in München. Cotta.
∂Φ(uopt ) 1
= (uopt − ũ) + 2 BTc Bc uopt − BTc j


S HIH , Y., G UENTER , B., AND J OSHI , N. 2012. Image enhancement using ⇔0=
∂u τ
calibrated lens simulations. In Computer Vision ECCV 2012, A. Fitzgib- 2 
bon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds. Lecture Notes XX 
+ 2λb (Dw Ha )T Dw Ha uopt − (Dw Ha )T Dw Ha il
in Computer Science, vol. 7575. Springer Berlin Heidelberg, 42–56.
l6=c a=1
T RIMECHE , M., PALIY, D., V EHVIL ÄINEN , M., AND K ATKOVNIK , V.
2005. Multichannel image deblurring of raw color components. In Proc. ∂Φ(uopt ) 1
= (uopt − ũ) + 2 BTc Bc uopt − BTc j


⇔
Computational Imaging III. 169–178. ∂u τ
2
W IENER , N. 1964. Extrapolation, Interpolation, and Smoothing of Station- XX
HTa D2w Ha uopt − HTa D2w Ha il = 0


ary Time Series. The MIT Press. + 2λb
l6=c a=1
Y UAN , L., S UN , J., Q UAN , L., AND S HUM , H.-Y. 2007. Image deblurring  
with blurred/noisy image pairs. ACM TOG (Proc. SIGGRAPH) 26, 3, 1. 2
XX
Y UAN , L., S UN , J., Q UAN , L., AND S HUM , H.-Y. 2008. Progressive ⇔ 2τ BTc Bc + I + 2τ λb HTa D2w Ha  uopt
inter-scale and intra-scale non-blind image deconvolution. ACM Trans. l6=c a=1
Graph. 27, 3 (Aug.), 74:1–74:10. 2
XX
= 2τ BTc jc + ũ + 2τ λb HTa D2w Ha il
APPENDIX l6=c a=1

⇔ Auopt = b
A. PROOFS
This section contains all the proofs for the mathematical results P ROOF FOR (21).
presented in the paper.
uopt = proxτ G (ũ) =
P ROOF FOR (10).
1 1 µ T 2
argmin ku − ũk22 + kIu − s · jk22 + 1 u−1 2
(ic )opt u 2τ λ λ
| {z }
5
X Φ(u)
= argmin kBc x − jc k22 + λc kHa xk1 ∂Φ(uopt ) 2 T

x ⇔ = I Iuopt − s · j
a=1
2
∂u λ
X X 1 2µ T

+ βcl kHa x · il − Ha il · xk1 + (uopt − ũ) + 1 uopt − 1 1 = 0
l6=c a=1
τ λ
λ λ
⇔ uopt − ũ + τ IT Iuopt − s · IT j + τ µ(Ouopt − 1) = 0


5 2
2 2
X X X

= argmin λc kHa xk1 + βcl Dil Ha − DHa il x 1 !
x
a=1 l6=c a=1 −1 τ sF (I)∗ F (j) + λ2 F (ũ) + τ µF (1)
⇔ uopt = F
+kBc x − jc k22 τ |F (I)|2 + λ2 + τ µF (O)
λc H1
 
 .. 
 . 

 λc H5
 B. COMPARISON OF HYPER-LAPLACIAN AND TV



Dil H1 − DH1 il  x
 REGULARIZATION
= βcl + kBc x − jc k22

..

 
 . 

 Qualitative Comparison. Often IRLS-based solvers or solvers tar-
βcl Dil H2 − DH2 il 
  geting non-convex objective functions fail to converge to a global
..
. minimum. Figure 21 shows an example for the non-convex func-
1
tion solved by Krishnan and Fergus [2009] using the kernels intro-
= kSxk1 + kBc x − jc k22 duced in the same paper. A single channel grayscale deconvolution
problem is solved using the unaltered code published from the au-
thors’ website. We compare these results against a single-channel
ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.
 • 13

Table II. Comparison of the `2/3 -regularized objective f (x) = λ2d kBx − jk22 + k∇xk2/3 and its regularizer component r(x) = k∇xk2/3
using our solver with plain TV-regularization and the method from [Krishnan and Fergus 2009] for different images and kernels. All values in
the table are in 103 . Although we are actually minimizing the `1 -regularized approximation of f (x), our method finds in most cases a better
objective with a more sparse solution than what the method from [Krishnan and Fergus 2009] produces.
Image: Mandrill Cameraman Lake Lena House
Kernel: a) b) c) d) a) b) c) d) a) b) c) d) a) b) c) d) a) b) c) d) Mean:
f (x) Hyper-Laplacian 230.86 155.80 254.67 324.29 313.79 173.04 330.45 394.65 271.05 150.83 296.63 383.01 248.62 94.68 253.23 351.58 453.08 349.97 488.47 525.24 302.19
f (x) our TV-solver 229.47 149.77 253.35 319.66 308.34 170.19 324.16 388.76 266.24 146.14 292.35 384.79 245.24 91.17 247.90 347.58 445.21 350.53 482.01 516.30 297.95
r(x) Hyper-Laplacian 15.70 31.24 13.75 22.31 12.86 17.37 12.28 15.67 18.05 24.39 17.96 22.41 15.87 20.02 15.84 18.52 14.17 16.68 14.95 17.02 17.85
r(x) our TV-solver 16.72 27.84 10.72 21.49 10.52 14.73 8.21 13.11 15.64 20.65 12.87 19.37 13.40 17.18 11.65 16.39 11.10 14.22 9.92 14.73 15.02

TV-regularizer with the same kernels and without scale-space and

residual deconvolution to make the comparison fair. The same reg-
ularization weights have been used, which means that the global
optimum of the Hyper-Laplacian objective function should have a
gradient distribution with a heavier tail than the one of the TV so-
lution.
However, as seen in the figure the distribution obtained using
Krishnan and Fergus’ method [2009] exhibits a lighter tail than the
solution from an `1 -solver with guaranteed convergence, suggest-
ing that the the global optimum for the Hyper-Laplacian objective
has not been found.
14
L1
Hyperlaplacian L
12 (2/3)
Log2 Gradient Magnitude

10

0
-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5
Gradient

Fig. 21. Gradient distributions of the right synthetic blurred single-

channel image deconvolved with the according PSF in its upper left. Gaus-
sian noise σ = 10−4 and quantization was applied. Note, that our `1 -
regularized solution features a heavier tail than the approximate solution
for the ` 2 -regularized problem computed with the method from [Krishnan
3
and Fergus 2009].

Quantitative Comparision. This suspicion can be further corrob-

orated by systematic experiments with larger sets of images. Using
the same images as for Table I, we compute both an `2/3 regular-
ized solution (again using the method by Krishnan and Fergus), as
well as a TV solution, and then compute the numerical value of
the `2/3 objective function for both results (see Table II). We note
that in most cases, the TV solver actually produces a more optimal
solution to the `2/3 problem than the dedicated Hyper-Laplacian
solver.
These results also explain why Krishnan and Fergus [2009] in
their work found only very minor differences in SNR values be-
tween the two methods. Our own experiments confirm these find-
ings.
We conclude, that, although Hyper-Laplacian norms should in
theory provide better regularization than simple TV terms, we in
practice do not see these improvements due to the inability of cur-
rent solvers to find the global optimum. These findings support
our choice of `1 norms for the various regularization terms in our
method, and stress the significance of convex methods with guar-
anteed convergence to a global optimum.

ACM Transactions on Graphics, Vol. VV, No. N, Article XXX, Publication date: Month YYYY.