Eliminating Ghosting and Exposure Artifacts in Image Mosaics

Matthew Uyttendaele Ashley Eden Richard Szeliski

Microsoft Research Harvard University Microsoft Research

Abstract areas of overlap between images. When the images are

stitched together, we take a composite of the
overlapping images in order to create a smooth transition
As panoramic photography becomes increasingly between neighbors. However, if regions of the scene are
popular, there is a greater need for high-quality not stationary, the overlap image will be slightly
software to automatically create panoramic images. different in each image contributing to the overlap.
Existing algorithms either produce a rough "stitch" that Thus, those regions of the composited image will
cannot deal with common artifacts, or require user contain combinations of pixel values from entirely
input. This paper presents methods for dealing with two different parts of the scene. For example, if a person
artifacts that often occur in practice. Our first moves his head in an area of overlap, the region
contribution is a method for dealing with objects that containing his head in the stitched image will be a
move between different views of a dynamic scene. If combination of head and background from both images.
such moving objects are left in, they will appear blurry It will give the head a ghosted look, not to mention that
and "ghosted". Treating such regions as nodes in a this ghosted head will appear in two locations.
graph, we use a vertex cover algorithm to selectively
remove all but one instance of each object. Our second In order to get around this problem of ghosting, we need
contribution is a method for continuously adjusting to display the stitched image as if nothing in the scene
exposure across multiple images in order to eliminate moved. Thus, when regions of the scene do have
visible shifts in brightness or hue. We compute movement, we would like to use pixel values from only
exposure corrections on a block-by block basis, then one of the contributing images for that region. In order
smoothly interpolate the parameters using a spline to to accomplish this, however, we need to determine a)
get spatially continuous exposure adjustment. Our where the movement occurred, and b) which image to
enhancements, combined with previously published use.
techniques for automatic image stitching, result in a
high-quality automated stitcher that exhibits far fewer Several methods have been proposed to eliminate
artifacts than existing software. ghosts. Shum and Szeliski [10] propose a method for
deghosting small misregistrations based on computing
optic flow and then doing a multi-way morph. Median
filters are often used and are effective when more than
1. Introduction half the images contain consistent pixels [3, 9]. This is
not the case for mosaics created from a relatively small
As panoramic photography becomes increasingly number of images. Davis [2] proposes cutting images
popular, there is a greater need for software to create between regions of movement, finding the best cut with
panoramic images. Ideally, the image stitching process Dijkstra’s algorithm. However, it is not clear how to
should be completely automatic, requiring no user generalize this to mosaics created from many
information in calculating the panorama [1,10,11,12]. overlapping images. We need a novel algorithm to
This not only applies to registering the images, but also tackle the more complicated problem of multiple
to fixing irregularities typical to amateur photography. overlapping regions of movement.
Two such irregularities discussed in this paper are
movement of objects within a scene, and differences in Another problem in automatic image stitching is
exposure between images. exposure differences between images. Exposure
differences are a common occurrence, especially with
One of the problems in automatic image stitching is that digital photographs. If the differences are not corrected,
of de-ghosting. When the images are taken, there is no the panorama will appear to have seams, even when the
guarantee that objects in the image stay stationary from images are blended in overlapping regions. Additionally
one image to the next. This becomes a problem in the our difference-based de-ghosting algorithm could

1
interpret exposure differences as moving objects. Thus,
we must find a way to equalize the exposure of each Note that corresponding RODs are not defined to be the
image based on the information in neighboring images same object in different images. In figure 1a the moving
while retaining local smoothness. face has slight motion. This motion induces the RODs
shown in figure 1b. In figure 1c the face has more
Previous work in this area uses a large number of images motion and does not happen to overlap in overlapping
of the same scene to do a radiometric calibration of the images. In this case the algorithm presented will map
camera [4,5,6]. In our work we don’t assume a the face into separate RODs as shown in figure 1d and
calibration step. In work by Hasler et al [7], the image these corresponding RODs will be handled separately.
registration and the camera’s parametric Opto-Electronic
Conversion Function are simultaneously computed for
pairs of images. This assumes a known parametric
model for the camera and it is not clear how to
generalize this to multiple overlapping images. Burt A C
and Adelson [13] use multi-resolution splines to perform B
spatial blending between different images. However,
since their method depends on band-pass image
pyramids, it is not clear how to apply it to the irregularly
shaped images present in general image mosaics.
Furthermore, current stitching techniques [1,10] already
use large "feathering" regions, so multi-resolution (a) – A 3 image mosaic with a moving face
splining may not help. In our work we compute 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
corrections on a block-by-block basis, and then 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
smoothly interpolate the parameters using a spline to get 0000000000000000000000000 0000000000000000000000000 0000000000000000000000000
spatially continuous exposure adjustment.
A B C
2. Ghosting Artifacts
(b) – Corresponding RODs caused by motion in (a)
2.1 Where Does Movement Occur? A C
The first step in our de-ghosting process is to determine B
which regions in the input images are not static and thus
differ across images. We limit the search for regions of
difference (ROD) to the areas of overlap between input
images. To identify RODs, a map is computed for each
input image by flagging pixels which differ by more than
a certain threshold from pixels in overlapping images. (c) – Another 3 image mosaic with a moving face
To smooth the difference maps, a morphological erode
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
and dilate step is then applied. Next, a region extraction
000000000000 000000000000 000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000000000
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000000000
00 00 00 00 00 00 00 00 00 00 00 00
algorithm is applied to identify and label contiguous
regions. Figure 1 depicts the construction of difference
maps for a simple mosaic.
A B C
Our overall goal is to use information from only one (d) – Corresponding RODs caused by motion in (c)
image for each ROD. Thus, we must group
Figure 1
corresponding regions across images, keep information
from one of the images, and ignore corresponding 2.2 Which Image to Use?
information in the other images. But how do we group Assume, for a moment, that RODs are vertices in a
corresponding regions? Not all corresponding regions graph and that corresponding RODs are linked by an
are the same size. Because more than one image may edge. We know by definition that non-corresponding
overlap with a given image, and each area of overlap is RODs will not overlap, and vice versa. Thus, each chain
not the same size or location, an image may have of corresponding regions will be its own graph. The
different sized RODs than its overlapping neighbors’ graph formed by the RODs in Figure 1 is shown in the
Figure 1. We define RODs in different images to be figure below.
corresponding if they have any overlap at all.

2
 Z+ on the vertices, find the cover V' ⊆ V that
minimizes the weight w (V'). The solution of this is
A B C known to be NP complete. In our implementation we
run an exhaustive solver for graphs with 8 or less
vertices and run a randomized approximation algorithm
[8] for larger graphs.

Figure 2 – Graph of corresponding ROD in figure 1b A

C
B
Now, consider the vertex cover of such a graph. The
vertex cover of an undirected graph G = (V, E) is a
subset V' ⊆ V such that each edge in E is incident on at
least one of the vertices in V'. In other words, each pair
of overlapping regions has at least one of the regions in B1 B2
the vertex cover. If we were to identify the vertex cover
and remove these vertices from the graph, we would be
left with a set of vertices that have no edges in common,
A C
i.e. do not overlap. Thus, if we were to remove the
cover, we would be left with a set of non-corresponding
regions. Next if we eliminate the contribution of pixel
data corresponding to the vertices in the cover, we Figure 3 – simple mosaic containing a moving oval
would have no conflicting regions, and no ghosting. shape and its associated ROD graphs.

A problem with this is that regions are removed In mosaics that contain many mutually overlapping
relatively randomly. AB, BC and AC are all minimum images, these graphs may become arbitrarily complex as
solutions to the vertex cover in the graph of Figure 2. To shown in Figure 4. There are several points worth
fix this we use the following idea to help select regions. noting in these graphs. There may be a group of images
To avoid having object discontinuities arising from that do not all overlap with each other, but all overlap
selecting a ROD (vertex) at an image boundary, we give with a ROD of at least one of the images (Figure 4).
RODs (vertices) a weight proportional to its size and Similarly, it is possible to have two RODs from the same
proximity to the center of its image. Larger and more image in the same graph. However, by definition, they
central RODs receive a higher weight. The weight is do not overlap, and thus do not share an edge.
computed by summing the feather weights, described in Therefore, edges connect only regions of different
[10], within the ROD. images.

Figure 3 depicts the advantages of using this heuristic.

In this figure, an oval object is moving during the
acquisition of 3 images that form a mosaic. Two simple C D
graphs arise from this configuration. If we were to solve C D
the basic vertex cover problem, either vertex in either A B
graph would be a reasonable solution. However by
weighting vertices that correspond to more central A B
regions higher and eliminating the minimum weight
vertex from the graph, we choose the region in the center
image. In this case B1 is weighted higher than A and B2 Figure 4 – Mosaic containing moving star shape plus
higher than C, so A and C are the minimum weight associated ROD graph.
vertices to be removed. This avoids the object
discontinuities that would have arisen from selecting The method of minimum weight vertex cover is not
either of the side images. failsafe, however. The solutions to the weighted vertex
cover for the graph in Figure 4 might be CB, ABD or
Thus, we must compute the minimum weight vertex ACD. If CB were chosen as the solution, this would
cover of the graph. The weighted vertex cover problem eliminate pixel data for images C and B in their RODs,
is summarized as follows. Given an undirected graph G thus causing a hole in the mosaic wherever the RODs for
= (V, E) and a positive integer weight function w: V → A or D overlap C and D. In practice however we find

3
that this situation is rare. Due to the high weight placed Second, for each pixel, we blend the results of applying
on large and central RODs the chosen solution is usually the transfer function from neighboring patches. This can
one of ABD or ACD. To eliminate this artifact we be done using bilinear or biquadratic interpolators as
detect when it occurs and in those regions we fall back shown in Equation 1. The resulting pixel values are the
to the fully composited mosaic. same as if we had interpolated a separate function for
each pixel, but since we implement the transfer function
3. Exposure Artifacts using a look-up table, the computational cost is much
smaller.
v =1
When capturing images to create mosaics, automatic y u =1
∑ Cv ( ) ∑ Cu ( ) fm + u , n + v ( p ( x, y ))
x
cameras often change their exposure settings. Even if p ( x, y ) =
v = −1 N u = −1 N
the images are blended in the areas of overlap, if the
exposure difference between images is too high, it will p(x, y) : pixel value at block location x, y
give the panorama a creased look around the edges of N : block size
the overlap. Feathering [10] the effect of the blend into
Cn ( x ) : bilinear or bicubic coefficient
the rest of the image will help remove creases, but the
exposure change will still be too abrupt to appear natural fu , v ( x) : transfer function for block u, v
(Figure 7).

Exposure differences can also confuse the previously Equation 1 – interpolated transfer function
described de-ghosting algorithm, causing it to interpret
differences across images as moving objects. Thus, it is
important to have an exposure compensation step prior Of course, exposure is not the only reason images might
to de-ghosting. Ideally we would do a full radiometric be different. Other effects such as vignetting may also
calibration of the camera; however given limited input cause image differences. An advantage of our method is
data this is infeasible for many mosaics. that rather than attempting to solve a parametric model
for such complex camera characteristics, the mapping of
To eliminate exposure adjustment artifacts in mosaics, pixel values varies smoothly across the image and will
we need to apply a transfer function to an input image to locally compensate for the previously mentioned effects.
make it look more like its neighbors. This becomes
complex as the number of overlapping neighbors
increases. To alleviate this complexity, we have 4. Results
developed a block-based exposure adjustment technique.
This allows us to vary the image’s transfer function as it Figures 5 and 6 show the results of our de-ghosting
overlaps with differently exposed neighbors. algorithm. In both cases, the algorithm was set with the
same parameters and ran completely automatically. In
Each image is divided into blocks. The block size is figure 5 a four image mosaic is shown. Notice how the
variable in our implementation, but we have found that individual images that comprise the mosaic contained
32x32 usually gives good results. Within each block, we moving objects. When a basic image registration and
compute a quadratic transfer function that in a least- blending algorithm [1] is applied (figure 5-A) the
squares sense best maps, in luminance, this image block moving objects appear as “ghosts”. Since they did not
to the composite of images overlapping this block. We remain in the same position between images they are
may then iterate and use the adjusted composite as the blended with the background. Figure 5-B is the result of
reference for the next step. In our implementation we applying our de-ghosting algorithm. Notice that the
find the 3 iterations yields good results. image of the person in the far right was ghosted in the
original while our algorithm removes this ghost in the
As with many block-based algorithms the results of result.
simply applying the transfer function in isolation within
the block are too “blocky” (Figure 7-B). To smooth the In figure 6, the algorithm is applied to a much more
variation in the transfer function distributions, we use a difficult sequence. Notice that many moving objects are
combination of two techniques. First, we average the contained in this scene in which many images have
functions in each patch with those of their neighbors. In significant overlap. Figure 6-A shows the results of a
our implementation, we use an iterated separable kernel
of (¼, ½, ¼), and typically use 2 iterations.

4
 (A) (B)
Figure 5 – (A) Ghosted mosaic. (B) Result of de-ghosting algorithm.

(A)

(B)
Figure 6 – (A) Ghosted mosaic. (B) Result of de-ghosting algorithm.

5
simple registration and blend. Applying our algorithm smoothing of the transfer functions we get the result in
to this mosaic yields the result in Figure 6-B. Again Figure 7-C.
notice the significant reduction in ghosting.
Our final example, shown in Figure 8, contains both
In Figure 7, we demonstrate the results of the exposure exposure differences and moving objects. In the mosaic
compensation algorithm. We used a block size of 32x32 in Figure 8-A notice the sharp transition in the sky
for all exposure compensation results presented here. caused by exposure differences, as well as the people
Two images with significant exposure differences are moving in the scene. Figure 8-B shows the result of
registered and feathered as shown in Figure 7-A. Notice applying exposure compensation followed by the de-
the sharp transition between images in the center of the ghosting algorithm. Notice that both types of artifacts
mosaic. If we simply do block based exposure are removed, resulting in a much better looking mosaic.
adjustment we get the result in 7-B. After applying the

(A) (B)

(C)
Figure 7 – (A) 2 image mosaic with exposure differences. (B) Blocky result before smooth interpolation of transfer
functions (C) Result of the exposure compensation algorithm.

6
 (A)

(B)
Figure 8 (A) Mosaic which contains both ghost and exposure artifacts. (B) Result of applying exposure
compensation followed by ghost removal.

5. Conclusion altering the image in blocks, it is possible to adjust

scenes where a single change in exposure would result in
Panoramic photography presents several problems in an under or overexposed image. In addition, taking into
creating a seamless mosaic. In this paper we have account local and overlap information for every pixel
presented novel algorithms for fixing two such common allows for smooth transitions not only between
problems. In order to remove ghost effects caused by overlapping images, but within an image containing
moving objects, we use a weighted vertex cover multiple regions of overlap.
algorithm. Although there are still cases of duplicate
images and holes, using the minimum cost directly
minimizes these effects. 6. References
In order to obtain a good difference metric for use in de- [1] R.Szeliski and H. Y. Shum., Creating Full View
ghosting, and for creating a smooth, natural looking Panoramic Image Mosaics and Environment Maps,
image, exposure adjustment was also needed. We Proc. SIGGRAPH'97 pg. 251-258
presented an algorithm of block-based adjustment, [2] J. Davis, Mosaics of Scenes with Moving Objects,
which alters the pixel values using a weighted average of IEEE Computer Society Conference on Computer
lookup tables from nearby parts of the image. By Vision and Pattern Recognition (CVPR'98) pg. 354-360

7
[3] T. Echigo, R. Radke, P. Ramadge, H. Miyamori, S. [9] M. Irani, P. Anandan, J. Bergen, R. Kumar, S. Hsu,
Iisaku, Ghost Error Elimination and Superimposition of Efficient Representations of Video Sequences and Their
Moving Objects in Video Mosaicing, IEEE International Applications, Signal Processing: Image Communication
Conference on Image Processing 1999 Volume 8, 1996 pg. 327-351
[4] T. Mitsunaga, S. Nayar, Radiometric Self [10] H.-Y. Shum, R. Szeliski, Construction of
Calibration, IEEE Computer Society Conference on Panoramic Image Mosaics with Global and Local
Computer Vision and Pattern Recognition (CVPR'99) Alignment, International Journal of Computer Vision,
pg. 374-380 Vol. 36(2), pp. 101-130, Feb. 2000
[5] P. Debevec, J. Malik, Recovering High Dynamic [11] F. Dufaux, F. Moscheni, Background Mosaicking
Range Radiance Maps from Photographs, Proc. for Low Bit Rate Video Coding, IEEE International
SIGGRAPH'97 pg. 369-378 Conference on Image Processing 1996
[6] S. Mann, R. Piccard, Being ‘Undigital’ with Digital [12] H. Sawhney, S. Ayer, Compact representations of
Cameras: Extending the Dynamic Range by Combining videos through dominant and multiple motion
Differently Exposed Pictures , Proc. Of ISTs 48th estimation, IEEE Trans. Pattern Analysis and Machine
Annual Conf. pg. 422-428, May 1995 Intelligence, pp. 814--830, Aug. 1996.
[7] D. Hasler, S. Süsstrunk, Colour Handling in [13] P. J. Burt and E. H. Adelson, A multiresolution
panoramic photography, Proceedings IS&T/SPIE spline with applications to image mosaics, ACM
Electronic Imaging 2001: Videometrics and Optical Transactions on Graphics, 2(4), pp. 217-236, October
Methods for 3D Shape Measurements, Vol. 4309 1983.
[8] R. Motwani, Lecture Notes on Approximation
Algorithms Volume I, Technical Report Dept. of CS,
Stanford, 1992