Fast Fourier Color Constancy

Jonathan T. Barron Yun-Ta Tsai

[email protected] [email protected]

Abstract based techniques in computer vision, both problems reduce

to just estimating the “best” illuminant from an image, and
We present Fast Fourier Color Constancy (FFCC), a the question of whether that illuminant is objectively true
color constancy algorithm which solves illuminant esti- or subjectively attractive is just a matter of the data used
mation by reducing it to a spatial localization task on a during training.
torus. By operating in the frequency domain, FFCC pro- Despite their accuracy, modern learning-based color
duces lower error rates than the previous state-of-the-art by constancy algorithms are not immediately suitable as prac-
13 − 20% while being 250 − 3000× faster. This unconven- tical white balance algorithms, as practical white balance
tional approach introduces challenges regarding aliasing, has several requirements besides accuracy:
directional statistics, and preconditioning, which we ad- Speed - An algorithm running in a camera’s viewfinder
dress. By producing a complete posterior distribution over must run at 30 FPS on mobile hardware. But a camera’s
illuminants instead of a single illuminant estimate, FFCC compute budget is precious: demosaicing, face detection,
enables better training techniques, an effective temporal auto exposure, etc, must also run simultaneously and in real
smoothing technique, and richer methods for error analy- time. Spending more than a small fraction (say, 5 − 10%)
sis. Our implementation of FFCC runs at ∼ 700 frames per of a camera’s compute budget on white balance is impracti-
second on a mobile device, allowing it to be used as an ac- cal, suggesting that our speed requirement is closer to 1 − 5
curate, real-time, temporally-coherent automatic white bal- milliseconds per frame.
ance algorithm. Impoverished Input - Most color constancy algorithms are
designed for full resolution, high bit-depth input images, but
operating on such large images is challenging and costly in
1. Intro practice. To be fast, the algorithm must work well on the
small, low bit-depth “preview” images (32 × 24 or 64 × 48
A fundamental problem in computer vision is that of esti-
pixels, 8-bit) which are usually computed by specialized
mating the underlying world that resulted in some observed
camera hardware for this task.
image [1, 5]. One subset of this problem is color constancy:
Uncertainty - In addition to the illuminant, the algorithm
estimating the color of the illuminant of the scene and the
should produce some confidence measure or a complete
colors of the objects in the scene viewed under a white light.
posterior distribution over illuminants, thereby enabling
Despite its apparent simplicity, this problem has yielded a
convenient downstream integration with hand-engineered
great deal of depth and challenge for both the human vision
heuristics or external sources of information.
and computer vision communities [17, 22]. Color constancy
Temporal Coherence - The algorithm should allow the es-
is also a practical concern in the camera industry: produc-
timated illuminant to be smoothed over time, to prevent
ing a natural looking photograph without user intervention
color composition in videos from varying erratically.
requires that the illuminant be automatically estimated and
discounted, a process referred to as “auto white balance” In this paper we present a novel color constancy al-
among practitioners. Though there is a profound histori- gorithm, which we call “Fast Fourier Color Constancy”
cal connection between color constancy and consumer pho- (FFCC). Viewed as a color constancy algorithm, FFCC is
tography (exemplified by Edwin Land, the inventor of both 13 − 20% more accurate than the state of the art on stan-
Retinex theory [26] and the Polaroid instant camera), “color dard benchmarks. Viewed as a prospective white balance al-
constancy” and “white balance” have come to mean differ- gorithm, FFCC addresses our previously described require-
ent things — color constancy aims to recover the veridical ments: Our technique is 250−3000× faster than the state of
world behind an image, while white balance aims to give an the art, and is capable of running at 1.44 milliseconds per
image a pleasant appearance consistent with some aesthetic frame on a standard consumer mobile platform using the
or cultural norm. But with the current ubiquity of learning- thumbnail images already produced by that camera’s hard-

1
 it also raises new problems: 1) pixel values are corrupted
with superimposed shapes that make detection difficult, 2)
detections must “wrap” around the edges of this toroidal
image, and 3) instead of an absolute, global location we can
only recover an aliased, incomplete location. FFCC works
by taking the large convolutional problem of CCC (ie, face
detection on A) and aliasing that problem down to a smaller
size where it can be solved efficiently (ie, face detection
on B). We will show that we can learn an effective color
constancy model in the face of the difficulty and ambiguity
(a) Image A (b) Aliased Image B
introduced by aliasing. This convolutional classifier will be
Figure 1: CCC [4] reduces color constancy to a 2D local- implemented and learned using FFTs, because the naturally
ization problem similar to object detection (1a). FFCC re- periodic nature of FFT convolutions resolves the problem of
peatedly wraps this 2D localization problem around a small detections “wrapping” around the edge of toroidal images,
torus (1b), which creates challenges but allows for faster and produces a significant speedup.
illuminant estimation. See the text for details. Our approach to color constancy introduces a number
of issues. The aforementioned periodic ambiguity result-
ing from operating on a torus (which we dub “illuminant
ware. FFCC produces a complete posterior distribution over aliasing”) requires new techniques for recovering a global
illuminants which allows us to reason about uncertainty and illuminant estimate from an aliased estimate (Section 3).
enables simple and effective temporal smoothing. Localizing the centroid of the illuminant on a torus is dif-
We build on the “Convolutional Color Constancy“ ficult, requiring that we adopt and extend techniques from
(CCC) approach of [4], which is currently one of the top- the directional statistics literature (Section 4). But our ap-
performing techniques on standard color constancy bench- proach presents a number of benefits. FFCC improves accu-
marks [12, 20, 30]. CCC works by observing that applying racy relative to CCC by 17 − 24% while retaining its flex-
a per-channel gain to a linear RGB image is equivalent to ibility, and allows us to construct priors over illuminants
inducing a 2D translation of the log-chroma histogram of (Section 5). By learning in the frequency-domain we can
that image, which allows color constancy to be reduced to construct a novel method for fast frequency-domain regu-
the task of localizing a signature in log-chroma histogram larization and preconditioning, making FFCC training 20×
space. This reduction is at the core of the success of CCC faster than CCC (Section 6). Our model produces a com-
and, by extension, our FFCC technique; see [4] for a thor- plete unimodal posterior over illuminants as output, allow-
ough explanation. The primary difference between FFCC ing us to construct a Kalman filter-like approach for pro-
is that instead of performing an expensive localization on a cessing videos instead of independent images (Section 7).
large log-chroma plane, we perform a cheap localization on
a small log-chroma torus. 2. Convolutional Color Constancy
At a high level, CCC reduces color constancy to object
detection — in the computability theory sense of “reduce”. Let us review the assumptions made in CCC and inher-
FFCC reduces color constancy to localization on a torus in- ited by our model. Assume that we have a photometrically
stead of a plane, and because this task has no intuitive ana- linear input image I from a camera, with a black level of
logue in computer vision we will attempt to provide one1 . zero and with no saturated pixels2 . Each pixel k’s RGB
Given a large image A on which we would like to perform value in image I is assumed to be the product of that pixel’s
object detection, imagine constructing a smaller n × n im- “true” white-balanced RGB value W (k) and some global
age B in which each pixel in B is the sum of all values in A RGB illumination L shared by all pixels:
separated by a multiple of n pixels in either dimension:  (k)   (k)   
X Ir Wr Lr
B(i, j) = A(i + nk, j + nl) (1) ∀k Ig(k)  = Wg(k)  ◦ Lg  (2)
   
k,l (k) (k) L
Ib Wb b

This amounts to taking A and repeatedly wrapping it around

The task of color constancy is to use the input image I to
a small torus (see Figure 1). Detecting objects this way may
estimate L, and with that produce W (k) = I (k) /L.
yield a speedup as the image being processed is smaller, but
Given a pixel from our input RGB image I (k) , CCC de-
1 We cannot speak to the merit of this idea in the context of object

detection, and we present it here solely to provide an intuition of our work 2 in practice, saturated pixels are identified and removed from all down-

on color constancy stream computation, similarly to how color checker pixels are ignored.
 (a) Input Image (b) Histogram (c) Aliased Histogram (d) Aliased Prediction (e) De-aliased Prediction (f) Output Image

Figure 2: An overview of our pipeline demonstrating the problem of illuminant aliasing. Similarly to CCC, we take an input
image (2a) and transform it into a log-chroma histogram (2b, presented in the same format as in [4]). But unlike CCC, our
histograms are small and toroidal, meaning that pixels can “wrap around” the edges (2c, with the torus “unwrapped” once
in every direction). This means that the centroid of a filtered histogram, which would simply be the illuminant estimate in
CCC, is instead an infinite family of possible illuminants (2d). This requires de-aliasing, some technique for disambiguating
between illuminants to select the single most likely estimate (2e, shown as a point surrounded by an ellipse visualizing the
output covariance of our model). Our model’s output (u, v) coordinates in this de-aliased log-chroma space corresponds to
the color of the illuminant, which can then be divided into the input image to produce a white balanced image (2f).

fines two log-chroma measures: with the properties of natural images has a significant effect,
    as we will demonstrate.
(k)
u(k) = log Ig(k) /Ir(k) v (k) = log Ig(k) /Ib (3) Similarly to CCC, given an input image I we construct a
histogram N from I, where N (i, j) is the number of pixels
The absolute scale of L is assumed to be unrecoverable, so in I whose log-chroma is near the (u, v) coordinates corre-
estimating L simply requires estimating its log-chroma: sponding to histogram position (i, j):
Lu = log (Lg /Lr ) Lv = log (Lg /Lb ) (4)
u(k) − ulo
X  
N (i, j) = mod − i, n < 1
After recovering (Lu , Lv ), assuming that L has a magnitude h
k
of 1 lets us recover the RGB values of the illuminant:  (k)  !
v − vlo
exp(−Lu ) 1 exp(−Lv ) ∧ mod − j, n < 1 (6)
Lr = Lg = Lb = h
p z z z
2 2
z = exp(−Lu ) + exp(−Lv ) + 1 (5) Where i, j are 0-indexed, n = 64 is the number of bins,
h = 1/32 is the bin size, and (ulo , vlo ) is the starting
Framing color constancy in terms of predicting log-chroma
point of the histogram. Because our histogram is too small
has several small advantages over the standard RGB ap-
to contain the wide spread of colors present in most nat-
proach (2 unknowns instead of 3, better numerical stability,
ural images, we use modular arithmetic to cause pixels to
etc) but the primary advantage of this approach is that us-
“wrap around” with respect to log-chroma (any other stan-
ing log-chroma turns the multiplicative constraint relating
dard boundary condition would violate our convolutional
W and I into an additive constraint [15], and this in turn
assumption and would cause many image pixels to be ig-
enables a convolutional approach to color constancy. As
nored). This means that, unlike standard CCC, a single
shown in [4], color constancy can be framed as a 2D spatial
(i, j) coordinate in the histogram no longer corresponds to
localization task on a log-chroma histogram N , where some
an absolute (u, v) color, but instead corresponds to an infi-
sliding-window classifier is used to filter that histogram and
nite family of (u, v) colors. Accordingly, the centroid of a
the centroid of that filtered histogram is used as the log-
filtered histogram no longer corresponds to the color of the
chroma of the illuminant.
illuminant, but instead is an infinite set of illuminants. We
will refer to this phenomenon as illuminant aliasing. Solv-
3. Illuminant Aliasing
ing this problem requires that we use some technique to de-
We assume the same convolutional premise of CCC, but alias an aliased illuminant estimate3 . A high-level outline of
with one primary difference to improve quality and speed:
3 It is tempting to refer to resolving the illuminant aliasing problem as
we use FFTs to perform the convolution that filters the log-
“anti-aliasing”, but anti-aliasing usually refers to preprocessing a signal to
chroma histogram, and we use a small histogram to make prevent aliasing during some resampling operation, which does not appear
that convolution as fast as possible. This change may seem possible in our framework. “De-aliasing” suggests that we allow aliasing
trivial, but the periodic nature of FFT convolution combined to happen to the input, but then remove the aliasing from the output.
our FFCC pipeline that illustrates illuminant (de-)aliasing of some loss computed on the de-aliased illuminant predic-
can be seen in Fig. 2. tion L̂ back onto the learned filters F . The centroid fitting
De-aliasing requires that we use some external informa- in Eq. 11 is performed by fitting a bivariate von Mises dis-
tion (or some external color constancy algorithm) to dis- tribution to a PDF, which we will now explain.
ambiguate between illuminants. An intuitive approach is
to select the illuminant that causes the average image color 4. Differentiable Bivariate von Mises
to be as neutral as possible, which we call “gray world de-
aliasing”. We compute average log-chroma values (ū, v̄) for Our architecture requires some mechanism for reducing
the entire image and use this to turn an aliased illuminant a toroidal PDF P (i, j) to a single estimate of the illumi-
estimate (L̂u , L̂v ) into a de-aliased illuminant (L̂0u , L̂0v ): nant. Localizing the center of mass of a histogram defined
on a torus is difficult: fitting a bivariate Gaussian may fail
P (k)
P (k)

ū = log ku v̄ = log kv (7) when the input distribution “wraps around” the sides of the
 0    
1 L̂u − ū

1
 PDF, as shown in Fig. 3. Additionally, for the sake of tem-
L̂u L̂
0 = u − (nh) + (8) poral smoothing (Section 7) and confidence estimation, we
L̂v L̂v nh L̂v − v̄ 2
want our model to predict a well-calibrated covariance ma-
Another approach, which we call “gray light de-aliasing”, trix around the center of mass of P . This requires that our
is to assume that the illuminant is as close to the center of model be trained end-to-end, which therefore requires that
the histogram as possible. This de-aliasing approach sim- our mean/covariance fitting be analytically differentiable
ply requires carefully setting the starting point of the his- and therefore usable as a “layer” in our learning architec-
togram (ulo , vlo ) such that the true illuminants in natural ture. To address these problems we present a variant of the
scenes all lie within the span of the histogram, and setting bivariate von Mises distribution [27], which we will use to
L̂0 = L̂. We do this by setting ulo and vlo to maximize efficiently localize the mean and covariance of P in a man-
the distance between the edges of the histogram and the ner that allows for easy backpropagation.
bounding box surrounding the ground-truth illuminants in The bivariate von Mises distribution (BVM) is a com-
the training data4 . Gray light de-aliasing is trivial to imple- mon parameterization of a PDF on a torus. There exist
ment but, unlike gray world de-aliasing, it will systemati- several parametrizations which mostly differ in how “con-
cally fail if the histogram is too small to fit all illuminants centration” is represented (“concentration” having a sim-
within its span. ilar meaning to covariance). All of these parametriza-
To summarize the difference between CCC [4] and our tions present problems in our use case: none have closed
approach with regards to illuminant aliasing, CCC (approx- form expressions for maximum likelihood estimators [24],
imately) performs illuminant estimation as follows: none lend themselves to convenient backpropagation, and
all define concentration in terms of angles and therefore
require “conversion” to covariance matrices during color
     
L̂u u
= lo + h arg max (N ∗ F ) (9) de-aliasing. For these reasons we present an alternative
L̂v vlo i,j
parametrization in which we directly estimate a BVM as
Where N ∗ F is performed using a pyramid convolution. a mean µ and covariance Σ in a simple and differentiable
FFCC corresponds to this procedure: closed form expression. Though necessarily approximate,
our estimator is accurate when the distribution is well-
P ← softmax (N ∗ F ) (10) concentrated, which is generally the case for our task.
(µ, Σ) ← fit bvm(P ) (11) Our input is a PDF P (i, j) of size n × n, where i and
  j are integers in [0, n − 1]. For convenience we define a
L̂u
← de alias(µ) (12) mapping from i or j to angles in [0, 2π) and the marginal
L̂v
distributions of P with respect to i and j:
Where N is a small and aliased toroidal histogram, convolu-
2πi X X
tion is performed with FFTs, and the centroid of the filtered θ(i) = Pi (i) = P (i, j) Pj (j) = P (i, j)
n
histogram is estimated and de-aliased as necessary. By con- j i
structing this pipeline to be differentiable we can train our
model in an end-to-end fashion by propagating the gradients We also define the marginal expectation of the sine and co-
sine of the angle:
4 Our histograms are shifted toward green colors rather than centered

around a neutral color, as cameras are traditionally designed with an more X X

sensitive green channel which enables white balance to be performed by yi = Pi (i) sin(θ(i)) xi = Pi (i) cos(θ(i)) (13)
gaining red and blue up without causing color clipping. Ignoring this prac- i i
tical issue, our approach can be thought of as centering our histograms
around a neutral white light With xj and yj defined similarly.
 Using this loss causes our model to produce a well-
calibrated complete posterior of the illuminant instead of
just a single estimate. This posterior will be useful when
processing video sequences (Section 7) and also allows us
to attach confidence estimates to our predictions using the
entropy of Σ (see the supplement).
Our entire system is trained end-to-end, which requires
Figure 3: We fit a bivariate von Mises distribution (shown that every step in BVM fitting and loss computation be an-
in solid blue) to toroidal PDFs P (i, j) to produce an aliased alytically differentiable. See the supplement for the analyt-
illuminant estimate. Contrast this with fitting a bivariate ical gradients for Eqs. 14, 17, and 18, which can be chained
Gaussian (shown in dashed red) which treats the PDF as together to backpropagate the gradient of f (·) onto the in-
if it lies on a plane. Both approaches behave similarly if put PDF P .
the distribution lies near the center of the unwrapped plane
(left) but fitting a Gaussian fails as the distribution begins to 5. Model Extensions
“wrap around” the edge (middle, right).
The system we have described thus far (compute a peri-
odic histogram of each pixel’s log-chroma, apply a learned
Estimating the mean µ of a BVM from a histogram just FFT convolution, apply a softmax, fit a de-aliased bivariate
requires computing the circular mean in i and j: von Mises distribution) works reasonably well (Model A in
   n

 Table 1) but does not produce state-of-the-art results. This
u mod 2π atan2(yi , xi ), n

µ = lo + h n (14) is likely because this model reasons about pixels indepen-
vlo mod 2π atan2(yj , xj ), n
dently, ignores all spatial information in the image, and does
Eq. 14 includes gray light de-aliasing, though gray world not consider the absolute color of the illuminant. Here we
de-aliasing can also be applied to µ after fitting. present extensions to the model which address these issues
We can fit the covariance of our model by simply “un- and improve accuracy accordingly.
wrapping” the coordinates of the histogram relative to the As explored in [4], a CCC-like model can be generalized
estimated mean and treating these unwrapped coordinates to a set of “augmented” images provided that these images
as though we are fitting a bivariate Gaussian. We define the are non-negative and “scale with intensity” [14]. This lets
“unwrapped” (i, j) coordinates such that the “wrap around” us apply certain filtering operations to image I and, instead
point on the torus lies as far away from the mean as possible, of constructing a single histogram from our image, con-
or equivalently, such that the unwrapped coordinates are as struct a “stack” of histograms constructed from the image
close to the mean as possible: and its filtered versions. Instead of learning and applying
 
µu − ulo

n
 one filter, we learn a stack of filters and sum across chan-
ī = mod i − + ,n nels after convolution. The general family of augmented
h 2
    images used in [4] are expensive to compute, so we instead
µv − vlo n use just the input image I and a local measure of absolute
j̄ = mod j − + ,n (15)
h 2 deviation in the input image:
Our estimated covariance matrix is simply the sample co- 1 1
variance of P (ī, j̄):
X X
1
E(x, y, c) = 8 |I(x, y, c) − I(x + i, y + j, c)| (19)
i=−1 j=−1
X X
E [ī] = Pi (i)ī E [j̄] = Pj (j)j̄ (16)
i j These two features appears to perform similarly to the four
+
 X
2
Pi (i)ī − E [ī]
2
X
P (i, j)īj̄ − E [ī] E [j̄]

features used in [4], while being cheaper to compute.

Σ=h 
 2 i i,j X

 (17) Just as a sliding-window object detector is often invariant
2
X
P (i, j)īj̄ − E [ī] E [j̄] + Pj (j)j̄ 2 − E [j̄]  to the absolute location of an object in an image, the convo-
i,j j
lutional nature of our baseline model makes it invariant to
We regularize the sample covariance matrix slightly by any global shift of the color of the input image. This means
adding a constant  = 1 to the diagonal. that our baseline model cannot rely on any statistical regu-
With our estimated mean and covariance we can com- larities of the illumination by, say, modeling black body ra-
pute our loss: the negative log-likelihood of a Gaussian (ig- diation, the specific properties of commonly manufactured
noring scale factors and constants) relative to the true illu- light bulbs, or any varying spectral sensitivity across cam-
minant L∗ : eras. Though CCC does not model illumination directly,
T it appears to indirectly reason about illumination by using
L∗u
   ∗  
−1 Lu
f (µ, Σ) = log |Σ| + − µ Σ − µ (18) the boundary conditions of its pyramid convolution to learn
L∗v L∗v
 a regularization term g(Z):

Z ∗ = arg min (f (Z) + g (Z)) (21)

Z

We require that the regularization g(Z) is the weighted sum

(a) Pixel Filter (b) Edge Filter (c) Illum. Gain (d) Illum. Bias of squared periodic convolutions of Z with some filter bank.
In our experiments g(Z) is the weighted sum of the squared
Figure 4: A complete learned model (Model J in Table 1)
difference between adjacent values (similar to a total varia-
shown in centered (u, v) log-chroma space, with bright-
tion loss [29]) and the sum of squared values:
ness indicating larger values. Our learned filters are cen-
tered around the origin (the predicted white point) and our
P 2
g(Z) =λ1 i,j (Z (i, j) − Z (mod(i + 1, n), j))
illuminant gain and bias maps model the black body curve 2

and varying camera sensitivity as two wrap-around line seg- + (Z (i, j) − Z (i, mod(j + 1, n)))
+λ0 i,j Z(i, j)2
P
ments (this dataset consists of images from two different (22)
cameras).
Where λ1 and λ0 are hyperparameters that determine the
strength of each smoothness term. We require that λ0 > 0
a model which is not truly spatially varying and is there- to prevent divide-by-zero issues during preconditioning.
fore sensitive to absolute color. Because a torus has no We use a variant of the standard FFT Fv (·) which bi-
boundaries, our model is invariant to global input color, so jectively maps from some real n × n image to a real n2 -
we must therefore introduce a mechanism for directly rea- dimensional vector, instead of the complex n×n image pro-
soning about illuminants. We use a per-illuminant “gain” duced by a standard FFT (See the supplement for a formal
map G(i, j) and “bias” map B(i, j), which together apply description). With this, we can rewrite Eq. 22 as follows:
a per-illuminant affine transformation to the output of our r 
previously-described convolution at (aliased) color (i, j). 1 2 2

w= λ1 |Fv ([1, −1])| + |Fv ([1; −1])| + λ0
The bias B causes our model to prefer certain illuminants n
T 2
over others, while the gain G causes the contribution of the g(Z) = Fv (Z) diag (w) Fv (Z) (23)
convolution at certain colors to be amplified.
Our two extensions (an augmented edge channel and an where the vector w is just some fixed function of the def-
illuminant gain/bias map) let us redefine the P in Eq. 10 as inition of g(Z) and the values of the hyperparameters λ1
! and λ0 . The 2-tap difference filters in Fv ([1, −1]) and
X Fv ([1; −1]) are padded to size (n × n) before the FFT.
P = softmax B + G ◦ (Nk ∗ Fk ) (20) With w we can define a mapping between our 2D image
k
space and a rescaled FFT vector space:
Where {Fk } are the set of learned filters for each augmented
z = w ◦ Fv (Z) (24)
channel’s histogram Nk , G is our learned gain map, and B
is our learned bias map. In practice we actually parametrize Where ◦ is an element-wise product. This mapping lets us
Glog when training and define G = exp(Glog ), which con- rewrite the optimization problem in Eq. 21 as:
straints G to be non-negative. Visualizations of G and B    z  
and our learned filters can be seen in Fig. 4.
 
2
Z ∗ = Fv−1 1
w arg min f Fv−1 + kzk (25)
z w
6. Fourier Regularization and Preconditioning
where Fv−1 (·) is the inverse of Fv (·), and division is
Our learned model weights ({Fk }, G, B) are all peri- element-wise. This reparametrization reduces the compli-
odic n × n images. To improve generalization, we want cated regularization of Z to a simple L2 regularization of z,
these weights to be small and smooth. In this section we which has a preconditioning effect.
present the general form of the regularization used during We use this technique during training to reparameterize
training, and we show how this regularization lets us pre- all model components ({Fk }, G, B) as rescaled FFT vec-
condition the optimization problem solved during training tors, each with their own values for λ0 and λ1 . The ef-
to find lower-cost minima in fewer iterations. Because this fect of this can be seen in Fig. 5, where we show the loss
frequency-domain optimization technique applies generally during our two training stages. We compare against naive
to any optimization problem concerning smooth and peri- time-domain optimization (Eq. 21) and non-preconditioned
odic images, we will describe it in general terms. frequency-domain optimization (Eq. 25 with w = 1). Our
Let us construct an optimization problem with respect to preconditioned reformulation exhibits a significant speedup
a single n × n image Z consisting of a data term f (Z) and and finds minima with lower losses.
 Logistic Loss BVM Loss
it with an zero-mean isotropic Gaussian (encoding our prior
belief that the illuminant may change over time) and then
multiplying that “fuzzed” Gaussian by the observed Gaus-
sian:
  −1 !−1
α 0
Σt+1 = Σt + + Σo (26)
0 α
  −1 !−1
α 0
µt+1 = Σt+1 Σt + µt + Σo µo
0 α

Where α is a parameter that defines the expected vari-

Figure 5: Loss traces for our two stages of training, for ance of the illuminant over time. This update resembles
three fold cross validation (each line represents a fold) on a Kalman filter but with a simplified transition model, no
the Gehler-Shi dataset using LBFGS. Our preconditioned control model, and variable observation noise.
frequency domain optimization produces lower minima at This temporal smoothing is not used in our benchmarks,
greater rates than are achieved by non-preconditioned opti- but its effect can be seen in the supplemental video.
mization in the frequency domain or naive optimization in
the time domain. 8. Results
We evaluate our technique using two standard color con-
For all experiments (excluding our “deep” variants, see stancy datasets: the Gehler-Shi dataset [20, 30] and the
the supplement), training is as follows: All model parame- Cheng et al. dataset [12] (see Tables 1 and 2). For the
ters are initialized to 0, then we have a convex pre-training Gehler-Shi dataset we present several ablations and vari-
step which optimizes Eq. 25 where f (·) is a logistic loss (de- ants of our model to show the effect of each design decision
scribed in the supplement) using LBFGS for 16 iterations, and to investigate trade-offs between speed and accuracy.
and then we optimize Eq. 25 where f (·) is the non-convex Models labeled “full” were run on 384 × 256 16-bit im-
BVM loss in Eq. 18 using LBFGS for 64 iterations. ages, while models labeled “thumb” were run on 48 × 32
8-bit images, which are the kind of images that a practi-
7. Temporal Smoothing cal white-balance system embedded on a hardware device
might use. Models labeled “4 channel” use the four fea-
Color constancy is usually studied in the context of indi- ture channels used in [4], while models labeled “2 chan-
vidual images, which are assumed to be IID. But a practical nel” use the two channels we present in Section 5. We also
white balance algorithm must run on a video sequence, and present models in which we only use the “pixel channel” I
must enforce some temporal smoothing of the predicted il- or the “edge channel” E as input. All models have a his-
luminant to avoid presenting the viewer with an erratically- togram size of n = 64 except for Models K and L where
varying image in the viewfinder. This smoothing cannot n is varied to show the impact of illuminant aliasing. Two
be too aggressive or else the viewfinder may appear unre- models use “gray world” de-aliasing, and the rest use “gray
sponsive when the illumination changes rapidly (a colorful light” de-aliasing. The former seems slightly less effective
light turning on, the camera quickly moving outdoors, etc). than the latter unless chroma histograms are heavily aliased,
Additionally, when faced with multiple valid hypotheses (a which is why we use it in Model K. Model C only has one
blue wall under white light vs a white wall under blue light, training stage that minimizes logistic loss for 64 iterations,
etc) we may want to use earlier images to resolve ambi- thereby removing the BVM fitting from training. Model E
guities. These desiderata of stability, responsiveness, and fixes G(i, j) = 1 and B(i, j) = 0, thereby removing the
robustness are at odds with each other, and so some com- model’s ability to reason about the absolute color of the il-
promise must be struck. luminant. Model B was trained only to minimize the data
Our task of constructing a temporally coherent illumi- term (ie, λ0 = λ1 = 0 in Eq. 22) while Model D uses L2
nant estimate is aided by the probabilistic nature of the out- regularization but not total variation (ie, λ1 = 0 in Eq. 22).
put of our per-frame model, which produces a posterior dis- Models N, O and P are variants of Model J in which, instead
tribution over illuminants parametrized as a bivariate Gaus- of learning a fixed model ({Fk }, G, B) we express those
sian. Let us assume that we have some ongoing estimate model parameters as the output of a small 2-layer neural
of the illuminant and its covariance (µt , Σt ). Given the network. As inputs to this network we use image metadata,
observed mean and covariance (µo , Σo ) provided by our which allows the model to reason about exposure time and
model we update our ongoing estimate by first convolving camera sensor type, and/or a CNN-produced feature vector
 Best Worst Test Train
[34], which allows the model to reason about semantics (see Algorithm Mean Med. Tri.
25% 25%
Avg.
Time Time
the supplement for details). For each experiment we tune all Support Vector Regression [18]
White-Patch [8]
8.08
7.55
6.73
5.68
7.19
6.35
3.35
1.45
14.89
16.12
7.21
5.76
-
0.16
-
-
λ hyperparameters to minimize the “average” error during Grey-world [9]
Edge-based Gamut [23]
6.36
6.52
6.28
5.04
6.28
5.43
2.33
1.90
10.58
13.58
5.73
5.40
0.15
3.6
-
1986
cross-validation, using cyclic coordinate descent. 1st-order Gray-Edge [32] 5.33 4.52 4.73 1.86 10.03 4.63 1.1 -
2nd-order Gray-Edge [32] 5.13 4.44 4.62 2.11 9.26 4.60 1.3 -
Model P achieves the lowest-error results, with a 20% re- Shades-of-Gray [16] 4.93 4.01 4.23 1.14 10.20 3.96 0.47 -
Bayesian [20] 4.82 3.46 3.88 1.26 10.49 3.86 97 764
duction in error on Gehler-Shi compared to the previously Yang et al. 2015 [35] 4.60 3.10 - - - - 0.88 -
General Gray-World [3] 4.66 3.48 3.81 1.00 10.09 3.62 0.91 -
best-performing published technique. This improvement in Natural Image Statistics [21] 4.19 3.13 3.45 1.00 9.22 3.34 1.5 10749
CART-based Combination [6] 3.90 2.91 3.21 1.02 8.27 3.14 - -
accuracy also comes with a significant speedup compared to Spatio-spectral Statistics [11] 3.59 2.96 3.10 0.95 7.61 2.99 6.9 3159
previous techniques: ∼ 30 ms/image for most models, com- LSRS [19]
Interesection-based Gamut [23]
3.31
4.20
2.80
2.39
2.87
2.93
1.14
0.51
6.39
10.70
2.87
2.76
2.6
-
1345
-
pared to the 520 ms of CCC [4] or the 3 seconds (on a GPU) Pixels-based Gamut [23]
Bottom-up+Top-down [33]
4.20
3.48
2.33
2.47
2.91
2.61
0.50
0.84
10.72
8.01
2.73
2.73
-
-
-
-
of Shi et al. [31]. Model Q (our fastest model) has an accu- Cheng et al. 2014 [12] 3.52 2.14 2.47 0.50 8.74 2.41 0.24 -
Exemplar-based [25] 2.89 2.27 2.42 0.82 5.97 2.39 - -
racy comparable to [4] and [31] but takes only 1.1 millisec- Bianco et al. 2015 [7] 2.63 1.98 - - - - - -
Corrected-Moment [14] 2.86 2.04 2.22 0.70 6.34 2.25 0.77 584
onds to process an image, making it hundreds or millions Chakrabarti et al. 2015 [10] 2.56 1.67 1.89 0.52 6.07 1.91 0.30 -
Cheng et al. 2015 [13] 2.42 1.65 1.75 0.38 5.87 1.73 0.25 245
of times faster than the current state-of-the art. Addition- CCC [4] 1.95 1.22 1.38 0.35 4.76 1.40 0.52 2168
Shi et al. 2016 [31] 1.90 1.12 1.33 0.31 4.84 1.34 3.0 -
ally, our model appears to be faster to train than the state- A) FFCC - full, pixel channel only, no illum. 2.88 1.90 2.05 0.50 6.98 2.08 0.0076 117
of-the-art, though training times for prior work are often not B) FFCC - full 2 channels, no regularization
C) FFCC - full 2 channels, no BVM loss
2.34
2.16
1.33
1.45
1.55
1.56
0.51
0.76
5.84
4.84
1.70
1.78
0.031
0.031
96
62
available. All runtimes in Table 1 for our model were com- D) FFCC - full 2 channels, no total variation
E) FFCC - full, 2 channels, no illuminant
1.92
2.14
1.11
1.34
1.27
1.52
0.28
0.37
4.89
5.27
1.30
1.53
0.028
0.031
104
94
puted on an Intel Xeon CPU E5-2680. Runtimes for the F) FFCC - full, pixel channel only 2.15 1.33 1.51 0.34 5.35 1.51 0.0063 67
G) FFCC - full, edge channel only 2.02 1.25 1.39 0.34 5.11 1.44 0.026 94
“full” model were produced using a Matlab implementa- H) FFCC - full, 2 channels, no precond. 2.91 1.99 2.23 0.57 6.74 2.18 0.025 152
I) FFCC - full, 2 channels, gray world 1.79 1.01 1.22 0.29 4.54 1.24 0.029 98
tion, while runtimes for the “thumb” model were produced J) FFCC - full, 2 channels 1.80 0.95 1.18 0.27 4.65 1.20 0.029 98
K) FFCC - full, 4 channels, n = 32, gray world 2.69 1.31 1.49 0.37 7.48 1.70 0.068 138
using a Halide [28] CPU implementation (our Matlab im- L) FFCC - full, 4 channels, n = 256 1.78 1.05 1.19 0.27 4.46 1.22 0.068 395
M) FFCC - full, 4 channels 1.78 0.96 1.14 0.29 4.62 1.21 0.070 96
plementation of Model Q takes 2.37 ms/image). Runtimes N) FFCC - full, 2 channels, +semantics[34] 1.67 0.96 1.13 0.26 4.23 1.15 - -
for our “+semantic” models are not presented as we were O) FFCC - full, 2 channels, +metadata
P) FFCC - full, 2 channels, +metadata +semantics[34]
1.65
1.61
0.86
0.86
1.07
1.02
0.24
0.23
4.44
4.27
1.10
1.07
0.036
-
143
-
unable to profile [34] accurately (CNN feature computation Q) FFCC - thumb, 2 channels 2.01 1.13 1.38 0.30 5.14 1.37 0.0011 73

appears to dominate runtime).

To demonstrate that our model is a viable automatic Table 1: Performance on the Gehler-Shi dataset [20, 30].
white balance system for consumer photography, we ran our We present five error metrics and their average (the geomet-
Halide code on a 2016 Google Pixel XL using the thumb- ric mean) with the lowest error per metric highlighted in
nail images computed by the device’s camera stack. This yellow. We present the time (in seconds) for training each
implementation ran at 1.44ms per image, which is equiva- model and for evaluating a single image, when available.
lent to 30 frames per second using < 5% of the total com-
pute budget, thereby satisfying our previously-stated speed Algorithm Mean Med. Tri.
Best Worst
Avg.
25% 25%
requirements. A video of our system running in real-time
White-Patch [8] 9.91 7.44 8.78 1.44 21.27 7.24
on a phone can be found in the supplement. Pixels-based Gamut [23] 5.27 4.26 4.45 1.28 11.16 4.27
Grey-world [9] 4.59 3.46 3.81 1.16 9.85 3.70
Edge-based Gamut [23] 4.40 3.30 3.45 0.99 9.83 3.45
9. Conclusion Shades-of-Gray [16] 3.67 2.94 3.03 0.98 7.75 3.01
Natural Image Statistics [21] 3.45 2.88 2.95 0.83 7.18 2.81
We have presented FFCC, a color constancy algorithm Local Surface Reflectance Statistics [19] 3.45 2.51 2.70 0.98 7.32 2.79
that produces a 13 − 20% reduction in error and a 250 − 2nd-order Gray-Edge [32] 3.36 2.70 2.80 0.89 7.14 2.76
1st-order Gray-Edge [32] 3.35 2.58 2.76 0.79 7.18 2.67
3000× speedup relative to prior work. In doing so we have Bayesian [20] 3.50 2.36 2.57 0.78 8.02 2.66
introduced the concept of convolutional color constancy on General Gray-World [3] 3.20 2.56 2.68 0.85 6.68 2.63
a torus, and we have introduced techniques for illuminant Spatio-spectral Statistics [11] 3.06 2.58 2.74 0.87 6.17 2.59
Bright-and-dark Colors PCA [12] 2.93 2.33 2.42 0.78 6.13 2.40
de-aliasing and differentiable bivariate von Mises fitting re- Corrected-Moment [14] 2.95 2.05 2.16 0.59 6.89 2.21
quired for this toroidal approach. We have also presented a Color Dog [2] 2.83 1.77 2.03 0.48 7.04 2.03
novel technique for fast Fourier-domain optimization sub- Shi et al. 2016 [31] 2.24 1.46 1.68 0.48 6.08 1.74
CCC [4] 2.38 1.48 1.69 0.45 5.85 1.74
ject to a certain family of regularizers. FFCC produces a Cheng 2015 [13] 2.18 1.48 1.64 0.46 5.03 1.65
complete posterior distribution over illuminants, which lets M) FFCC - full, 4 channels 1.99 1.31 1.43 0.35 4.75 1.44
Q) FFCC - thumb, 2 channels 2.06 1.39 1.53 0.39 4.80 1.53
us assess the model’s confidence and also enables a Kalman
filter-like temporal smoothing model. FFCC’s speed, ac-
curacy, and temporal consistency allows it to be used for Table 2: Performance on the dataset from Cheng et al.[12],
real-time white balance on a consumer camera. in the same format as Table 1, excluding runtimes. As was
done in [4] we present the average performance (the geo-
metric mean) over all 8 cameras in the dataset.
References [26] E. H. Land and J. J. McCann. Lightness and retinex theory.
JOSA, 1971.
[1] E. H. Adelson and A. P. Pentland. The perception of shading
[27] K. V. Mardia. Statistics of directional data. Journal of the
and reflectance. Perception As Bayesian Inference, 1996.
Royal Statistical Society, Series B, 1975.
[2] N. Banic and S. Loncaric. Color dog - guiding the global
[28] J. Ragan-Kelley, A. Adams, S. Paris, M. Levoy, S. Amaras-
illumination estimation to better accuracy. VISAPP, 2015.
inghe, and F. Durand. Decoupling algorithms from schedules
[3] K. Barnard, L. Martin, A. Coath, and B. Funt. A compari- for easy optimization of image processing pipelines. SIG-
son of computational color constancy algorithms — part 2: GRAPH, 2012.
Experiments with image data. TIP, 2002.
[29] L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total varia-
[4] J. T. Barron. Convolutional color constancy. ICCV, 2015.
tion based noise removal algorithms. Physica D: Nonlinear
[5] H. G. Barrow and J. M. Tenenbaum. Recovering Intrinsic Phenomena, 1992.
Scene Characteristics from Images. Academic Press, 1978.
[30] L. Shi and B. Funt. Re-processed version of
[6] S. Bianco, G. Ciocca, C. Cusano, and R. Schettini. Auto- the gehler color constancy dataset of 568 images.
matic color constancy algorithm selection and combination. http://www.cs.sfu.ca/ colour/data/.
Pattern Recognition, 2010.
[31] W. Shi, C. C. Loy, and X. Tang. Deep specialized network
[7] S. Bianco, C. Cusano, and R. Schettini. Color constancy for illuminant estimation. ECCV, 2016.
using cnns. CVPR Workshops, 2015.
[32] J. van de Weijer, T. Gevers, and A. Gijsenij. Edge-based
[8] D. H. Brainard and B. A. Wandell. Analysis of the retinex color constancy. TIP, 2007.
theory of color vision. JOSA A, 1986.
[33] J. van de Weijer, C. Schmid, and J. Verbeek. Using high-level
[9] G. Buchsbaum. A spatial processor model for object colour
visual information for color constancy. ICCV, 2007.
perception. Journal of the Franklin Institute, 1980.
[34] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang,
[10] A. Chakrabarti. Color constancy by learning to predict chro-
J. Philbin, B. Chen, and Y. Wu. Learning fine-grained im-
maticity from luminance. NIPS, 2015.
age similarity with deep ranking. CVPR, 2014.
[11] A. Chakrabarti, K. Hirakawa, and T. Zickler. Color con-
[35] K.-F. Yang, S.-B. Gao, and Y.-J. Li. Efficient illuminant es-
stancy with spatio-spectral statistics. TPAMI, 2012.
timation for color constancy using grey pixels. CVPR, 2015.
[12] D. Cheng, D. K. Prasad, and M. S. Brown. Illuminant es-
timation for color constancy: why spatial-domain methods
work and the role of the color distribution. JOSA A, 2014.
[13] D. Cheng, B. Price, S. Cohen, and M. S. Brown. Effective
learning-based illuminant estimation using simple features.
CVPR, 2015.
[14] G. D. Finlayson. Corrected-moment illuminant estimation.
ICCV, 2013.
[15] G. D. Finlayson and S. D. Hordley. Color constancy at a
pixel. JOSA A, 2001.
[16] G. D. Finlayson and E. Trezzi. Shades of gray and colour
constancy. Color Imaging Conference, 2004.
[17] D. H. Foster. Color constancy. Vision research, 2011.
[18] B. V. Funt and W. Xiong. Estimating illumination chromatic-
ity via support vector regression. Color Imaging Conference,
2004.
[19] S. Gao, W. Han, K. Yang, C. Li, and Y. Li. Efficient color
constancy with local surface reflectance statistics. ECCV,
2014.
[20] P. Gehler, C. Rother, A. Blake, T. Minka, and T. Sharp.
Bayesian color constancy revisited. CVPR, 2008.
[21] A. Gijsenij and T. Gevers. Color constancy using natural
image statistics and scene semantics. TPAMI, 2011.
[22] A. Gijsenij, T. Gevers, and J. van de Weijer. Computational
color constancy: Survey and experiments. TIP, 2011.
[23] A. Gijsenij, T. Gevers, and J. vande Weijer. Generalized
gamut mapping using image derivative structures for color
constancy. IJCV, 2010.
[24] T. Hamelryck, K. Mardia, and J. Ferkinghoff-Borg. Bayesian
methods in structural bioinformatics. Springer, 2012.
[25] H. R. V. Joze and M. S. Drew. Exemplar-based color con-
stancy and multiple illumination. TPAMI, 2014.