Computer Vision

Spring 2006 15-385,-685

Instructor: S. Narasimhan

Wean 5403
T-R 3:00pm – 4:20pm
 Lightness and Retinex:
An Early Vision Problem

Lecture #10

Readings: Horn Chapter 9.

Vision (Marr): pgs 250-258.
“The perception of Surface Blacks and Whites”, A. L. Gilchrist, Scientific American, 240 (pgs
112-114), 1979
Webpages of Prof. Edward Adelson, MIT
Checker Shadow Illusion

[E. H. Adelson]
Checker Shadow Illusion

[E. H. Adelson]
 Land’s Experiment (1959)

• Cover all patches except a blue rectangle

• Make it look gray by changing illumination
• Uncover the other patches

Color Constancy
We filter out illumination variations
 Color Constancy in Gold Fish

In David Ingle's experiment, a goldfish has been trained to swim to a patch of a given
color for a reward—a piece of liver. It swims to the green patch regardless of the
exact setting of the three projectors' intensities. The behavior is strikingly similar to
the perceptual result in humans.
http://neuro.med.harvard.edu/site/dh/b45.htm
Color Cube Illusion

D. Purves, R. Beau Lotto, S. Nundy “Why We See What We do,” American Scientist
 Lightness Recovery and Retinex Theory

• Problem: Recover surface reflectance / color in

varying illumination conditions.

• We use tools developed before:

- Sensing: Intensity / Color

- Image Processing: Fourier Transform and Convolution

- Edge Operators

- Iterative Techniques
 Image Brightness (Intensity)
Scene
e' (  ) Irradiance

b' q ( )
r ' ( )
Image p ' ( ) Scene
Intensity Reflectance

• Monochromatic Light : (  i )
b ' ( x, y )  r ' ( x, y ) e' ( x, y ) q (i )  1

NOTE: The analysis can be applied to COLORED LIGHT using FILTERS

 Recovering Lightness
• Image Intensity: b ' ( x, y )  r ' ( x, y ) e' ( x, y )
Can we recover e' and r ' from b' ?

• Assumptions: - Sharp changes in Reflectance

- Smooth changes in Illumination

• Frequency spectrum (Fourier transform)

 
B' R' E'
We want to filter out e’
 Recovering Lightness
• Image Intensity: b' ( x, y )  r ' ( x, y ) e' ( x, y )

• Take Logarithm: log b' ( x, y )  log r ' ( x, y )  log e' ( x, y )

OR b ( x, y )  r ( x , y )  e ( x, y )

• Use Laplacian:
 2
 2

d   2b   2 r   2 e 2  2  2
x y
• Sharp changes in reflectance r '  x, y 
 2 r has 2 infinite spikes near edges and  2 r  0 elsewhere
• Smooth changes in illumination
e'  x , y 
 2e  0 everywhere
Lightness Recovery (Retinex Scheme)

Image:
b  re

Laplacian :
d   2b   2 r   2 e

Thresholding :
t  T  d   2r

Reconstruction :
l  x, y 
l  x, y   k  r  x, y 
(lightness) (reflectance)
 Solving the Inverse Problem
Image Laplacian Thresholding Lightness
b  r e d   2b   2 r   2 e t  Td   r 2
l  x, y 

Find lightness l(x,y) from t(x,y):

Poisson’s Equation
 2l  t
 2  
2
 2  2 l  x, y   t  x, y 
 x y 
We have to find g(x,y) which satisfies
 
l  x, y     t  u, v  g  x  u, y  v  dudv
 

l  x, y   t  x, y   g  x, y 
 Solving Poisson’s Equation
We have
 2l  x, y   t  x, y  and l  x, y   t  x, y   g  x, y 

Fourier transform

?  T  u, v  and L u , v   T  u , v  G  u , v 

So
 
 u 2  v 2 L u, v   T  u, v  and L u , v   T  u , v  G  u , v 

Thus
1 1
G u, v    2 2 g  x, y   log x 2  y 2  c
u v 2
 Lightness Recovery

Log of image
l  x, y 
t  x, y 
d  x, y  tg g  x, y  
1
log x 2  y 2
b x, y  Td 2
b 2

l  x , y   k  r  x, y 
(lightness) (reflectance)

Which means: l '  x, y   kr '  x, y 

Normalize:
Assume maximum value of l '  x, y   l 'max corresponds to r '  1
Then: l '  x, y  b '  x, y 
r '  x, y   e '  x, y  
(reflectance) lmax (illumination) r '  x, y 
Computing Lightness (Discrete Case)

Log of image

l  x, y 
t  x, y  1
d  x, y  tg g  x, y   log x 2  y 2
b x, y  Td 2
b2

1 4 1 Basically, inverse of Laplacian mask

1
4 -20 4
But, discrete approximation of the inverse
6 1 4 1 Laplacian would not be sufficiently accurate!

Solve  l  t directly
2
 Computing Lightness (Discrete Case)

T 
1 4 1 1 4 1
1 1
4 -20 4
 l t 2
6
4 -20 4 b
6 1 4 1 1 4 1

 20li , j  4 li 1, j  li , j 1  li 1, j  li , j 1    li 1, j 1  li 1, j 1  li 1, j 1  li 1, j 1   6ti , j

Solve iteratively:

n 1
li, j
1 n
5
 n n n

 l i1, j  l i , j 1  l i1, j  l i , j 1 
1 n
20
 n n n
 3
l i1, j1  l i1, j1  l i1, j1  l i1, j1  ti , j
10

Use a discrete approximation to the inverse of Laplacian to

obtain initial estimate of l
 Lightness from Multiple Images taken
under Varying Illumination

Illumination is not smooth

Use spatial statistics of edges
Derivative operator responses are sparse
 b  x , y , t   r  x, y   e x , y , t 
r  x, y   median b x, y, t  
t Y. Weiss ICCV 2001
Lightness from Multiple Images taken
under Varying Illumination

Y. Weiss ICCV 2001

Lightness from Multiple Images taken
under Varying Illumination

Y. Weiss ICCV 2001

Using Lightness

w/o decomposition w/ decomposition

Y. Weiss ICCV 2001

Using Lightness

Y. Matsushita and K. Nishino CVPR 2003

 Comments on Retinex Theory

• Not applicable to smooth reflectance variations

• Not applicable to curved objects

In general:
Intensity=f( Shape, Reflectance, Illumination )

For very good illusions, see:

http://web.mit.edu/persci/gaz/gaz-teaching/index.html

http://www.michaelbach.de/ot/index.html
 Tricking the Human Eye

Eye Tremors (~ 35 cycles/second)

Tremors used for edge detection

Edges disappear when edge motion is synchronous with Tremors!

 Next Class

• Surface Reflectance and BRDF

• Reading: Horn, Chapter 10.
• F. E. Nicodemus, J.C. Richmond and J.J. Hsia,
“Geometrical Considerations and Nomenclature for
Reflectance”, Institute of Basic Standards, National
Bureau of Standards, October 1977