COMP/ECE/ENGI: COMPUTER

VISION

Lecture 3: Human Vision, Color Spaces and Transforms

COMPUTER VISION www.mun.ca

Perception - from light to sight

by Ted Adelson

Readings
• Michael Brown, ICCV19 color tutorial
• Radiolab podcast on color
Properties of light
Today
• What is light?
• How do we perceive it?
What is light?
Electromagnetic radiation (EMR) moving along rays in space
• R(λ) is EMR, measured in units of power (watts)
– λ is wavelength

Plenoptic Function
• We can describe all of the light in the scene by specifying the radiation
(or “radiance” along all light rays) arriving at every point in space and
from every direction
The light field

• Known as the plenoptic function

• If you know R, you can predict how the scene would appear from
any viewpoint. How?

The light field t is not time (different from above t !)

• Assume radiance does not change along a ray
– what does this assume about the world?
• Parameterize rays by intersection with two planes:

• Usually drop λ and time parameters

• How could you capture a light field?
Capturing light fields

Stanford lightfield gantry Broxton et al., Google

What is light?
Electromagnetic radiation (EMR) moving along rays in space
• R(λ) is EMR, measured in units of power (watts)
– λ is wavelength

Perceiving light
• How do we convert radiation into “color”?
• What part of the spectrum do we see?
The visible light spectrum
We “see” electromagnetic radiation in a small range of wavelengths

from Michael Brown, ICCV19 color tutorial

Light spectrum
The appearance of light depends on its spectral power distribution (SPD)
• How much power (or energy) at each wavelength

wavelength (nm)

daylight incandescent bulb

near IR (heat)
SPD of different light sources
The appearance of light depends on its SPD
• How much power (or energy) at each wavelength
The human visual system

Color perception
• Light hits the retina, which contains photosensitive cells

• These cells convert the spectrum into a few discrete values

Density of rods and cones

Rods and cones are non-uniformly distributed on the retina

• Rods responsible for intensity, cones responsible for color
• Fovea - Small region (1 or 2°) at the center of the visual field containing the
highest density of cones (and no rods).
• Less visual acuity in the periphery—many rods wired to the same neuron
Demonstrations of visual acuity

With one eye shut, at the right distance, all of these letters
should appear equally legible (Glassner, 1.7).
Demonstrations of visual acuity

With left eye shut, look at the cross on the left. At the right
distance, the circle on the right should disappear (Glassner, 1.8).
Theorem of Perception

Perceived brightness ~= # photons

Theorem of Perception

Perceived brightness ~= # photons

Brightness contrast
The apparent brightness depends on the surrounding region

by Ted Adelson
Brightness constancy
A surface looks the same under widely varying lighting conditions
Visual dynamic range
Light response is nonlinear

Linear

perceptually linear (sRGB)

Jacob Bell, Programmers guide to gamma correction

Light response is nonlinear
Our visual system has a large dynamic range
• We can resolve both light and dark things at the same time
• One mechanism for achieving this is that we sense light
intensity on a logarithmic scale
– an exponential intensity ramp will be seen as a linear ramp
• Another mechanism is adaptation
– rods and cones adapt to be more sensitive in low light, less
sensitive in bright light.
Fixational eye movement
- Receptors adjust, lose sensitivity over time
- Eye keeps moving to expose new parts to light
- Microsaccades
- Short linear movement
- Sporadic
- Ocular drift
- Constant slow movement
- Microtremors
- Tiny vibrations
- Synchronized between eyes
- For seeing fine details
After images
Tired photoreceptors
• Send out negative response after a strong stimulus

https://michaelbach.de/ot/mot-adaptSpiral/index.html
Color perception

L response curve

Three types of cones

• Each is sensitive in a different region of the spectrum
– but regions overlap
– Short (S) corresponds to blue
– Medium (M) corresponds to green
– Long (L) corresponds to red
• Different sensitivities: we are more sensitive to green than red
– varies from person to person (and with age)
• Colorblindness—deficiency in at least one type of cone
2
7
Color perception
M L
Power

Wavelength

Rods and cones act as filters on the spectrum (SPD)

• To get the output of a filter, multiply its response curve by the
spectrum, integrate over all wavelengths
– Each cone yields one number
• Q: How can we represent an entire spectrum with 3 numbers?
• A: We can’t! Most of the information is lost.
– As a result, two different spectra may appear indistinguishable
» such spectra are known as metamers
» https://cs.brown.edu/courses/cs123/demos/metamers/index.html
We need metamers!
We need metamers!
Perception summary
The mapping from radiance to perceived color is quite complex!
• We throw away most of the data
• We apply a logarithm
• Brightness affected by pupil size
• Brightness contrast and constancy effects
• Afterimages
Camera response function
Now how about the mapping from radiance to pixels?
• It’s also complex, but better understood
• This mapping known as the film or camera response function

How can we recover radiance values given pixel values?

Why should we care?
• Useful if we want to estimate material properties
• Shape from shading requires radiance
• Enables creating high dynamic range images
What does the response function depend on?
Recovering the camera response
Method 1
• Carefully model every step in the pipeline
– measure aperture, model film, digitizer, etc.
– this is *really* hard to get right
Method 2
• Calibrate (estimate) the response function
– Image several objects with known radiance
– Measure the pixel values
– Fit a function
pixel intensity response curve
=

radiance

– Find the inverse: maps pixel intensity to radiance

Recovering the camera response
Method 3
• Calibrate the response function from several images
– Consider taking images with shutter speeds 1/1000, 1/100,
1/10, and 1
– Q: What is the relationship between the radiance or pixel
values in consecutive images?
– A: 10 times as much radiance
– Can use this to recover the camera response function

pixel intensity response curve

=

exposure
≅ radiance *
For more info time
=
• P. E. Debevec and J. Malik.
Recovering High Dynamic Range Radiance Maps from Photographs. In SIGGRAPH 97,
August 1997
sRGB

Linear

perceptually linear (sRGB)

sRGB:
● industry standard encoding of colors for most cameras, printers,
displays, etc. Most images are encoded as sRGB.
● To convert from sRGB to linear-RGB, a gamma function is applied to
each color component x of each pixel (assuming the values are
normalized [0,1] rather than [0,255]): f(x) = x^2.2
● You often want to convert to linear before you process an image and
then convert back to sRGB f(x) = x^(1/2.2)
Image formation

Let’s design a camera

• Idea 1: put a piece of film in front of an object
• Do we get a reasonable image?
Pinhole camera

Add a barrier to block off most of the rays

• This reduces blurring
• The opening known as the aperture
• How does this transform the image?
Pinhole Camera (Camera Obscura)

The first camera

• Known to Aristotle
Camera Obscura

The first camera

• Known to Aristotle
• How does the aperture size affect the image?
Shrinking the aperture

Why not make the aperture as small as possible?

Shrinking the aperture
Adding a lens

“circle of
confusion”

A lens focuses light onto the film

• There is a specific distance at which objects are “in focus”
– other points project to a “circle of confusion” in the image
• Changing the shape of the lens changes this distance
Lenses

focal point
optical center
(Center Of Projection)

A lens focuses parallel rays onto a single focal point

• focal point at a distance f beyond the plane of the lens
– f is a function of the shape and index of refraction of the lens
• Aperture of diameter D restricts the range of rays
– aperture may be on either side of the lens
• Lenses are typically spherical (easier to produce)
Thin lenses

Thin lens equation:

• Any object point satisfying this equation is in focus

• What is the shape of the focus region?
• How can we change the focus region?
• Thin lens applet: http://www.phy.ntnu.edu.tw/java/Lens/lens_e.html (by Fu-Kwun Hwang )
Depth of field

f / 5.6

f / 32
Changing the aperture size affects depth of field
• A smaller aperture increases the range in which the object is
approximately in focus

Flower images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_field

From light to pixels

https://sites.google.com/a/globalsystemsscience.org/digital-earth-watch/tools/digi
tal-cameras-overview/what-happens-to-the-near-infared-entering-the-camera
Bayer filters

https://en.wikipedia.org/wiki/Bayer_filter

¼ of pixels see red light (e.g.)

• Q: how do you get red at every pixel?
• A: Need to interpolate -- called debayering
Debayering

100 10 30

50 50 100

10 30 10

¼ of pixels see red light (e.g.)

• Q: how do you get red at every pixel?
• A: Need to interpolate -- called debayering
RGB images (three channel)

what we see

What we get out of the camera

From now on: what to do with these RGB images!