1

Digital photography

15-463, 15-663, 15-862

Computational Photography
http://graphics.cs.cmu.edu/courses/15-463 Fall 2024, Lecture 2
 2

Course announcements
Complete the start-of-semester survey:

https://docs.google.com/forms/d/e/1FAIpQLScY5gtWcuSZ4X1n7MUtT5DPto92
1t5A80jFlB1mmq43oetVrA/viewform

Sign up for a camera:

https://docs.google.com/spreadsheets/d/1aWtAWuxstwshd5eq7BIW-
idBc4RJLVh1OMoL_8FbucY/

Both links available on Slack.

 3

Overview of today’s lecture

• Imaging sensor primer.

• Color primer.

• In-camera image processing pipeline.

• Some general thoughts on the image processing pipeline.

Take-home message: The values of pixels in a photograph and the values

output by your camera’s sensor are two very different things.
 4

Slide credits

A lot of inspiration and quite a few examples for these slides were taken directly from:

• Kayvon Fatahalian (15-769, Fall 2016).

• Michael Brown (CVPR 2016 Tutorial on understanding the image processing pipeline).

• Marc Levoy (Stanford CS 178, Spring 2014).

 5

The modern photography pipeline

 6

The modern photography pipeline

post-capture processing
(lectures 5-10)

sensor, analog in-camera image

optics and
front-end, and processing
optical controls
color filter array pipeline
(lectures 2-3, 11-20) (today, lecture 23) (today)
 7

Imaging sensor primer

 8

Imaging sensors

• Very high-level overview of digital imaging sensors.

• We could spend an entire course covering imaging sensors.

• Lecture 23 will cover sensors and noise issues in more detail. Canon 6D sensor
(20.20 MP, full-frame)
 9

What does an imaging sensor do?

When the camera shutter opens…

photons
… exposure begins…

array of photon buckets close-up view of photon buckets

… photon buckets begin to store photons...

… until the camera shutter closes. Then, they

convert stored photons to intensity values.
 10

Shutter speed
 11

Nobel Prize in Physics

Who is this?
 12

Nobel Prize in Physics

What is he known for?

 13

Photoelectric effect
incident emitted
photons electrons

Albert Einstein

Einstein’s Nobel Prize in Physics in 1921 “for his services to Theoretical

Physics, and especially for his discovery of the law of the photoelectric effect”
 14

Basic imaging sensor design

Canon 6D sensor
(20.20 MP, full-frame)
 15

Basic imaging sensor design

a pixel

microlens microlens microlens

color filter color filter color filter

photosite photosite
potential potential Canon 6D sensor
well well (20.20 MP, full-frame)

silicon for read- stores emitted made of silicon, emits

out etc. circuitry electrons electrons from photons

The term “photosite” can be used to refer to both the

entire pixel and only the photo-sensitive area.
 16

Photosite quantum efficiency (QE)

How many of the incident photons will the photosite
convert into electrons?

# electrons incident emitted

QE = photons electrons
# photons

• Fundamental optical performance metric of imaging sensors.

• Not the only important optical performance metric!

• We will see a few more later in the lecture.

 17

Photosite response
The photosite response is mostly linear

number of electrons
what does this slope equal?

number of photons
 18

Photosite response
The photosite response is mostly linear
what happens here?

number of electrons
QE

number of photons
 19

Photosite response
The photosite response is mostly linear, but:

• non-linear when potential well is saturated

saturation

number of electrons
(over-exposure)

• non-linear near zero (due to noise)

We will see how to deal with these issues in a

later lecture (high-dynamic-range imaging).
under-exposure
(non-linearity due
to sensor noise)
QE

number of photons
over-exposure
(non-linearity due Saturation means that the potential
to sensor saturation) well is full before exposure ends.
 20

Photosite full-well capacity

How many electrons can the

photosite store before saturation?

• Another important optical performance metric of imaging sensors.

 21

Pixel pitch and fill factor

pixel pitch: size of one side of
square pixel (typically 1-20 μm)

microlens microlens microlens

color filter color filter color filter

photosite photosite
potential potential
well well

silicon for read-

out etc. circuitry
fill factor: percentage of pixel area
taken up by photo-sensitive material
 22

Microlenses (also called lenslets)

pixel pitch: size of one side of What is the role of the microlenses?
square pixel (typically 1-20 μm)

microlens microlens microlens

color filter color filter color filter

photosite photosite
potential potential
well well

silicon for read-

out etc. circuitry
fill factor: percentage of pixel area
taken up by photo-sensitive material
 23

Microlenses (also called lenslets)

What is the role of the microlenses?
• Microlenses help photosite
collect more light by bending rays
towards photosensitive pixel area.
• Microlenses increase the effective
microlens microlens fill factor.
color filter color filter

photosite photosite
potential potential
well well

silicon for read-

out etc. circuitry
fill factor: percentage of pixel area
taken up by photo-sensitive material
 24

Microlenses

oblique view of microlens array

close-up of sensor cross-section shifted microlenses for improved fill factor

 25

Microlenses (also called lenslets)

What is the role of the microlenses?
• Microlenses help photosite
collect more light by bending rays
towards photosensitive pixel area.
• Microlenses increase the effective
microlens microlens fill factor.
• Microlenses also spatially low-
color filter color filter
pass filter the image to prevent
photosite photosite aliasing artifacts.
potential potential What kind of spatial filter do the
well well microlenses implement?

silicon for read-

out etc. circuitry
fill factor: percentage of pixel area
taken up by photo-sensitive material
 26

Microlenses (also called lenslets)

What is the role of the microlenses?
• Microlenses help photosite
collect more light by bending rays
towards photosensitive pixel area.
• Microlenses increase the effective
microlens microlens fill factor.
• Microlenses also spatially low-
color filter color filter
pass filter the image to prevent
photosite photosite aliasing artifacts by implementing
potential potential a pixel-sized 2D rect (box) filter.
well well • Often an additional optical low-
pass filter (OLPF) is placed in front
silicon for read- of the sensor to improve prefilter.
out etc. circuitry
fill factor: percentage of pixel area
taken up by photo-sensitive material
 27

Quick aside: optical low-pass filter

• Sensors often have a separate glass sheet in front of them acting as an optical low-pass filter
(OLPF, also known as optical anti-aliasing filter).
• The OLPF is typically implemented as two birefringent layers, combined with the infrared filter.
• The two layers split 1 ray into 4 rays, implementing a 4-tap discrete convolution filter kernel.

birefringence in a calcite crystal birefringence ray diagram

 28

Quick aside: optical low-pass filter

• Sensors often have a separate glass sheet in front of them acting as an optical low-pass filter
(OLPF, also known as optical anti-aliasing filter).
• The OLPF is typically implemented as two birefringent layers, combined with the infrared filter.
• The two layers split 1 ray into 4 rays, implementing a 4-tap discrete convolution filter kernel.

• However, the OLPF means you also lose resolution.

• Nowadays, due the large number of pixels, OLPF are
becoming unnecessary.
• Photographers often hack their cameras to remove the
OLPF, to avoid the loss of resolution (“hot rodding”).
• Camera manufacturers offer camera versions with and
without an OLPF.
• The OLPF can be problematic also when working with
coherent light (spurious fringes).
 29

Quick aside: optical low-pass filter

Example where OLPF is needed

without OLPF with OLPF

 30

Quick aside: optical low-pass filter

Example where OLPF is unnecessary

without OLPF with OLPF

 31

Quick aside: optical low-pass filter

Identical camera model with and without an OLPF (no need for customization).

Nikon D800 Nikon D800E

 32

Sensor size

• “Full frame” corresponds to

standard film size.
• Digital sensors are often smaller
than film because of cost.
 33

Two main types of imaging sensors

Do you know them?

 34

Two main types of imaging sensors

Charged coupled device (CCD): Complementary metal oxide semiconductor (CMOS):

• row brigade shifts charges row-by-row • per-pixel amplifiers convert charges to voltages
• amplifiers convert charges to voltages row-by-row • multiplexer reads voltages row-by-row

Can you think of advantages and disadvantages of each type?

 35

Two main types of imaging sensors

Charged coupled device (CCD): Complementary metal oxide semiconductor (CMOS):

• row brigade shifts charges row-by-row • per-pixel amplifiers convert charges to voltages
• amplifiers convert charges to voltages row-by-row • multiplexer reads voltages row-by-row

Can you think of advantages and disadvantages of each type?

 36

Two main types of imaging sensors

Charged coupled device (CCD): Complementary metal oxide semiconductor (CMOS):

• row brigade shifts charges row-by-row • per-pixel amplifiers convert charges to voltages
• amplifiers convert charges to voltages row-by-row • multiplexer reads voltages row-by-row

 higher sensitivity  faster read-out

 lower noise  lower cost
 37

Artifacts of the two types of sensors

sensor bloom smearing artifacts

Which sensor type can have these artifacts?

 38

Artifacts of the two types of sensors

sensor bloom smearing artifacts

(CMOS and CCD) (CCD only)
Overflow from saturated pixels
• mitigated by more electronics to contain charge
(at the cost of photosensitive area)
 39

CCD vs CMOS

• Modern CMOS sensors

have optical performance
comparable to CCD
sensors.

• Most modern
commercial and
industrial cameras use
CMOS sensors.
 40

CMOS sensor (very) simplified layout

optically black region
photosite (no light gets here)
(pixel)

row selection Can anyone guess why there are

register pixels in the optically black region?
…
exposed region
(light gets here)
active pixel sensor
(2D array of pixels)
analog front-end

row buffer bits

 41

Analog front-end

analog analog discrete discrete

voltage voltage signal signal

analog amplifier (gain): analog-to-digital look-up table (LUT):

• gets voltage in range converter (ADC): • corrects non-linearities in
needed by A/D converter. • depending on sensor, sensor’s response function
• accommodates ISO settings. output has 10-16 bits. (within proper exposure).
• accounts for vignetting. • most often (?) 12 bits. • corrects defective pixels.
 42

Vignetting
Fancy word for: pixels far off the center receive less light

white wall under uniform light more interesting example of vignetting

 43

Vignetting
Four types of vignetting:

• Mechanical: light rays blocked by hoods, filters, and other objects.

• Lens: similar, but light rays blocked by lens elements.

• Natural: due to radiometric laws (“cosine fourth falloff”).

• Pixel: angle-dependent sensitivity of photosites.

non-uniform
gain
 44

What does an imaging sensor do?

When the camera shutter opens, the sensor:

• at every photosite, converts incident photons into electrons

• stores electrons into the photosite’s potential well while it is not full

… until camera shutter closes. Then, the analog front-end:

• reads out photosites’ wells, row-by-row, and converts them to analog signals

• applies a (possibly non-uniform) gain to these analog signals

• converts them to digital signals

• corrects non-linearities

… and finally returns an image.

 45

Remember these?
helps photosite • Lenslets also filter the image
collect more light to avoid resolution artifacts.
(also called lenslet) • Lenslets are problematic when
working with coherent light.
microlens microlens • Many modern cameras do not
color filter color filter
have lenslet arrays.

photosite photosite We will discuss these issues in

potential potential more detail at a later lecture.
well well

silicon for read- stores emitted made of silicon, emits

out etc. circuitry electrons electrons from photons

We will see what the color filters are for later in this lecture.
 46

Color primer
 47

Color

• Very high-level of color as it relates to digital photography.

• We could spend an entire course covering color.

• We will discuss color in more detail in a later lecture.

color is complicated
 48

Color is an artifact of human perception

• “Color” is not an objective physical property of light (electromagnetic radiation).
• Instead, light is characterized by its wavelength.

electromagnetic
spectrum

What we call “color” is how we

subjectively perceive a very small
range of these wavelengths.
 49

Spectral Power Distribution (SPD)

• Most types of light “contain” more than one wavelengths.
• We can describe light based on the distribution of power over different wavelengths.

We call our sensation

of all of these
distributions “white”.
 50

Spectral Sensitivity Function (SSF)

• Any light sensor (digital or not) has different sensitivity to different wavelengths.

• This is described by the sensor’s spectral sensitivity function .

• When measuring light of some SPD , the sensor produces a scalar response:

light SPD sensor SSF

sensor
response

Weighted combination of light’s SPD: light contributes more at

wavelengths where the sensor has higher sensitivity.
 51

Spectral Sensitivity Function of Human Eye

• The human eye is a collection of light sensors called cone cells.

• There are three types of cells with different spectral sensitivity functions.

• Human color perception is three-dimensional (tristimulus color).

“short”

“medium”

“long”
cone distribution
for normal vision
(64% L, 32% M)
 52

Color filter arrays (CFA)

• To measure color with a digital sensor, mimic cone cells of human vision system.

• “Cones” correspond to pixels that are covered by different color filters, each with its
own spectral sensitivity function.

microlens microlens microlens

color filter color filter color filter

photosite photosite photosite

potential potential potential
well well well
 53

What color filters to use?

Two design choices:

• What spectral sensitivity functions to use for each color filter?

• How to spatially arrange (“mosaic”) different color filters?

Bayer SSF for

mosaic Canon 50D

Why more Generally do not

green pixels? match human LMS.
 54

Many different CFAs

Finding the “best” CFA mosaic is an active research area.

CYGM RGBE
Canon IXUS, Powershot Sony Cyber-shot

How would you go about designing your own

CFA? What criteria would you consider?
 55

Many different spectral sensitivity functions

Each camera has its more or less unique, and most of the time secret, SSF.
• Makes it very difficult to correctly reproduce the color of sensor measurements.
• We will see more about this in the color lecture.

Images of the same scene captured using 3 different cameras with identical settings.
 56

Aside: can you think of other ways to capture color?

 57

Aside: can you think of other ways to capture color?

[Slide credit: Gordon Wetzstein]

 58

Aside: can you think of other ways to capture color?

Lippmann self portrait:
one of the very first holograms (around 1890!)

Modern
camera with
Lippmann plate
[credit: Hans
Bjelkhagen]

Lippmann’s Nobel Prize in Physics in 1908 “for his method, based on the phenomenon
of interference, which permits the reproduction of colours by photography.”
 59

What does an imaging sensor do?

When the camera shutter opens, the sensor:

• at every photosite, converts incident photons into electrons using mosaic’s SSF

• stores electrons into the photosite’s potential well while it is not full

… until camera shutter closes. Then, the analog front-end:

• reads out photosites’ wells, row-by-row, and converts them to analog signals

• applies a (possibly non-uniform) gain to these analog signals

• converts them to digital signals

• corrects non-linearities

… and finally returns an image.

 60

After all of this, what does an image look like?

lots of mosaicking
noise artifacts

• Kind of disappointing.
• We call this the RAW image.
 61

The modern photography pipeline

post-capture processing
(lectures 5-10)

sensor, analog in-camera image

optics and
front-end, and processing
optical controls
color filter array pipeline
(lectures 2-3, 11-20) (today, lecture 23) (today)
 62

The in-camera image processing pipeline

 63

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 64

Quick notes on terminology

• Sometimes the term image signal processor (ISP) is used to refer to the image
processing pipeline itself.

• The process of converting a RAW image to a “conventional” image is often called

rendering (unrelated to the image synthesis procedure of the same name in
graphics).

• The inverse process, going from a “conventional” image back to RAW is called
derendering.
 65

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

see color lecture see 18-793

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 66

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 67

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.

[Slide credit: Todd Zickler]

 68

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.

[Slide credit: Todd Zickler]

 69

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.

Retinal vs
perceived color.

[Slide credit: Todd Zickler]

 70

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.

Retinal vs
perceived color.

[Slide credit: Todd Zickler]

 71

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.

Retinal vs
perceived color.

[Slide credit: Todd Zickler]

 72

White balancing
Human visual system has chromatic adaptation:
• We can perceive white (and other colors) correctly under different light sources.
• Cameras cannot do that (there is no “camera perception”).

White balancing: The process of removing color casts so that colors that we would
perceive as white are rendered as white in final image.

image captured image white-

different whites
under fluorescent balanced to daylight
 73

White balancing presets

Cameras nowadays come with a large number of presets: You can select which light you
are taking images under, and the appropriate white balancing is applied.
 74

Manual vs automatic white balancing

Manual white balancing:
• Select a camera preset based on lighting.

Can you think of any other way to do manual white balancing?

 75

Manual vs automatic white balancing

Manual white balancing:
• Select a camera preset based on lighting.
• Manually select object in photograph that is color-neutral and use it to normalize.

How can we do automatic white balancing?

 76

Manual vs automatic white balancing

Manual white balancing:
• Select a camera preset based on lighting.
• Manually select object in photograph that is color-neutral and use it to normalize.

Automatic white balancing:

• Grey world assumption: force average color of scene to be grey.
• White world assumption: force brightest object in scene to be white.
• Sophisticated histogram-based algorithms (what most modern cameras do).
 77

Automatic white balancing

Grey world assumption:
• Compute per-channel average.
• Normalize each channel by its average.
• Normalize by green channel average.

white-balanced sensor RGB

RGB

White world assumption:

• Compute per-channel maximum.
• Normalize each channel by its maximum.
• Normalize by green channel maximum.

white-balanced sensor RGB

RGB
 78

Automatic white balancing example

input image grey world white world

 79

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 80

CFA demosaicing
Produce full RGB image from mosaiced sensor output.

Any ideas on how to do this?

 81

CFA demosaicing
Produce full RGB image from mosaiced sensor output.

Interpolate from neighbors:

• Bilinear interpolation (needs 4 neighbors).
• Bicubic interpolation (needs more neighbors, may overblur).
• Edge-aware interpolation (more on this later).
 82

Demosaicing by bilinear interpolation

Bilinear interpolation: Simply average your 4 neighbors.
G2
G1 + G2 + G3 + G4
G1 G? G3 G? =
4
G4

Neighborhood changes for different channels:

 83

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 84

Noise in images
Can be very pronounced in low-light images.
 85

Three types of sensor noise

1) (Photon) shot noise:

• Photon arrival rates are a random process (Poisson distribution).

• The brighter the scene, the larger the variance of the distribution.

2) Dark-shot noise:

• Emitted electrons due to thermal activity (becomes worse as sensor gets hotter.)

3) Read noise:

• Caused by read-out and AFE electronics (e.g., gain, A/D converter).

Bright scene and large pixels: photon shot noise is the main noise source.
 86

How to denoise?
 87

How to denoise?
Look at the neighborhood around you. I1 I2 I3
I4 I5 I6
I7 I8 I9
• Mean filtering (take average):

I1 + I2 + I3 + I4 + I5 + I6 + I7 + I8 + I9
I’ 5 =
9
• Median filtering (take median):

I’ 5 = median( I1 , I2 , I3 , I4 , I5 , I6 , I7 , I8 , I9 )

Large area of research. We will see some more about filtering in a later lecture.
 88

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 89

Perceived vs measured brightness by human eye

We have already seen that sensor response is linear.

Human-eye response (measured brightness) is also

linear.

However, human-eye perception (perceived

brightness) is non-linear:
• More sensitive to dark tones.
• Approximately equal to square root function
(concave “gamma curve” Lγ with γ = 1/2.2).
 90

Gamma encoding
After this stage, we perform compression, which includes changing from 12 to 8 bits.
• Apply non-linear curve to use available bits to better encode the information human
vision is more sensitive to.
 91

Demonstration
original (8-bits, 256 tones)

Can you predict what will happen if we linearly encode this tone range with only 5 bits?

Can you predict what will happen if we gamma encode this tone range with only 5 bits?
 92

Demonstration
original (8-bits, 256 tones)

linear encoding (5-bits, 32 tones)

all of this range gets all of these tones

mapped to just one tone look the same

Can you predict what will happen if we gamma encode this tone range with only 5 bits?
 93

Demonstration
original (8-bits, 256 tones)

linear encoding (5-bits, 32 tones)

all of this range gets all of these tones

mapped to just one tone look the same

gamma encoding (5-bits, 32 tones)

tone encoding becomes a lot

more perceptually uniform
 94

Tone reproduction pipeline

sensor: ISP: concave display: convex

linear curve gamma curve gamma curve
 95

Tone reproduction pipeline

sensor: ISP: concave display: convex net effect: linear

linear curve gamma curve gamma curve curve
 96

Tone reproduction pipeline

sensor: ISP: concave display: convex net effect: linear

linear curve gamma curve gamma curve curve

gamma encoding gamma correction

 97

Tone reproduction pipeline

human visual
system: concave
gamma curve

image a human
would see at
different stages of
the pipeline
 98

RAW pipeline
gamma encoding display still applies
is skipped! gamma correction!

human visual
system: concave
gamma curve

image a human
RAW image appears very
would see at
dark! (Unless you are
different stages of
using a RAW viewer)
the pipeline
 99

Historical note

• CRT displays used to have a response curve that was (almost) exactly equal to the inverse
of the human sensitivity curve. Therefore, displays could skip gamma correction and
display directly the gamma-encoded images.

• It is sometimes mentioned that gamma encoding is done to undo the response curve of a
display. This used to (?) be correct, but it is not true nowadays. Gamma encoding is
performed to ensure a more perceptually-uniform use of the final image’s 8 bits.
 100

Gamma encoding curves

The exact gamma encoding curve depends on the camera.
• Often well approximated as Lγ, for different values of the power γ (“gamma”).
• A good default is γ = 1/2.2 (approximately the square root).

before gamma after gamma

Warning: Our values are no longer linear relative to scene radiance!
 101

The (in-camera) image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
 102

Some general thoughts on the image processing

pipeline
 103

Do I ever need to use RAW?

 104

Do I ever need to use RAW?

Emphatic yes!

• Every time you use a physics-based computer vision algorithm, you need linear
measurements of radiance.

• Examples: photometric stereo, shape from shading, image-based relighting, illumination

estimation, anything to do with light transport and inverse rendering, etc.

• Applying the algorithms on non-linear (i.e., not RAW) images will produce completely
invalid results.
 105

What if I don’t care about physics-based vision?

 106

What if I don’t care about physics-based vision?

You often still want (rather than need) to use RAW!

• If you like re-finishing your photos (e.g., on Photoshop), RAW makes your life much easier
and your edits much more flexible.
 107

Are there any downsides to using RAW?

 108

Are there any downsides to using RAW?

Image files are a lot bigger.

• You burn through multiple memory cards.

• Your camera will buffer more often when shooting in burst mode.

• Your computer needs to have sufficient memory to process RAW images.

 109

Is it even possible to get access to RAW images?

 110

Is it even possible to get access to RAW images?

Quite often yes!

• Most high-end cameras provide an option to store RAW image files.

• Certain phone cameras allow, directly or indirectly, access to RAW.

• Sometimes, it may not be “fully” RAW. The Lightroom app provides images after
demosaicking but before tone reproduction.
 111

I forgot to set my camera to RAW, can I still get the RAW file?

Nope, tough luck.

• The image processing pipeline is lossy: After all the steps, information about the original
image is lost.

• Sometimes we may be able to reverse a camera’s image processing pipeline if we know

exactly what it does (e.g., by using information from other similar RAW images).

• The conversion of PNG/JPG back to RAW is known as “derendering” and is an active

research area.
 112

Derendering
 113

Why did you use italics in the previous slide?

What I described today is an “idealized” version of what we think commercial cameras do.

• Almost all of the steps in both the sensor and image processing pipeline I described
earlier are camera-dependent.

• Even if we know the basic steps, the implementation details are proprietary information
that companies actively try to keep secret.

• I will go back to a few of my slides to show you examples of the above.

 114

The hypothetical image processing pipeline

The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end?
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising?
demosaicing? balance?

final RGB
color tone compression
image (non-
transforms? reproduction? ?
linear, 8-bit)
 115

The hypothetical analog front-end

analog analog discrete discrete

voltage voltage signal signal

analog amplifier (gain): analog-to-digital look-up table (LUT):

• gets voltage in range converter (ADC): • corrects non-linearities in
needed by A/D converter? • depending on sensor, sensor’s response function
• accommodates ISO settings? output has 10-16 bits. (within proper exposure)?
• accounts for vignetting? • most often (?) 12 bits. • corrects defective pixels?
 116

Various curves
All of these sensitivity curves are different from camera to camera and kept secret.
 117

Serious inhibition for research

• Very difficult to get access to ground-truth data at intermediate stages of the pipeline.

• Very difficult to evaluate effect of new algorithms for specific pipeline stages.
 118

…but things are getting better

 119

…but things are getting better

 120

How do I open a RAW file in Python?

You can’t (not easily at least). You need to use one of the following:

• dcraw – tool for parsing camera-dependent RAW files (specification of file formats are
also kept secret).

• Adobe DNG – recently(-ish) introduced file format that attempts to standardize RAW file
handling.

See Homework 1 for more details.

 121

Is this the best image processing pipeline?

It depends on how you define “best”. This definition is task-dependent.

• The standard image processing pipeline is designed to create “nice-looking” images.

• If you want to do physics-based vision, the best image processing pipeline is no pipeline
at all (use RAW).

• What if you want to use images for, e.g., object recognition? Tracking? Robotics SLAM?
Face identification? Forensics?

Developing task-adaptive image processing pipelines is an active area of research.

 122

Take-home messages

The values of pixels in a photograph and the values output by your

camera’s sensor are two very different things.

The relationship between the two is complicated and unknown.

 123

References
Basic reading:
• Szeliski textbook, Section 2.3.
• Michael Brown, “Understanding the In-Camera Image Processing Pipeline for Computer Vision,” CVPR 2016,
slides available at: http://www.comp.nus.edu.sg/~brown/CVPR2016_Brown.html

Additional reading:
• Adams et al., “The Frankencamera: An Experimental Platform for Computational Photography,” SIGGRAPH 2010.
The first open architecture for the image processing pipeline, and precursor to the Android Camera API.
• Heide et al., “FlexISP: A Flexible Camera Image Processing Framework,” SIGGRAPH Asia 2014.
Discusses how to implement a single-stage image processing pipeline.
• Buckler et al., “Reconfiguring the Imaging Pipeline for Computer Vision,” ICCV 2017.
• Diamond et al., “Dirty Pixels: Optimizing Image Classification Architectures for Raw Sensor Data,” arXiv 2017.
Both papers discuss how to adaptively change the conventional image processing pipeline so that it is better suited to various
computer vision problems.
• Chakrabarti et al., “Rethinking Color Cameras,” ICCP 2014.
Discusses different CFAs, including ones that have white filters, and how to do demosaicing for them.
• Gunturk et al., “Demosaicking: Color Filter Array Interpolation,” IEEE Signal Processing Magazine 2005
A nice review of demosaicing algorithms.
• Kim et al., “A New In-Camera Imaging Model for Color Computer Vision and Its Application,” PAMI 2012.
• Chakrabarti et al., “Probabilistic Derendering of Camera Tone-mapped Images,” PAMI 2014.
Two papers that discuss in detail how to model and calibrate the image processing pipeline, how to (attempt to) derender an image
that has already gone through the pipeline, and how to rerender an image under a different camera’s pipeline.
• Baechler et al., “Shedding light on 19th century spectra by analyzing Lippmann photography,” PNAS 2021.
A recent paper analyzing Lippmann color photography.