Color and Texture
How do we quantify them?
How do we use them to segment
an image?
1
Color
z Used heavily in human vision
z Color is a pixel property, making
some recognition problems easy
z Visible spectrum for humans is 400
nm (blue) to 700 nm (red)
z Machines can “see” much more; ex.
X-rays, infrared, radio waves
2
Factors that Affect Perception
• Light: the spectrum of energy that
illuminates the object surface
• Reflectance: ratio of reflected light to incoming light
• Specularity: highly specular (shiny) vs. matte surface
• Distance: distance to the light source
• Angle: angle between surface normal and light
source
• Sensitivity how sensitive is the sensor 3
Difference Between Graphics and
Vision
z In graphics we are given
values for all these
parameters, and we
create a view of the
surface.
z In vision, we are given a
view of the surface, and
we have to figure out
what’s going on. What’s going on?
4
Some physics of color:
Visible part of the electromagnetic spectrum
z White light is composed of all visible frequencies (400-700)
z Ultraviolet and X-rays are of much smaller wavelength
z Infrared and radio waves are of much longer wavelength 5
Coding methods for humans
• RGB is an additive system (add colors to black)
used for displays.
• CMY is a subtractive system for printing.
• HSI is a good perceptual space for art, psychology,
and recognition.
• YIQ used for TV is good for compression.
6
RGB color cube
• R, G, B values
normalized to
(0, 1) interval
• human
perceives gray
for triples on
the diagonal
• “Pure colors”
on corners 7
Color palette and normalized
RGB
Intensity I = (R+G+B) / 3
Color triangle for normalized
RGB coordinates is a slice Normalized red r = R/(R+G+B)
through the points [1,0,0],
[0,1,0], and [0,0,1] of the Normalized green g = G/(R+G+B)
RGB cube. The blue axis
is perpendicular to the page. Normalized blue b = B/(R+G+B)
In this normalized representation,
b = 1 – r –g, so we only need
to look at r and g to characterize
the color.
8
Color hexagon for HSI (HSV)
Hue is encoded as an angle (0 to 2π).
Saturation is the distance to the vertical axis (0 to 1).
Intensity is the height along the vertical axis (0 to 1).
H=120 is green saturation
intensity
hue
H=180 is cyan I=1 H=0 is red
H=240 is blue
9
I=0
Editing saturation of colors
(Left) Image of food originating from a digital camera;
(center) saturation value of each pixel decreased 20%;
(right) saturation value of each pixel increased 40%. 10
YIQ and YUV for TV signals
z Have better compression properties
z Luminance Y encoded using more bits than chrominance
values I and Q; humans more sensitive to Y than I,Q
z Luminance used by black/white TVs
z All 3 values used by color TVs
z YUV encoding used in some digital video and JPEG and
MPEG compression
11
Conversion from RGB to YIQ
An approximate linear transformation from RGB to YIQ:
We often use this for color to gray-tone conversion.
12
CIE, the color system we’ve been using
in recent object recognition work
z Commission Internationale de l'Eclairage -
this commission determines standards for
color and lighting. It developed the Norm
Color system (X,Y,Z) and the Lab Color
System (also called the CIELAB Color
System).
13
CIELAB, Lab, L*a*b
z One luminance channel (L)
and two color channels (a and
b).
z In this model, the color
differences which you perceive
correspond to Euclidian
distances in CIELab.
z The a axis extends from green
(-a) to red (+a) and the b axis
from blue (-b) to yellow (+b).
The brightness (L) increases
from the bottom to the top of
the three-dimensional model.
14
References
z The text and figures are from
http://www.sapdesignguild.org/resources/glossary_color/index1.ht
ml
z CIELab Color Space
http://www.fho-emden.de/~hoffmann/cielab03022003.pdf
z Color Spaces Transformations
http://www.couleur.org/index.php?page=transformations
z 3D Visualization
http://www.ite.rwth-
aachen.de/Inhalt/Forschung/FarbbildRepro/Farbkoerper/Visual3D.html
15
Colors can be used for image
segmentation into regions
z Can cluster on color values and pixel
locations
z Can use connected components and an
approximate color criteria to find regions
z Can train an algorithm to look for certain
colored regions – for example, skin color
16
Color histograms can represent
an image
z Histogram is fast and easy to compute.
z Size can easily be normalized so that
different image histograms can be compared.
z Can match color histograms for database
query or classification.
17
Histograms of two color images
18
Retrieval from image database
Top left
image is
query
image. The
others are
retrieved by
having
similar color
histogram
(See Ch 8).
19
How to make a color histogram
z Make 3 histograms and concatenate them
z Create a single pseudo color between 0 and 255
by using 3 bits of R, 3 bits of G and 2 bits of B
(which bits?)
z Use normalized color space and 2D histograms.
20
Apples versus Oranges
Separate HSI histograms for apples (left) and oranges
(right) used by IBM’s VeggieVision for recognizing produce
at the grocery store checkout station (see Ch 16).
21
Skin color in RGB space (shown as
normalized red vs normalized green)
Purple region
shows skin color
samples from
several people.
Blue and yellow
regions show
skin in shadow
or behind a
beard.
22
Finding a face in video frame
z (left) input video frame
z (center) pixels classified according to RGB space
z (right) largest connected component with aspect
similar to a face (all work contributed by Vera
Bakic) 23
Swain and Ballard’s Histogram Matching
for Color Object Recognition
(IJCV Vol 7, No. 1, 1991)
Opponent Encoding: • wb = R + G + B
• rg = R - G
• by = 2B - R - G
Histograms: 8 x 16 x 16 = 2048 bins
Intersection of image histogram and model histogram:
numbins
intersection(h(I),h(M)) = ∑ min{h(I)[j],h(M)[j]}
j=1
Match score is the normalized intersection:
numbins
match(h(I),h(M)) = intersection(h(I),h(M)) / ∑ h(M)[j] 24
j=1
(from Swain and Ballard)
cereal box image 3D color histogram
25
Four views of Snoopy Histograms
26
The 66 models objects Some test objects 27
More test objects used in occlusion experiments
28
Results
Results were surprisingly good.
At their highest resolution (128 x 90), average match
percentile (with and without occlusion) was 99.9.
This translates to 29 objects matching best with
their true models and 3 others matching second best
with their true models.
At resolution 16 X 11, they still got decent results
(15 6 4) in one experiment; (23 5 3) in another. 29
Color Clustering by K-means Algorithm
Use for HW 2
Form K-means clusters from a set of n-dimensional vectors
1. Set ic (iteration count) to 1
2. Choose randomly a set of K means m1(1), …, mK(1).
3. For each vector xi, compute D(xi,mk(ic)), k=1,…K
and assign xi to the cluster Cj with nearest mean.
4. Increment ic by 1, update the means to get m1(ic),…,mK(ic).
5. Repeat steps 3 and 4 until Ck(ic) = Ck(ic+1) for all k.
30
K-means Clustering Example
Original RGB Image Color Clusters by K-Means
31
Texture
Texture is a description of the spatial arrangement of color or
intensities in an image or a selected region of an image.
Structural approach: a set of texels in some regular or repeated pattern
32
Problem with Structural Approach
How do you decide what is a texel?
Ideas?
33
Natural Textures from VisTex
grass leaves
What/Where are the texels? 34
The Case for Statistical Texture
• Segmenting out texels is difficult or impossible in real images.
• Numeric quantities or statistics that describe a texture can be
computed from the gray tones (or colors) alone.
• This approach is less intuitive, but is computationally efficient.
• It can be used for both classification and segmentation.
35
Some Simple Statistical Texture
Measures
1. Edge Density and Direction
• Use an edge detector as the first step in texture analysis.
• The number of edge pixels in a fixed-size region tells us
how busy that region is.
• The directions of the edges also help characterize the texture
36
Two Edge-based Texture Measures
1. edgeness per unit area
Fedgeness = |{ p | gradient_magnitude(p) ≥ threshold}| / N
where N is the size of the unit area
2. edge magnitude and direction histograms
Fmagdir = ( Hmagnitude, Hdirection )
where these are the normalized histograms of gradient
magnitudes and gradient directions, respectively.
37
Original Image Frei-Chen Thresholded
Edge Image Edge Image
38
Local Binary Pattern Measure
• For each pixel p, create an 8-bit number b1 b2 b3 b4 b5 b6 b7 b8,
where bi = 0 if neighbor i has value less than or equal to p’s
value and 1 otherwise.
• Represent the texture in the image (or a region) by the
histogram of these numbers.
1 2 3
100 101 103
8 40 50 80 4 11111100
50 60 90 5
7 6
39
Fids (Flexible Image Database
System) is retrieving images
similar to the query image
using LBP texture as the
texture measure and comparing
their LBP histograms
40
Low-level
measures don’t
always find
semantically
similar images.
41
Co-occurrence Matrix Features
A co-occurrence matrix is a 2D array C in which
• Both the rows and columns represent a set of possible
image values.
• Cd (i,j) indicates how many times value i co-occurs with
value j in a particular spatial relationship d.
• The spatial relationship is specified by a vector d = (dr,dc).
42
1
012
1 1 0 0 i
1 1 0 0 3
0 103
0 0 2 2 1 202 Cd
0 0 2 2 j 2 001
0 0 2 2
0 0 2 2 d = (3,1) co-occurrence
matrix
gray-tone
image
From Cd we can compute Nd , the normalized co-occurrence matrix,
where each value is divided by the sum of all the values.
43
Co-occurrence Features
What do these measure?
sums.
Energy measures uniformity of the normalized matrix. 44
But how do you choose d?
• This is actually a critical question with all the
statistical texture methods.
• Are the “texels” tiny, medium, large, all three …?
• Not really a solved problem.
Zucker and Terzopoulos suggested using a χ2 statistical
test to select the value(s) of d that have the most structure
for a given class of images.
45
Laws’ Texture Energy Features
• Signal-processing-based algorithms use texture filters
applied to the image to create filtered images from which
texture features are computed.
• The Laws Algorithm
• Filter the input image using texture filters.
• Compute texture energy by summing the absolute
value of filtering results in local neighborhoods
around each pixel.
• Combine features to achieve rotational invariance.
46
Law’s texture masks (1)
47
Law’s texture masks (2)
Creation of 2D Masks
E5
L5
E5L5 48
9D feature vector for pixel
z Subtract mean neighborhood intensity from pixel
z Dot product 16 5x5 masks with neighborhood
z 9 features defined as follows:
49
Features from sample images
50
water
tiger
fence
flag
grass
small flowers Is there a
neighborhood
big flowers size problem
with Laws?
51
A classical texture measure:
Autocorrelation function
z Autocorrelation function can detect repetitive patterns of texels
z Also defines fineness/coarseness of the texture
z Compare the dot product (energy) of non shifted image with a
shifted image
52
Interpreting autocorrelation
z Coarse texture Æ function drops off slowly
z Fine texture Æ function drops off rapidly
z Can drop differently for r and c
z Regular textures Æ function will have peaks and
valleys; peaks can repeat far away from [0, 0]
z Random textures Æ only peak at [0, 0]; breadth
of peak gives the size of the texture
53
Fourier power spectrum
z High frequency power Æ fine texture
z Concentrated power Æ regularity
z Directionality Æ directional texture
54
What else?
• Gabor filters (we’ve used a lot)
• 3D textons (Leung and Malik)
• Polarity, anisotropy, and local contrast (Blobworld)
55
