C280 Computer Vision
C280,ComputerVision
Prof.TrevorDarrell
[email protected]
Lecture19:Tracking
Tracking scenarios
Follow a point
Follow
ll a template
l
Follow a changing template
Follow all the elements of a moving person, fit a
model to it.
Things to consider in tracking
What are the dynamics of the thing being tracked?
How is
i it
i observed?
b
d?
Three main issues in tracking
Simplifying Assumptions
Kalman filter graphical model and
corresponding factorized joint probability
x1
x2
x3
y1
y2
y3
P( x1 , x2 , x3 , y1 , y2 , y3 )
P( x1 ) P( y1 | x1 ) P( x2 | x1 ) P( y2 | x2 ) P( x3 | x2 ) P( y3 | x3 )
6
Tracking as induction
Make a measurement starting in the 0th frame
Then:
h
assume you have
h
an estimate
i
at the
h ith
ih
frame, after the measurement step.
Show
Sh th
thatt you can do
d prediction
di ti for
f the
th i+1th
frame, and measurement for the i+1th frame.
Base case
Prediction step
given
Update step
given
10
The Kalman Filter
Key ideas:
Li
Linear models
d l interact
i t
t uniquely
i l well
ll with
ith Gaussian
G
i
noise - make the prior Gaussian, everything else
Gaussian and the calculations are easy
Gaussians are really easy to represent --- once you
know the mean and covariance, youre done
11
Recall the three main issues in tracking
(Ignore data association for now)
12
The Kalman Filter
13
[figure from http://www.cs.unc.edu/~welch/kalman/kalmanIntro.html]
The Kalman Filter in 1D
Dynamic Model
Notation
Predicted mean
Corrected mean
14
The Kalman Filter
15
Prediction for 1D Kalman filter
The new state is obtained by
multiplying
lti l i old
ld state
t t by
b known
k
constant
t t
adding zero-mean noise
Therefore, predicted mean for new state is
constant times mean for old state
Old variance is normal random variable
variance is multiplied by square of constant
and variance of noise is added.
16
17
The Kalman Filter
18
Measurement update for 1D Kalman filter
Notice:
if measurement noise is small,
we rely mainly on the measurement,
if its large, mainly on the prediction
does
d
nott depend
d
d on y
19
20
Kalman filter for computing an on-line average
What Kalman filter parameters and initial
conditions should we pick so that the optimal
estimate for x at each iteration is just the average
of all the observations seen so far?
21
Kalman filter model
d i 1, mi 1, d i 0, mi 1
Initial conditions
x0 0 0
Iteration
y0
y0
x
x
y0 y1
2
1
2
2
y0 y1
2
y0 y1 y2
3
1
2
1
3
22
What happens if the x dynamics are given a
non-zero
non
zero variance?
23
Kalman filter model
d i 1, mi 1, d i 1, mi 1
Initial conditions
x0 0 0
Iteration
y0
xi
y0
2
3
y0 2 y1
3
y0 2 y1
3
y0 2 y1 5 y2
8
5
3
5
8
24
Linear dynamic models
A linear dynamic model has the form
x i N Di1x i1 ; d i
y i N Mi x i ; m i
This is much, much more general than it looks, and extremely
powerful
25
Examples of linear state space
models
Drifting points
x i N Di1x i1 ; d i
y i N Mi x i ; m i
assume that
th t the
th new position
iti off the
th point
i t is
i the
th old
ld one,
plus noise
D = Identity
cic.nist.gov/lipman/sciviz/images/random3.gif 26
http://www.grunch.net/synergetics/images/random
3.jpg
x i N Di1x i1 ; d i
Constant velocity
y i N Mi x i ; m i
We have
ui ui1 tvi1 i
vi vi1 i
(the Greek letters denote noise terms)
Stack (u, v) into a single state vector
u 1
vi 0
which is the form we had above
xi
t u
i
noise
1 v i1
Di-1
xi-1
27
velocity
position
pposition
time
measurement,position
Constant
C
Velocity
Model
time
28
x i N Di1x i1 ; d i
Constant acceleration
y i N Mi x i ; m i
We have
ui ui1 tvi1 i
vi vi1 tai1 i
ai ai1 i
(the Greek letters denote noise terms)
Stack (u, v) into a single state vector
u 1 t
v 0 1
ai 0 0
which is the form we had above
xi
Di-1
0 u
t v noise
1 a i1
xi-1
29
velocity
position
position
Constant
Acceleration
cce e o
Model
time
30
Periodic motion
x i N Di1x i1 ; d i
y i N Mi x i ;
mi
Assume we have a point, moving on a line with
a periodic movement defined with a
differential eq:
can be defined as
with
i h state d
defined
fi d as stacked
k d position
i i and
d
velocity u=(p, v)
31
x i N Di1x i1 ; d i
Periodic motion
y i N Mi x i ; m i
Take discrete approximation.(e.g., forward
Euler integration with t stepsize.)
xi
Di-1
xi-1
32
n-D
Generalization to n-D is straightforward but more complex.
33
n-D
Generalization to n-D is straightforward but more complex.
34
n-D Prediction
Generalization to n-D is straightforward but more complex.
Prediction:
Multiply estimate at prior time with forward model:
Propagate covariance through model and add new noise:
35
n-D Correction
Generalization to n-D is straightforward but more complex.
Correction:
Update a priori estimate with measurement to form a
posteriori
36
n-D correction
Find linear filter on innovations
which minimizes a posteriori error covariance:
E xx
x x
K is the Kalman Gain matrix. A solution is
37
Kalman Gain Matrix
As measurement becomes more reliable, K weights residual
more heavily,
1
K
M
lim i
m 0
As prior covariance approaches 0, measurements are ignored:
li K i 0
lim
i 0
38
39
posittion
veloocity
position
time
Constant Velocity Model
40
time
41
possition
posittion
This is figure 17.3 of Forsyth and Ponce. The notation is a bit involved, but is logical. We
plot the true state as open circles, measurements as xs, predicted means as *s
with three standard deviation bars, corrected means as +s with three
standard deviation bars.
time
42
posittion
time
The o-s give state, x-s measurement.
43
Smoothing
Idea
W
We dont
d t have
h
the
th best
b t estimate
ti t off state
t t - what
h t about
b t
the future?
Run two filters, one moving forward, the other
backward in time.
Now combine state estimates
The crucial point here is that we can obtain a smoothed
estimate by viewing the backward filters prediction as yet
another measurement for the forward filter
44
positionn
Forward estimates.
The o-s give state, x-s measurement.
time
45
positionn
Backward estimates.
The o-s give state, x-s measurement.
time
46
positionn
Combined forward-backward estimates.
The o-s give state, x-s measurement. time
47
2-D constant velocity example from Kevin Murphys Matlab toolbox
48
[figure from http://www.ai.mit.edu/~murphyk/Software/Kalman/kalman.html]
22-D
D constant velocity example from Kevin Murphys Matlab toolbox
MSE of filtered estimate is 4.9; of smoothed estimate. 3.2.
Not only is the smoothed estimate better, but we know that it is better,
as illustrated
ill
d bby the
h smaller
ll uncertainty
i
ellipses
lli
Note how the smoothed ellipses are larger at the ends, because these
points have seen less data.
Also, note how rapidly the filtered ellipses reach their steady-state
(Ricatti) values.
49
[figure from http://www.ai.mit.edu/~murphyk/Software/Kalman/kalman.html]
Resources
Kalman filter homepage
h //
http://www.cs.unc.edu/~welch/kalman/
d /
l h/k l
/
(kalman filter demo applet)
Kevin Murphys Matlab toolbox:
http://www.ai.mit.edu/~murphyk/Software/Kalman/k
alman.html
50
Embellishments for tracking
Richer models of P(xn|xn-1)
Richer
i h models
d l off P(y
( n|x
| n)
Richer model of probability distributions
51
Abrupt changes
What if environment is sometimes unpredictable?
Do people move with constant velocity?
Test several models of assumed dynamics, use the
best.
52
Multiple model filters
Test several models of assumed dynamics
53
[figure from Welsh and Bishop 2001]
MM estimate
Two models: Position (P), Position+Velocity (PV)
54
[figure from Welsh and Bishop 2001]
P likelihood
55
[figure from Welsh and Bishop 2001]
No lag
56
[figure from Welsh and Bishop 2001]
Smooth when still
57
[figure from Welsh and Bishop 2001]
Embellishments for tracking
Richer
i h models
d l off P(y
( n|x
| n)
Richer model of probability distributions
58
Jepson, Fleet, and El-Maraghi tracker
59
Wandering, Stable, and Lost appearance model
Introduce 3 competing models to explain the
appearance of the tracked region:
A stable modelGaussian with some mean and
covariance.
A 2-frame motion tracker appearance model, to rebuild
the stable model when it gets lost
An outlier modeluniform probability over all
appearances.
Use an on-line
on line EM algorithm to fit the (changing)
model parameters to the recent appearance data.
60
Jepson, Fleet, and El-Maraghi tracker for toy 1-pixel
image model
Red line: observations
Blue line: true appearance state
Black line: mean of the stable process
61
Mixing
probabilities for
Stable (black),
Wandering (red)
and Lost (green)
processes
62
Non-toy image representation
Phase of a steerable quadrature pair (G2, H2).
Steered to 4 different orientations
orientations, at 2 scales.
scales
63
The motion tracker
Motion prior, P(x n | x n1) , prefers slow velocities
and small accelerations
accelerations.
The WSL appearance model gives a likelihood for
each possible new position
position, orientation
orientation, and scale
of the tracked region.
They combine that with the motion prior to find
the most probable position, orientation, and scale
of the tracked region
g
in the next frame.
Gives state-of-the-art tracking results.
64
Jepson, Fleet, and El-Maraghi tracker
65
Jepson, Fleet, and El-Maraghi tracker
Add fleet&jepson tracking slides
Far right
column: the
stable
components
mixing
probability.
Note its
i
behavior at
occlusions.
66
Embellishments for tracking
Richer
i h model
d l off P(y
( n|x
| n)
Richer model of probability distributions
Particle filter models, applied to tracking humans
68
(KF) Distribution propagation
prediction from previous time frame
Noise
added to
that
h
prediction
Make new measurement at next time frame
69
[Isard 1998]
Distribution propagation
70
[Isard 1998]
Representing non-linear Distributions
71
Representing non-linear Distributions
Unimodal parametric models fail to capture realworld densities
densities
72
Discretize by evenly sampling over
the entire state space
Tractable for 1-d problems like stereo, but not for
high dimensional problems.
high-dimensional
problems
73
Representing Distributions using
Weighted Samples
Rather than a parametric form, use a set of samples
to represent a density:
74
Representing Distributions using
Weighted Samples
Rather than a parametric form, use a set of samples
to represent a density:
Sample positions
Probability mass at each sample
This gives us two knobs to adjust when representing a probability density by samples:
the locations of the samples, and the probability weight on each sample.
75
Representing distributions using
weighted samples,
samples another picture
76
[Isard 1998]
Sampled representation of a
probability distribution
You can also think of this as a sum of dirac delta functions,
each of weight w:
p f ( x ) w i ( x u i )
i
77
Tracking, in particle filter representation
x1
x2
x3
y1
y2
y3
p f ( x ) w i ( x u i )
i
P(x n | y1 ...y n ) k P(y n | x n ) dx n1P(x n | x n1 )P(x n1 | y1 ...y n1 )
P di ti step
Prediction
t
Update step
78
Particle filter
Lets apply this sampled probability density
machinery to generalize the Kalman filtering
framework.
More general probability density representation
than uni-modal Gaussian.
Allows for general state dynamics,
dynamics f(x) + noise
79
Sampled Prediction
= ?
Drop elements to marginalize to get
~=
80
Sampled Correction (Bayes rule)
Prior posterior
Reweight
i h every sample
l with
i h the
h likelihood
lik lih d off the
h
observations, given that sample:
yielding
i ldi a set off samples
l describing
d
ibi the
h probability
b bili
distribution after the correction (update) step:
81
Nave PF Tracking
Start with samples from something simple
(Gaussian)
Take each particle from the prediction step and
Repeat
modify the old weight by multiplying by the
new likelihood
Correct
Predict
s
Run everyy pparticle through
g the dynamics
y
function and add noise.
But doesnt work that well because of sample
impoverishment
82
Sample impoverishment
10 of the 100 particles, along with the true Kalman
filter track
track, with variance:
time
83
Resample the prior
In a sampled density representation, the frequency of
samples can be traded off against weight:
s.t.
These
h
new samples
l are a representation
i off the
h same
density.
I.e., make N draws with replacement from the
original set of samples,
samples using
sing the weights
eights as the
probability of drawing a sample.
84
Resampling concentrates samples
85
A practical particle filter with resampling
86
Pictorial view
87
[Isard 1998]
