Module: 3D Vision

Lecture 4: Camera calibration. Pose estimation.

Lecturer: Gloria Haro
Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Camera distortion
• Pose estimation
Camera model

Remember:

x ≡ PX, X ∈ P3 , x ∈ P2 ,
where P is a 3 × 4 matrix, the camera matrix,

P = K [R | t]
    
p11 p12 p13 p14 K11 K12 K13 R11 R12 R13 t1
 p21 p22 p23 p24  =  K21 K22 K23  R21 R22 R23 t2 
p31 p32 p33 p34 K31 K32 K33 R31 R32 R33 t3

▶ K is the calibration matrix containing the internal parameters of

the camera.
▶ R and t contain the external parameters of the camera: position
and orientation of the camera in the reference system of the world
(camera pose).
Calibration and pose estimation

• Calibration
 
? ? ? ?
P= ? ? ? ?  −→ K
? ? ? ?

• Pose estimation
  
K11 K12 K13 ? ? ? ?
P =  K21 K22 K23  ? ? ? ? 
K31 K32 K33 ? ? ? ?
Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Radial distortion
• Pose estimation
Camera resectioning
It uses a calibration object (or already reconstructed 3D points)

Image source: [B. Thibault]

for establishing a set of 3D to 2D point correspondences

Image source: [V. Lepetit]

Camera resectioning
It uses a calibration object (or already reconstructed 3D points)

Image source: [B. Thibault]

for establishing a set of 3D to 2D point correspondences

Image source: [V. Lepetit]

Camera resectioning
From N ≥ 6 known 3D to 2D point correspondences Xi ←→ xi

xi ≡ PXi , i = 1, 2, . . . , N where xi ∈ P2 , Xi ∈ P3 are known.

Remark: As in DLT algorithm is important to previously normalize

data, xi and Xi , i = 1, 2, . . . , N.
Using the same steps as in the DLT algorithm:
 
 T T T
 p1  
0 −wi Xi yi X i 0
p 2 = ,
w i XT 0T −xi XT
 
i i 0
p3

pT
 
1
where P =  pT
2
 .
T
p3 3×4
Camera resectioning
From N ≥ 6 known 3D to 2D point correspondences Xi ←→ xi

xi ≡ PXi , i = 1, 2, . . . , N where xi ∈ P2 , Xi ∈ P3 are known.

Remark: As in DLT algorithm is important to previously normalize

data, xi and Xi , i = 1, 2, . . . , N.
Using the same steps as in the DLT algorithm:
 
 T T T
 p1  
0 −wi Xi yi X i 0
p 2 = ,
w i XT 0T −xi XT
 
i i 0
p3

pT
 
1
where P =  pT
2
 . Or, in short,
T
p3 3×4

Ai p = 0
(p is the 12 × 1 vector of unknowns).
Camera resectioning
Each correspondence Xi ←→ xi
produces 2 equations for the 12 unknowns: Ai p = 0

As P3×4 has 11 degrees of freedom (3 × 4 − 1 scale factor),

we need N ≥ 6 correspondences Xi ←→ xi .

If A denotes the matrix obtained by

 
A1
A =  ... 
AN 2N×12

with Ai the matrix of each correspondence Xi ←→ xi ,

then the system of equations is

Ap = 0 ∈ R2N

The vector p is in the null space of A.

Camera resectioning

Ap = 0
If the correspondences have noise, the identity is not satisfied and in
practice we look for p such that

min ∥Ap∥
such that ∥p∥2 = 1

We compute the solution by SVD :

p ≡ last column of V in the SVD decomposition of A = UDVT .
Camera resectioning

Ap = 0
If the correspondences have noise, the identity is not satisfied and in
practice we look for p such that

min ∥Ap∥
such that ∥p∥2 = 1

We compute the solution by SVD :

p ≡ last column of V in the SVD decomposition of A = UDVT .

In presence of outlier correspondences

we may use the RANSAC version of the DLT algorithm
Camera resectioning

Ap = 0
If the correspondences have noise, the identity is not satisfied and in
practice we look for p such that

min ∥Ap∥
such that ∥p∥2 = 1

We compute the solution by SVD :

p ≡ last column of V in the SVD decomposition of A = UDVT .

In presence of outlier correspondences

we may use the RANSAC version of the DLT algorithm

This estimatePmay be refined by minimizing the geometric reprojection

error: min i d(xi , PXi )2

Complete algorithm: Algorithm 7.1, Hartley-Zisserman book, page 181

Camera resectioning
Once P has been estimated, K , R, and t may be extracted:
1. QR factorization: decomposition of the matrix A into a product
A = QR, where Q is orthogonal, and R is an upper-triangular
matrix.

P = K [R | t] where K is upper-triangular and R is orthogonal

QR decomposition of P3×3 gives K and R.

The ambiguity in the decomposition is removed by requiring that K

has positive diagonal entries.

Remark: RQ factorization may be also used.

Camera resectioning
Once P has been estimated, K , R, and t may be extracted:
2. Case of zero skew:
T
A = P3×3 P3×3 = (KR)(KR)T = KRR T K T = KK T
#" 2 2
A11 A12 A13 fx 0 x0 fx 0 0 fx + x0 x0 y0 x0
" #" #" #
A21 A22 A23 = 0 fy y0 0 fy 0 = x0 y0 fy2 + y02 y0
A31 A32 A33 0 0 1 x0 y0 1 x0 y0 1

• Normalize (P3×3 P3×3

T
)33 = A33 = 1
• Identify parameters of K :

x0 = A13 , y0 = A23
q q
fx = A11 − x02 , fy = A22 − y02
• Extract extrinsics (pose):

[R|t] = K −1 P
Remark:R may not be necessarily orthogonal, thus we further impose it.
Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Radial distortion
• Pose estimation
Calibration from a planar pattern (Zhang’s method)

Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22 (11), 2000.
Calibration from a planar pattern (Zhang’s method)
Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection:
Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection: H
      
λu h11 h12 h13 x x
 λ v  =  h21 h22 h23   y  = [h1 h2 h3 ]  y 
λ h31 h32 h33 1 1
| {z }
H
Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection: H
      
λu h11 h12 h13 x x
 λ v  =  h21 h22 h23   y  = [h1 h2 h3 ]  y 
λ h31 h32 h33 1 1
| {z }
H

2. Flat object at plane Z = 0 projected to the image

Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection: H
      
λu h11 h12 h13 x x
 λ v  =  h21 h22 h23   y  = [h1 h2 h3 ]  y 
λ h31 h32 h33 1 1
| {z }
H

2. Flat object at plane Z = 0 projected to the image

 
  x
αu
 α v  = K r1 r2 r3 t  y  =
h i 
 0 
α
| {z }
R 1
Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22 (11), 2000.
Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection: H
      
λu h11 h12 h13 x x
 λ v  =  h21 h22 h23   y  = [h1 h2 h3 ]  y 
λ h31 h32 h33 1 1
| {z }
H

2. Flat object at plane Z = 0 projected to the image

 
  x  
αu x
y
h i 
 α v  = K r1 r2 r3 t   = [K r1 K r2 K t]  y 
| {z }  0 
α R 1
1
Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22 (11), 2000.
Calibration from a planar pattern (Zhang’s method)
1. Relation of a flat object and its image projection: H
      
λu h11 h12 h13 x x
 λ v  =  h21 h22 h23   y  = [h1 h2 h3 ]  y 
λ h31 h32 h33 1 1
| {z }
H

2. Flat object at plane Z = 0 projected to the image

 
  x  
αu x
y
h i 
 α v  = K r1 r2 r3 t   = [K r1 K r2 K t]  y 
| {z }  0  |
α 1
{z }
R 1 ∼[h h h ] 1 2 3

Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22 (11), 2000.
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated,
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated, K, r1 , r2 and t are unknowns
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated, K, r1 , r2 and t are unknowns
BUT, we know that r1 and r2 are columns of an orthogonal matrix

r1T r2 = 0

(1)
r1T r1 = r2T r2 = 1
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated, K, r1 , r2 and t are unknowns
BUT, we know that r1 and r2 are columns of an orthogonal matrix

r1T r2 = 0

(1)
r1T r1 = r2T r2 = 1

Isolating
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated, K, r1 , r2 and t are unknowns
BUT, we know that r1 and r2 are columns of an orthogonal matrix

r1T r2 = 0

(1)
r1T r1 = r2T r2 = 1

Isolating
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

Applying (1)
−T −1
hT

1K K h2 = 0
T −T −1 −T −1
h1 K K h1 = hT 2K K h2
Calibration from a planar pattern

[K r1 K r2 K t] ∼ [h1 h2 h3 ]
H can be estimated, K, r1 , r2 and t are unknowns
BUT, we know that r1 and r2 are columns of an orthogonal matrix

r1T r2 = 0

(1)
r1T r1 = r2T r2 = 1

Isolating
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

Applying (1)
−T −1
hT

1K K h2 = 0
T −T −1 −T −1
h1 K K h1 = hT 2K K h2
where ω = (KK T )−1 = K −T K −1 is the Image of the Absolute Conic
Calibration from a planar pattern

We have
hT

1 ω h2 = 0
h1 ω h1 = hT
T
2 ω h2

ω = (KK T )−1 = K −T K −1 is a symmetric matrix

 
ω11 ω12 ω13
ω =  ω12 ω22 ω23 
ω13 ω23 ω33
Calibration from a planar pattern

We have
hT

1 ω h2 = 0
h1 ω h1 = hT
T
2 ω h2

ω = (KK T )−1 = K −T K −1 is a symmetric matrix

 
ω11 ω12 ω13
ω =  ω12 ω22 ω23  −→ Ω = (ω11 ω12 ω13 ω22 ω23 ω33 )T
ω13 ω23 ω33
Calibration from a planar pattern

We have
hT

1 ω h2 = 0
−→ two constraints on ω
h1 ω h1 = hT
T
2 ω h2

ω = (KK T )−1 = K −T K −1 is a symmetric matrix

 
ω11 ω12 ω13
ω =  ω12 ω22 ω23  −→ Ω = (ω11 ω12 ω13 ω22 ω23 ω33 )T
ω13 ω23 ω33
Calibration from a planar pattern
  
ω11 ω12 ω13 h1j
hT T
i ωhj = (h1i h2i h3i )  ω12 ω22 ω23   h2j  = VijT Ω
ω13 ω23 ω33 h3j
where VijT =
(h1i h1j , h1i h2j + h2i h1j , h1i h3j + h3i h1j , h2i h2j , h2i h3j + h3i h2j , h3i h3j )
Calibration from a planar pattern
  
ω11 ω12 ω13 h1j
hT T
i ωhj = (h1i h2i h3i )  ω12 ω22 ω23   h2j  = VijT Ω
ω13 ω23 ω33 h3j
where VijT =
(h1i h1j , h1i h2j + h2i h1j , h1i h3j + h3i h1j , h2i h2j , h2i h3j + h3i h2j , h3i h3j )
Then
T

V12 Ω=0
T T
(V11 − V22 )Ω = 0
Calibration from a planar pattern
  
ω11 ω12 ω13 h1j
hT T
i ωhj = (h1i h2i h3i )  ω12 ω22 ω23   h2j  = VijT Ω
ω13 ω23 ω33 h3j
where VijT =
(h1i h1j , h1i h2j + h2i h1j , h1i h3j + h3i h1j , h2i h2j , h2i h3j + h3i h2j , h3i h3j )
Then
T

V12 Ω=0
T T
(V11 − V22 )Ω = 0
2 equations for 6 unknowns −→ we need n = 3 views (homographies) to
find ω
We have to solve
V2n×6 Ω = 0
Calibration from a planar pattern
  
ω11 ω12 ω13 h1j
hT T
i ωhj = (h1i h2i h3i )  ω12 ω22 ω23   h2j  = VijT Ω
ω13 ω23 ω33 h3j
where VijT =
(h1i h1j , h1i h2j + h2i h1j , h1i h3j + h3i h1j , h2i h2j , h2i h3j + h3i h2j , h3i h3j )
Then
T

V12 Ω=0
T T
(V11 − V22 )Ω = 0
2 equations for 6 unknowns −→ we need n = 3 views (homographies) to
find ω
We have to solve
V2n×6 Ω = 0
Remark: if we use additional constraints on K

• zero skew −→ we need n = 2 views

• zero skew and known principal point −→ we need n = 1 view
Calibration from a planar pattern

We have to solve
VΩ=0
In practice we solve:
min ∥V Ω∥2
Ω

such that ∥Ω∥2 = 1

Solution: Ω is the singular vector associated to the smallest singular
value (last column of Ū, where V = UD Ū T )
SVD
Calibration from a planar pattern

We have to solve
VΩ=0
In practice we solve:
min ∥V Ω∥2
Ω

such that ∥Ω∥2 = 1

Solution: Ω is the singular vector associated to the smallest singular
value (last column of Ū, where V = UD Ū T )
SVD
SVD rearrangement Cholesky
V −−→ Ω −−−−−−−−→ ω = K −T K −1 −−−−−→ K
Calibration from a planar pattern

How to find R and t (pose) from K ?

Calibration from a planar pattern

How to find R and t (pose) from K ?

We have
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

and since ∥r1 ∥ = ∥r2 ∥ = 1,

then,
Calibration from a planar pattern

How to find R and t (pose) from K ?

We have
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

and since ∥r1 ∥ = ∥r2 ∥ = 1,

then,
K −1 h1 K −1 h2
r1 = , r2 = , r3 = r1 × r2
∥K −1 h1 ∥ ∥K −1 h2 ∥

K −1 h3
t=
∥K −1 h1 ∥
Calibration from a planar pattern

How to find R and t (pose) from K ?

We have
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

and since ∥r1 ∥ = ∥r2 ∥ = 1,

then,
K −1 h1 K −1 h2
r1 = , r2 = , r3 = r1 × r2
∥K −1 h1 ∥ ∥K −1 h2 ∥

K −1 h3
t=
∥K −1 h1 ∥

But, not guarantee (noise) that R = (r1 r2 r3 ) is an orthogonal matrix.

Calibration from a planar pattern

How to find R and t (pose) from K ?

We have
[r1 r2 t] ∼ K −1 h1 K −1 h2 K −1 h3
 

and since ∥r1 ∥ = ∥r2 ∥ = 1,

then,
K −1 h1 K −1 h2
r1 = , r2 = , r3 = r1 × r2
∥K −1 h1 ∥ ∥K −1 h2 ∥

K −1 h3
t=
∥K −1 h1 ∥

But, not guarantee (noise) that R = (r1 r2 r3 ) is an orthogonal matrix.

Then, we impose it:
R̃ = U Ū T
where R = UD Ū T .
SVD
Calibration from a planar pattern

Steps of the calibration algorithm

• Estimate the n homographies that relate the planar pattern and the
n images.
• Build matrix V .
• Find ω by solving the SVD of V .
• Extract calibration parameters, K , by Cholesky factorization of ω.
• (Optional) Extract pose, R and t.
Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Camera distortion
• Pose estimation
Image of the absolute conic

The absolute conic, Ω∞ , is a conic on the plane at infinity, Π∞ .

It consists of points X = (x1 , x2 , x3 , 0)T such that x12 + x22 + x32 = 0.
| {z }
d
T
In matricial form: d I d = 0, then

Ω∞ = I3×3

Property: The absolute conic remains fixed by a similarity.

Image of the absolute conic

Remark: Ω∞ is defined in an Euclidean frame, then if we change

coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.
Image of the absolute conic

Remark: Ω∞ is defined in an Euclidean frame, then if we change

coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.

Let us see it:

Image of the absolute conic

Remark: Ω∞ is defined in an Euclidean frame, then if we change

coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.

Let us see it:  

d
Points on Ω∞ :
0
Image of the absolute conic
Remark: Ω∞ is defined in an Euclidean frame, then if we change
coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.

Let us see it:  

d
Points on Ω∞ :
0
Their image (camera projection) is:
   
d d
P = K [R t] = KR d = x
0 0
Image of the absolute conic
Remark: Ω∞ is defined in an Euclidean frame, then if we change
coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.

Let us see it:  

d
Points on Ω∞ :
0
Their image (camera projection) is:
   
d d
P = K [R t] = |{z}
KR d = x
0 0
H

H = KR is the (2D) homography that relates Π∞ and the image plane.

Image of the absolute conic
Remark: Ω∞ is defined in an Euclidean frame, then if we change
coordinates to a projective frame the image of Ω∞ will be, in general, a
finite conic represented by matrix ω.

Let us see it:  

d
Points on Ω∞ :
0
Their image (camera projection) is:
   
d d
P = K [R t] = |{z}
KR d = x
0 0
H

H = KR is the (2D) homography that relates Π∞ and the image plane.

Remember how a conic is transformed by a homography: C ′ = H −TCH −1
Then,

(KR)−T I (KR)−1 = K −T RR −1 −1 −T −1 T −1
| {z } K = K K = (KK ) = ω
I
Image of the absolute conic
Remarks:
• ω depends only on the internal parameters of the camera, K .
• ω allows to measure angles in 3D world from image points.
Image of the absolute conic
Remarks:
• ω depends only on the internal parameters of the camera, K .
• ω allows to measure angles in 3D world from image points.
If d1 and d2 are the directions of two lines in R3 , the angle θ they
form satisfies:
dT
1 d2 dT
1 Ω∞ d2
cos θ = p p =p p
T T
d1 d1 d2 d2 d1 Ω∞ d1 dT
T
2 Ω∞ d2
Image of the absolute conic
Remarks:
• ω depends only on the internal parameters of the camera, K .
• ω allows to measure angles in 3D world from image points.
If d1 and d2 are the directions of two lines in R3 , the angle θ they
form satisfies:
dT
1 d2 dT
1 Ω∞ d2
cos θ = p p =p p
T T
d1 d1 d2 d2 d1 Ω∞ d1 dT
T
2 Ω∞ d2

Since their points at infinity are (dT T T T

1 , 0) and (d2 , 0) , their image
projections (vanishing points) are:

x1 = KRd1 and x2 = KRd2

Image of the absolute conic
Remarks:
• ω depends only on the internal parameters of the camera, K .
• ω allows to measure angles in 3D world from image points.
If d1 and d2 are the directions of two lines in R3 , the angle θ they
form satisfies:
dT
1 d2 dT
1 Ω∞ d2
cos θ = p p =p p
T T
d1 d1 d2 d2 d1 Ω∞ d1 dT
T
2 Ω∞ d2

Since their points at infinity are (dT T T T

1 , 0) and (d2 , 0) , their image
projections (vanishing points) are:

x1 = KRd1 and x2 = KRd2

We have that:
−T
dT T
i Ω∞ dj = xi (KR) Ω∞ (KR)−1 xj = xT
i ω xj , for i, j ∈ {1, 2}
Image of the absolute conic

Then,
xT
1 ωx2
cos θ = p p
x1 ωx1 xT
T
2 ωx2
where x1 and x2 are two vanishing points.
Image of the absolute conic

Then,
xT
1 ωx2
cos θ = p p
x1 ωx1 xT
T
2 ωx2
where x1 and x2 are two vanishing points.

In particular, for orthogonal directions

xT
1 ω x2 = 0

It provides a constraint on ω (K )
Image of the absolute conic

Then,
xT
1 ωx2
cos θ = p p
x1 ωx1 xT
T
2 ωx2
where x1 and x2 are two vanishing points.

In particular, for orthogonal directions

xT
1 ω x2 = 0

It provides a constraint on ω (K ) −→ Useful for self-calibration!

Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Camera distortion
• Pose estimation
Camera distortion
Pinhole cameras: linear model of the imaging process
Real (non-pinhole) cameras: lense distortion (non-linear), usually radial
distortion

short vs long focal length

Image source: [Hartley and Zisserman 2004]

Camera distortion
Radial distortion

Barrel and Pincushion examples

Coordinates in the observed image are displaced away, barrel distortion,
or towards, pincushion distortion, the image center by an amount
proportional to their radial distance.

xd = L(r ) x = (1 + k1 r + k2 r 2 + k3 r 3 + ...) x

yd = L(r ) y = (1 + k1 r + k2 r 2 + k3 r 3 + ...) y
p
where r = x2 + y2
Image source: [Szeliski 2010]
Camera distortion
How to estimate the distortion parameters:
• Plumb-line method, adjust parameters ki so that images of
straight lines become straight in the image.

• Parameters ki are estimated jointly with those of P by

iterative minimization of a geometric error.
XX
min ∥xij − f (k1 , k2 , ..., kn , K , Ri , ti ,Xj )∥22
i j
Outline

• Calibration
• Camera resectioning
• Calibration from a planar pattern
• Image of the absolute conic
• Camera distortion
• Pose estimation
Pose estimation
Estimate camera pose from n 3D-to-2D point correspondences.

If we assume known calibration parameters, K , the problem is called

Perspective-n-Point (PnP) problem
  
K11 K12 K13 ? ? ? ?
P =  K21 K22 K23  ? ? ? ? 
K31 K32 K33 ? ? ? ?

Only R and t (6 dof) need to be determined.

Image source: [https://docs.opencv.org]
PnP methods
PnP problem: Estimate the absolute pose of a calibrated camera
Data: n 3D-2D point correspondences + camera internal parameters K
Goal: estimate camera pose, R and t

Applications:
localization of robots and object manipulation, augmented reality, ...

Collet et al. Object Recognition and Full Pose Registration from a Single Image for Robotic Manipulation. In Proc. of IEEE Int. Conf. on
Robotics and Automation (ICRA), 2009.
Müller et al. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. International journal of computer assisted
radiology and surgery, 2013.
PnP methods
Different methods (according to n):
• P3P [1,2]
• P4P [3,4] Closed-form solutions, very efficient but very sensitive to noise
• P5P [4]
• PnP methods
• iterative (geometric or algebraic errors) −→ accurate but sensitive to
local minima
• non-iterative −→ no local minima
▶ EPnP [5] (1st efficient non-iterative solution for large n)
▶ RPnP [6]
▶ ASPnP [7]
▶ OPnP [8]
▶ Robustified EPnP [9] −→ fast, accurate, robust to outliers
[1] D. DeMenthon, L. Davis. Exact and approximate solutions of the perspective-three-point problem. PAMI 14(11), 1992.
[2] L. Kneip, D. Scaramuzza, R. Siegwart. A novel parametrization of the perspective-three-point problem for a direct computation of
absolute camera position and orientation. CVPR, 2011.
[3] M.A. Fischler and R.C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and
automated cartography. Communications of the ACMA, 1981.
[4] B. Triggs. Camera pose and calibration from 4 or 5 known 3D points. ICCV, 1999.
[5] F. Moreno-Noguer, V. Lepetit, P. Fua. Accurate non-iterative o(n) solution to the PnP problem. ICCV 2007.
[6] S. Li, C. Xu, M. Xie. A robust o(n) solution to the perspective-n-point problem. PAMI 34(7), 2012.
[7] Y. Zheng, S. Sugimoto, M. Okutomi. AsPnP: An accurate and scalable solution to the perspective-n-point problem. Trans. on
Information and Systems, 96(7), 2013.
[8] Y. Zheng, Y. Kuang, S. Sugimoto, K. Aström, M. Okutomi. Revisiting the PnP problem: A fast, general and optimal solution. ICCV,
2013.
[9] L. Ferraz, X. Binefa, F. Moreno-Noguer. Very fast solution to the PnP problem with algebraic outlier rejection, CVPR 2014.
Pose estimation with deep learning

Posenet: estimates 3D position and orientation of the camera given a

single monocular image taken from a large previously explored scene.
A. Kendall, M. Grimes, R. Cipolla. PoseNet: A convolutional network for real-time 6-DOF camera relocalization, ICCV, 2015.
Pose estimation: camera orientation

Vanishing lines and pose orientation

• Horizon
• Camera roll angle (tilted toward right): horizon angle
• Camera pitch (tilt) angle (looking down): horizon offset
Camera parameters and vanishing points

A. Criminisi, I. Reid and A. Zisserman, Single view metrology, IEEE International Conference on Computer Vision (ICCV), 1999.
Pose estimation with deep learning

Horizon lines in the wild

Workman S., Zhai M., Jacobs N. Horizon Lines in the Wild. In: British Machine Vision Conference (BMVC), 2016
Camera calibration with deep learning
Estimation of roll, pitch and focal length

• Calibration
• Image retrieval

• Geometrically-consistent object transfer

• Virtual object insertion

Y. Hold-Geoffroy, K. Sunkavalli, J. Eisenmann, M. Fisher,E. Gambaretto, S. Hadap, and J.-F. Lalonde, A Perceptual Measure for Deep
Single Image Camera Calibration, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Camera calibration with deep learning

Estimation of roll, pitch, focal length and radial distortion

M. López-Antequera, R. Marı́, P. Gargallo, Y. Kuang, J. Gonzalez-Jimenez, G. Haro, Deep Single Image Camera Calibration with Radial
Distortion, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Summary

We have seen:
• Camera calibration
• Image of the absolute conic
• Pose estimation
Summary

We have seen:
• Camera calibration
• Image of the absolute conic
• Pose estimation

Next class:
• The geometry of two views
References

[Hartley and Zisserman 2004] R.I. Hartley and A. Zisserman, Multiple

View Geometry in Computer Vision, Cambridge University Press, 2004.

[Szeliski 2010] R. Szeliski, Computer Vision: Algorithms and

Applications, Springer, 2010.