CS 294-167 Spring 2022

Geometry and Learning for 3D Vision

Yi Ma

UC Berkeley
 Lecture 2

Rigid-Body Motion
and
Geometry of Image Formation

MASKS © 2004 Invitation to 3D vision

 OUTLINE

3-D EUCLIDEAN SPACE & RIGID-BODY MOTION

• Coordinates and coordinate frames
• Rigid-body motion and homogeneous coordinates

GEOMETRIC MODELS OF IMAGE FORMATION

• Pinhole camera model

CAMERA INTRINSIC PARAMETERS

• From metric to pixel coordinates

IMAGES OF BASIC GEOMETRIC ELEMENTS

• Points, lines, planes etc.

MASKS © 2004 Invitation to 3D vision

 3-D EUCLIDEAN SPACE - Cartesian Coordinate Frame

Standard base vectors:

Coordinates of a point in space:

MASKS © 2004 Invitation to 3D vision

 3-D EUCLIDEAN SPACE - Vectors

A “free” vector is defined by a pair

of points :

Coordinates of the vector :

MASKS © 2004 Invitation to 3D vision

 3-D EUCLIDEAN SPACE – Inner Product and Cross Product

Inner product between two vectors:

Cross product between two vectors:

MASKS © 2004 Invitation to 3D vision

 RIGID-BODY MOTION – Rotation

Rotation matrix:

Coordinates are related by:

Special Orthogonal Group:

SO(3) = R 2 R3⇥3 | RT R = RRT = I, det R = 1

MASKS © 2004 Invitation to 3D vision

 RIGID-BODY MOTION – Rotation

Rotation matrix:

• Preserve inner product (or vector length)

kRvk22 = v T RT Rv = kvk22 , 8v 2 R3
<latexit sha1_base64="uVhzQ/e2mVLjQAK5mWRYs7XyIZc=">AAACKnicbVDLSgMxFM3UV62vqks3wSK4kDJTRQURqm5c1tIXNG3JpGkNZjJDkhko036PG3/FTRdKceuHmLaz0NZLAodz7uWee9yAM6Vte2KlVlbX1jfSm5mt7Z3dvez+QU35oSS0Snzuy4aLFeVM0KpmmtNGICn2XE7r7svDVK9HVCrmi4oeBLTl4b5gPUawNlQne4eGZRihYafQLsBbGLUrZfMiA9Ewoc8guoGo50vMOYwgYgIiD+tn143Lo/Z5J5uz8/as4DJwEpADSZU62THq+iT0qNCEY6Wajh3oVoylZoTTUQaFigaYvOA+bRoosEdVK56dOoInhulCY8Z8oeGM/T0RY0+pgeeazqlHtahNyf+0Zqh7162YiSDUVJD5ol7IofbhNDfYZZISzQcGYCKZ8QrJM5aYaJNuxoTgLJ68DGqFvHOZt58ucsX7JI40OALH4BQ44AoUwSMogSog4BW8gw/wab1ZY2tifc1bU1Yycwj+lPX9AyVJpLU=</latexit>

• Preserve cross product (or orientation)

Ru ⇥ Rv = R(u ⇥ v)
<latexit sha1_base64="0Ts+odF7V58K0g98IqJfFXTMbYU=">AAACBXicbVDLSgMxFL1TX7W+Rl3qIliEuikzIupGKLpxWYt9QDuUTJppQzOZIckUSunGjb/ixoUibv0Hd/6NaTuIth4InHvOvdzc48ecKe04X1ZmaXlldS27ntvY3NresXf3aipKJKFVEvFINnysKGeCVjXTnDZiSXHoc1r3+zcTvz6gUrFI3OthTL0QdwULGMHaSG37sJKglmYhVagyQFeoUvipBydtO+8UnSnQInFTkocU5bb92epEJAmp0IRjpZquE2tvhKVmhNNxrpUoGmPSx13aNFRgs8cbTa8Yo2OjdFAQSfOERlP198QIh0oNQ990hlj31Lw3Ef/zmokOLr0RE3GiqSCzRUHCkY7QJBLUYZISzYeGYCKZ+SsiPSwx0Sa4nAnBnT95kdROi+550bk7y5eu0ziycABHUAAXLqAEt1CGKhB4gCd4gVfr0Xq23qz3WWvGSmf24Q+sj2/U/5be</latexit>

• The useful “hat” operator:

<latexit sha1_base64="Orb01g+ABfIg/pHEAcAeMQfr2tY=">AAACM3icbVDLSgMxFM34rPVVdekmWIQKUmasqCBC0Y24sda+oNOWTJppQzOZIckoZeg/ufFHXAjiQhG3/oOZtoi2HgicnHsv99zjBIxKZZovxszs3PzCYmIpubyyurae2tisSD8UmJSxz3xRc5AkjHJSVlQxUgsEQZ7DSNXpXcT16h0Rkvq8pPoBaXiow6lLMVJaaqWu7HvaJl2komI4gGewCH8E/S82S/YptF1fIMZgCG3Koe0h1XWcqDho5vbjfq3dXmdye61U2syaQ8BpYo1JGoxRaKWe7LaPQ49whRmSsm6ZgWpESCiKGRkk7VCSAOEe6pC6phx5RDai4c0DuKuVNtTO9OMKDtXfExHypOx7ju6MDcvJWiz+V6uHyj1pRJQHoSIcjxa5IYPKh3GAsE0FwYr1NUFYUO0V4i4SCCsdc1KHYE2ePE0qB1nrKGveHKbz5+M4EmAb7IAMsMAxyINLUABlgMEDeAZv4N14NF6ND+Nz1DpjjGe2wB8YX99UdajO</latexit>

c = Rb
Ru uRT 8u 2 R3 , R 2 SO(3)
<latexit sha1_base64="3yDEHYVhmAU+jiPS/G4RCxy2Nz8=">AAACYHicbVFNaxsxENVu0tZxk8ZpbsllqCn0EMxuWtpCKYT00mOyxEnA65hZWbZFtNpFGtmYZf9kbz300l9SrWNCPjogeHozj3l6ykolLUXR7yDc2Hzx8lVrq/16e+fNbmfv7aUtnOGizwtVmOsMrVBSiz5JUuK6NALzTImr7PZH07+aC2NloS9oWYphjlMtJ5IjeWrUWaQLORYzpCpxdTKH75DAPeXqOaTfIE3kdEZoTLForsnNBTxSQSN7IIL50Uo2KQwqBe7ID6RSQ5ojzbKsSuqbj6NON+pFq4LnIF6DLlvX2ajzKx0X3OVCE1do7SCOShpWaEhyJep26qwokd/iVAw81JgLO6xWAdXw3jNj8H780QQr9qGiwtzaZZ75ycajfdpryP/1Bo4mX4eV1KUjofndoolTQAU0acNYGsFJLT1AbqT3CnyGBjn5P2n7EOKnT34OLo978ededP6pe3K6jqPFDtk79oHF7As7YT/ZGeszzv4EG8F2sBP8DVvhbrh3NxoGa80+e1ThwT/SV7SK</latexit>

Proof: RuRv
c c
uv ) RT RuRv
= Rb bv, 8u, v 2 R3
=u

MASKS © 2004 Invitation to 3D vision

 Euclidean MOTION – Reflection

Reflection matrix:
P

Pr

T z
<latexit sha1_base64="XIunPkzd6XHYf6L4t+X2AMoJ4Dg=">AAACAnicbVDLSgMxFM3UV62vUVfiJliEClpmRFQQoehGd7X0BZ2xZNJMG5rJDElGKENx46+4caGIW7/CnX9j2s5CWw9cODnnXnLv8SJGpbKsbyMzN7+wuJRdzq2srq1vmJtbdRnGApMaDlkomh6ShFFOaooqRpqRICjwGGl4/euR33ggQtKQV9UgIm6Aupz6FCOlpba5U7mvVuAlvD2EzgV0OkQVKgf6fWS3zbxVtMaAs8ROSR6kKLfNL6cT4jggXGGGpGzZVqTcBAlFMSPDnBNLEiHcR13S0pSjgEg3GZ8whPta6UA/FLq4gmP190SCAikHgac7A6R6ctobif95rVj5525CeRQrwvHkIz9mUIVwlAfsUEGwYgNNEBZU7wpxDwmElU4tp0Owp0+eJfXjon1atO5O8qWrNI4s2AV7oABscAZK4AaUQQ1g8AiewSt4M56MF+Pd+Ji0Zox0Zhv8gfH5A9vmk+A=</latexit>

R R = I, det(R) = 1
<latexit sha1_base64="c3m5A4wyQBtE1KcugLdzdmOG3iw=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KomIeix68diC/YA2lM120q7dbMLuRqihv8CLB0W8+pO8+W/ctjlo64OBx3szzMwLEsG1cd1vZ2V1bX1js7BV3N7Z3dsvHRw2dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nfqtR1Sax/LejBP0IzqQPOSMGivVn3qlsltxZyDLxMtJGXLUeqWvbj9maYTSMEG17nhuYvyMKsOZwEmxm2pMKBvRAXYslTRC7WezQyfk1Cp9EsbKljRkpv6eyGik9TgKbGdEzVAvelPxP6+TmvDaz7hMUoOSzReFqSAmJtOvSZ8rZEaMLaFMcXsrYUOqKDM2m6INwVt8eZk0zyveZcWtX5SrN3kcBTiGEzgDD66gCndQgwYwQHiGV3hzHpwX5935mLeuOPnMEfyB8/kD6ueNAw==</latexit>

Just like rotation except not 2 3

<latexit sha1_base64="kI/EgVEAI0c+6LyskKrPJyMIB5o=">AAACNnicbVBNS8QwEE39tn6tevQSXBQvLq2IehS9eBEU3A/YliXNTneDaVqSqbgUf5UXf4c3Lx4U8epPMF0XUdeBIS9v3iQzL8qkMOh5T87E5NT0zOzcvLuwuLS8Ullda5g01xzqPJWpbkXMgBQK6ihQQivTwJJIQjO6Pi3rzRvQRqTqCgcZhAnrKRELztBSncp5ICHGthtE0BOqSBhqcXvn7vp0m3plBoFbnv4PPLy7Aajutz7QotfHsFOpejVvGHQc+CNQJaO46FQeg27K8wQUcsmMaftehmHBNAouwb6bG8gYv2Y9aFuoWAImLIZr39Ety3RpnGqbCumQ/dlRsMSYQRJZpZ2zb/7WSvK/WjvH+CgshMpyBMW/PopzSTGlpYe0KzRwlAMLGNfCzkp5n2nG0TrtWhP8vyuPg8ZezT+oeZf71eOTkR1zZINskh3ik0NyTM7IBakTTu7JE3khr86D8+y8Oe9f0gln1LNOfoXz8Qm1naXp</latexit>

1 0 0
preserving orientation: 4 0 1 05
Ru ⇥ Rv = R(u ⇥ v)
<latexit sha1_base64="pqvmQDjgjU5HnNQ4WET5Qqjk72s=">AAACB3icbVBNS8NAEJ3Ur1q/oh4FWSxCPVgSEfUiFL14rMV+QBvKZrtpl242YXdTKKU3L/4VLx4U8epf8Oa/cdsG0eqDgcd7M8zM82POlHacTyuzsLi0vJJdza2tb2xu2ds7NRUlktAqiXgkGz5WlDNBq5ppThuxpDj0Oa37/euJXx9QqVgk7vQwpl6Iu4IFjGBtpLa9X0lQS7OQKlQZoEt0jCqFb2Vw1LbzTtGZAv0lbkrykKLctj9anYgkIRWacKxU03Vi7Y2w1IxwOs61EkVjTPq4S5uGCmz2eKPpH2N0aJQOCiJpSmg0VX9OjHCo1DD0TWeIdU/NexPxP6+Z6ODCGzERJ5oKMlsUJBzpCE1CQR0mKdF8aAgmkplbEelhiYk20eVMCO78y39J7aTonhWd29N86SqNIwt7cAAFcOEcSnADZagCgXt4hGd4sR6sJ+vVepu1Zqx0Zhd+wXr/AqHglz8=</latexit>

0 0 1

Orthogonal Group:
O(3) = R 2 R3⇥3 | RT R = I, det(R) = ±1
<latexit sha1_base64="d83kzyVqPvR1dBPxoDjm4n4NRNE=">AAACO3icbVBNa9tAFFylX4n75SbHXh41BReKkeLS5BIIyaU9NRFxEvA6ZrV6spesVmL3qWCE/lcv+RO55dJLDw2l196zdnxokw4sDDPvsW8mKbVyFIZXwcqDh48eP1ldaz199vzFy/ar9WNXVFbiQBa6sKeJcKiVwQEp0nhaWhR5ovEkOd+f+ydf0TpVmCOalTjKxcSoTElBXhq34y/d/jvYAa4xI15DzJUBnguaJkkdN2d1n5PK0UG/8bJKIT47iv385/fAU6RuvFguc4iAWzWZEm/G7U7YCxeA+yRakg5b4mDcvuRpIascDUktnBtGYUmjWlhSUmPT4pXDUshzMcGhp0b4e0b1InsDb72SQlZY/wzBQv17oxa5c7M88ZPzVO6uNxf/5w0ryrZHtTJlRWjk7UdZpYEKmBcJqbIoSc88EdIqfyvIqbBCkq+75UuI7ka+T443e9HHXnj4obO7t6xjlb1mb1iXRWyL7bJP7IANmGTf2Hf2k10HF8GP4Ffw+3Z0JVjubLB/EPy5AZCRqjk=</latexit>

MASKS © 2004 Invitation to 3D vision

 Euclidean MOTION – Object-level Symmetry

Rotational Symmetry Reflective Symmetry

MASKS © 2004 Invitation to 3D vision

 RIGID-BODY MOTION – Rotation and Translation

Coordinates are related by:

Velocities are related by:

MASKS © 2004 Invitation to 3D vision

 RIGID-BODY MOTION – Homogeneous Coordinates

3-D coordinates are related by:

Homogeneous coordinates:
2 3
2 3
<latexit sha1_base64="cWdCVZwGJGxwmIMUvBcHAfs8RsU=">AAACp3iclVFLb9QwEHbCoyW8FjhysViQ4LJKSlW4IFVwQVz63HSldYgmzmTXquME26FdRflr/Ahu/Jt6d1OpDzgwku3P3zfjGc9ktRTGhuEfz79z9979jc0HwcNHj588HTx7Hpuq0RzHvJKVnmRgUAqFYyusxEmtEcpM4kl2+mWpn/xEbUSlju2ixqSEmRKF4GAdlQ5+sRLsPCvauKOfKJNY2GnAMpwJ1TpFi/MuiNOIMRqnW6v9fcBQ5ZciZVrM5jah7EcDeX8Drauzngn+PwF1R/iPLELR9YNZe9h9304Hw3AUrozeBlEPhqS3/XTwm+UVb0pUlkswZhqFtU1a0FZwiV3AGoM18FOY4dRBBSWapF31uaNvHJPTotJuKUtX7NWIFkpjFmXmPJc1mpvakvybNm1s8TFphaobi4qvExWNpLaiy6HRXGjkVi4cAK6Fq5XyOWjg1o02cE2Ibn75Noi3RtHOKDzYHu5+7tuxSV6SV+QticgHsku+kn0yJtx77X3zjrxj/52/58f+ZO3qe33MC3LNfLgAqdXOqQ==</latexit>

V1
V1 6 V2 7
V = 4 V2 5 ! V=6 7
4 V3 5 2 R
4

V3
0

Homogeneous coordinates/velocities are related by:

MASKS © 2004 Invitation to 3D vision

IMAGE FORMATION – Projection of 3D to 2D

Parallel Projection, Chinese Paintings, Qing Dynasty

 IMAGE FORMATION – Shifting Perspective (Scroll Painting)

Along the River During the Qingming Festival, Zhang Zeduan Song Dynasty
IMAGE FORMATION – Projection of 3D to 2D

Frescos, 1BD Pompeii Renaissance, Raphael, 1518

Image courtesy of C. Taylor

 IMAGE FORMATION – History of Perspective Imaging

MASKS © 2004
 IMAGE FORMATION – Perspective Imaging

Mathematical
Model of a Camera

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Pinhole Camera Model

Pinhole

Frontal
pinhole

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Pinhole Camera Model

2-D coordinates

Homogeneous coordinates

MASKS © 2004 Invitation to 3D vision

 CAMERA PARAMETERS – Pixel Coordinates

metric
coordinates

Linear transformation

pixel
coordinates

MASKS © 2004 Invitation to 3D vision

 CAMERA PARAMETERS – Calibration Matrix and Camera Model

Pinhole camera Pixel coordinates

Calibration matrix
(intrinsic parameters)
Projection matrix

Camera model

MASKS © 2004 Invitation to 3D vision

 CAMERA PARAMETERS – Radial Distortion

Nonlinear transformation along the radial direction

Distortion correction: make lines straight

MASKS © 2004 Invitation to 3D vision

 CAMERA PARAMETERS – Radial Distortion Calibration

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Image of a Point

Homogeneous coordinates of a 3-D point

Homogeneous coordinates of its 2-D image

Projection of a 3-D point to an image plane

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Images of Points or Junctions

Learning to Parse Wireframes in Images of Man-Made Environments, Kun Huang and Yi Ma, CVPR 2018.
 IMAGE FORMATION – Image of a Line

Homogeneous representation of a 3-D line

Homogeneous representation of its 2-D image

Projection of a 3-D line to an image plane

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Images of Lines

Learning to Parse Wireframes in Images of Man-Made Environments, Kun Huang and Yi Ma, CVPR 2018.
IMAGE FORMATION – Detection of Points and Lines

LSD AFM Wireframe LCNN (Ours) Ground truth

End-to-End Wireframe Parsing, Yichao Zhou, Haozhi Qi, Yi Ma, ICCV 2019.
 IMAGE FORMATION – Vanishing Points from Perspective
2 3 <latexit sha1_base64="q2gmiLZwwH8lEkqc3a6K+0IzTqg=">AAAConicbVFta9RAEN7Etza+nfqxH1w8BUE4klrUL0JRBPGDtuqlp7cxbDaTu6WbTdidiEfID+vf6Df/jZtrCrW9gRkenmdmdnYmq5W0GIZ/Pf/a9Rs3b21tB7fv3L13f/TgYWyrxgiYikpVZpZxC0pqmKJEBbPaAC8zBUfZ8fteP/oNxspKf8dVDUnJF1oWUnB0VDo6YSXHZVa0sy6t6AvKlKvNOT2n446+dSQUOA9YBgupW6cY+acLZozRH85/UheigIHOzzXKjFwsMaEXO25sEqeRq47T3XV82bcKN7diUg9TZe3X7tdeOhqHk3Bt9CqIBjAmgx2ko1OWV6IpQaNQ3Np5FNaYtNygFAq6gDUWai6O+QLmDmpegk3a9Yo7+swxOS0q41wjXbMXK1peWrsqM5fZz2gvaz25SZs3WLxJWqnrBkGLs4eKRlGsaH8vmksDAtXKAS6MdLNSseSGC3RXDdwSostfvgri3Un0ahIe7o333w3r2CI75Al5TiLymuyTj+SATInwHnsfvM/eF/+p/8k/9L+dpfreUPOI/Gc++wcbDMu8</latexit>

2 3
X V1
6Y 7 6 V2 7 V
<latexit sha1_base64="9rKPqQ6YXpU7La/ewR2tAx+Gbn8=">AAAB8nicbVDLSsNAFL2pr1pfVZduBovgqiQi6rLoxmUF+4A2lMl00g6dTMLMjVBCP8ONC0Xc+jXu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wSTu9zvPHFtRKwecZpwP6IjJULBKFqp148ojoMwa8/IoFpz6+4cZJV4BalBgeag+tUfxiyNuEImqTE9z03Qz6hGwSSfVfqp4QllEzriPUsVjbjxs3nkGTmzypCEsbZPIZmrvzcyGhkzjQI7mUc0y14u/uf1Ugxv/EyoJEWu2OKjMJUEY5LfT4ZCc4ZyagllWtishI2ppgxtSxVbgrd88ippX9S9q7r7cFlr3BZ1lOEETuEcPLiGBtxDE1rAIIZneIU3B50X5935WIyWnGLnGP7A+fwBJNaRKQ==</latexit>

A line in 3D: Xo + V = 6 7
4Z 5 +
6 7 2 R4
4 V3 5
Xo
<latexit sha1_base64="eWkW+ZpFqPfiiKJDpkAQhT+ABAw=">AAAB9HicbVDLSgMxFL1TX7W+Rl26CRbBVZkRUZdFNy4r2Ae0Q8mkmTY0k4xJplCGfocbF4q49WPc+Tdm2llo64HA4Zx7uScnTDjTxvO+ndLa+sbmVnm7srO7t3/gHh61tEwVoU0iuVSdEGvKmaBNwwynnURRHIectsPxXe63J1RpJsWjmSY0iPFQsIgRbKwU9GJsRmGUdWZ9ifpu1at5c6BV4hekCgUafferN5AkjakwhGOtu76XmCDDyjDC6azSSzVNMBnjIe1aKnBMdZDNQ8/QmVUGKJLKPmHQXP29keFY62kc2sk8pF72cvE/r5ua6CbImEhSQwVZHIpSjoxEeQNowBQlhk8twUQxmxWREVaYGNtTxZbgL395lbQuav5VzXu4rNZvizrKcAKncA4+XEMd7qEBTSDwBM/wCm/OxHlx3p2PxWjJKXaO4Q+czx+zhJIN</latexit>

1 0

Projection of a line: 02 3 2 31
2 3 2 32 3
<latexit sha1_base64="Xn2LB6C1VGsxlSYzYmtex4kpbw8=">AAADgXicdVJbb9MwFHYTLiPcOnjk5YgC2gRUSUGANCFN8MLjkGhXqKPIcZzWmuNEtgML0X4H/4s3/gzCSUPZumDpOF++4/Odi05cCK6N7/8aOO6Vq9eu79zwbt66fefucPfeTOelomxKc5GreUw0E1yyqeFGsHmhGMliwY7jk/eN//grU5rn8pOpChZmZCl5yikxlop2Bz+wYKlZgIdjtuSyzohR/PTMOwWMoWqu79bHZLLxYMWXKxPCW8u3sduhKTwBvzWMvQak538aC/oV/6MXbPSsdSrBdo4N2SvtQSu9159gbrv8bO1L0+7F2uBvbfDUSti5JgT6RWZRYKNn0aS9XzRSfr/U+rv/DB/8k2wpolT+DTCXqami4cgf++2ByyDowAh15yga/sRJTsuMSUMF0XoR+IUJa6IMp4LZSZSaFYSekCVbWChJxnRYtxt0Bo8tk0CaK2vSQMuej6hJpnWVxfal7Walt30N2edblCZ9E9ZcFqVhkq4TpaUAk0OzjpBwxagRlQWEKm5rBboiilBjl9azQwi2W74MZpNx8Grsf3w5OnzXjWMHPUAP0R4K0Gt0iD6gIzRFdPDbeeQ8d8au6+67vjtZP3UGXcx9dOG4B38AbgsMJg==</latexit>

X V1
x f 0 0 1 0 0 0 B6 7 6 V2 7 C
4y 5 = 4 0 f 05 40 1 0 05 B 6Y 7 + 6 7C , !1
@4 Z 5 4 V3 5 A
z 0 0 1 0 0 1 0
1 0

MASKS © 2004 Invitation to 3D vision

 IMAGE FORMATION – Vanishing Points from Perspective

NeurVPS Results

NeurVPS: Neural Vanishing Point Scanning via Conic Convolution, Yichao Zhou, Haozhi Qi, Jingwei Huang, Yi Ma. NeurIPS 2019.
3D Vision – Binocular Stereopsis (Convergence)

Image from Jitendra Malik

 3D Vision – Reconstruction from Multiple Images

. .
.

1. Images are all “incident” at the corresponding features in space;

2. Features in space have many types of incidence relationships;
3. Features in space have many types of metric relationships.

MASKS © 2004 Invitation to 3D vision

3D Vision – Reconstruction from Multiple Images

Image courtesy of Prof. Yasutaka Furukawa