CS294-137
Lecture 6: Fundamentals of
Computer Vision
Allen Y. Yang
Fall, 2017
Course Schedule Update
Week 1 (8-23): Introduction and Capstone Options
Week 2 (8-30): Human Perception in the Context of VR
Week 3 (9-6): Basic Unity3D/VR Programming Workshop
Week 4 (9-13): Course project proposal presentation
Week 5 (9-20): Optics and Display technologies
Week 6 (9-27): Vision Accommodation and Vergence
Week 7 (10-4): Computer Vision related topics
Week 8 (10-11): Computer Graphics related topics
****************
Week 9: (10-18) Telemedicine (Ruzena Bajcsy/Gregorij Korillo)
Week 10 (10-25): Gaming (Jack McCauley)
Week 11 (11-1): VR Film Making (Richard Hernandez)
Week 12 (11-8): AR/VR in Arts & Design (Ted Selker)
Week 13 (11-15): Computational Imaging for VR (Ren Ng)
Week 14 (11-22): No class
Week 15 (11-29): Final project presentation
Week 16 (12-6): Final project presentation
Recommended Reading Material
• Perception: Sensation and Perception
by Bruce Goldstein
• Virtual Reality: Virtual Reality
By Steven LaValle (and checkout his YouTube lectures)
• Computer Graphics: Fundamentals of CG
by Peter Shirley
• Computer Vision: An Invitation to 3-D Vision
by Yi Ma, et al.
• Display: Mobile Displays
by Achin Bhowmik, et al.
• AR/VR Market Research: Virtual & Augmented Reality,
understanding the race for the next computing platform
by Goldman Sachs
Anatomy of an AR Device: HoloLens
Including perception & display, end-to-end latency
not exceeding 16ms (60 fps)
What can Computer Vision do?
Fundamental Problems of Computer
Vision
Camera Obscura, circa 400BC Holmes stereoscope, 1861
Image Matching using Robust Features
Part I: Basic Linear Algebra
Rigid Body Motion
Change of Coordinate Systems
Special Orthogonal Group
Be Aware of Left-Handed or Right-
Handed Coordinate Systems
Homogeneous Coordinates &
Special Euclidean Group SE(3)
Concatenation:
Inverse:
Estimation of (R, T)
Translation Only:
Rotation Only:
Don’t forget to check right-handedness!
Part II: Geometry of Pinhole Camera
* OpenCV Online Documentation
Pinhole Camera Parameters
Intrinsic parameters Extrinsic parameters
Camera Calibration using OpenCV
Camera Distortion Rectification
Part III: Structure from Motion
SfM Problem
Assume multiple 2D images of 3D points and their correspondence are
known, estimate their 3D locations and the transformations (R, T).
Definition: hat operator
Quick Facts:
Epipolar Constraint
Properties of Epipolar Constraint
Conditions on the epipoles
Conditions on the epipolar lines (by co-images)
Estimation of Essential Matrix
Enforcing Essential Matrix
8-Point or 7-Point Algorithm
What if correspondence has error:
Random Sample Consensus (RANSAC)
Decomposition of E Matrix
Decomposition of E Matrix
About Depth Cameras
Time of Flight
Blur:
Structured Light
Light Field Camera
z1
z0
s A
s
Depth from Defocus
b
Part IV: Feature Matching
Features in images are not just 0-dim abstract points, their local appearance can
be used to improve matching across images
SIFT (Scale-Invariant Feature Transform)
Step 1: Feature Detector
What is a corner point?
SIFT (Scale-Invariant Feature Transform)
Step 1: Feature Detector
David Lowe’s Solution:
Difference of Gaussian
Typically, detection of SIFT combines both corner detection and DoG detection
* David Lowe, Distinctive image features from scale-invariant keypoints, IJCV 2004
SIFT (Scale-Invariant Feature Transform)
Step 2: Feature Descriptor
SIFT (Scale-Invariant Feature Transform)
Step 3: Histogram Matching
SIFT (Scale-Invariant Feature Transform)
Step 3: Histogram Matching
OpenCV Sample Code
