Photogrammetry & Robotics Lab Motivation

Visual Features: Keypoints

(Harris, Shi-Tomasi, Förstner, DoG)

Cyrill Stachniss

Most slides have been created by Cyrill Stachniss but for several slides
courtesy by Gil Levi, A. Efros, J. Hayes, D. Lowe and S. Savarese
1 2

Motivation Visual Features:

Keypoints and Descriptors
§ Keypoint is a (locally) distinct
location in an image
§ The feature descriptor summarizes
the local structure around the keypoint

3 4
Keypoint and Descriptor Today’s Topics
§ Keypoints: Finding distinct points
keypoint § Harris corners
§ Shi-Tomasi corner detector
descriptor at
the keypoint § Förstner operator
§ Difference of Gaussians

§ Features: Describing a keypoint

§ SIFT – Scale Invariant Feature Transform
§ BRIEF – Binary Robust Independent
Elementary Features
§ ORB – Oriented FAST Rotated BRIEF
5 6

Corners
§ Corners are often highly distinct points

Keypoints
“Finding locally distinct points”

Part 1: Corners

7 8
Corners & Edges Finding Corners
§ Corners are often highly distinct points § To find corners we need to search for
§ Corners are invariant to translation, intensity changes in two directions
rotation, and illumination § Compute the SSD of neighbor pixels
§ Corner = two edges in roughly around
orthogonal directions
§ Edge = a sudden brightness change

local patch sum of squared differences

around (x,y) of image intensity values of
pixels under a given shift
(du, dv)
9 10

Finding Corners Finding Corners

§ To find corners we need to search for § The Taylor approximation leads to
intensity changes in two directions
§ Compute the SSD of neighbor pixels
around
§ Written in matrix form as

§ Using Taylor expansion, we obtain

Jacobian 11 12
Finding Corners Structure Matrix
§ Given § The structure matrix is key to finding
edges and corners
§ It encodes the changes in image
intensities in a local area
§ Move the sums inside the matrix

§ Built from the image gradients

structure matrix
13 14

Computing the Structure Matrix Computing the Structure Matrix

§ Matrix build from the image gradients § Matrix build from the image gradients

§ Jacobians computed via a convolution with a § Jacobians via Scharr or Sobel Op:
gradient kernel such as Scharr or Sobel:

15 16
Structure Matrix Structure Matrix Examples
§ Summarizes the dominant directions
of the gradient around a point

17 18

Structure Matrix Examples Corners from Structure Matrix

YES! YES!

NO! Key idea:

Considers points as corners if their
structure matrix has two large
Eigenvalues
NO!
19 20
Harris, Shi-Tomasi & Förstner Harris Corner Criterion
§ Three similar approaches § Criterion
§ Proposed in
§ 1987 (Förstner)
§ 1988 (Harris)
§ 1994 (Shi-Tomasi) § with
§ All rely on the structure matrix : flat region
§ Use different criterion for deciding of a
point is a corner or not : edge
§ Förstner offers subpixel estimation : corner
21 22

Harris Criterion Illustrated Shi-Tomasi Corner Detector

§ Criterion: Threshold smallest
Eigenvalue
“edge”

“corner”

: corner

“edge”

“flat” 23 24
Shi-Tomasi Criterion Illustrated Förstner Operator Criterion
§ Very similar to Harris corner detector
§ Defined on the inverse of the M
(covariance matrix of possible shifts)
“edge” “corner” § Similar criterion on size and roundness
of the error ellipse of covariance
matrix
§ Extension for sub-pixel estimation
“flat” “edge”

25 26

Non-Maxima Supression Implementation Remarks

§ Within a local region, looks for the § RGB to gray-scale conversion first
position with the maximum value § Real images are affected by noise,
( ) and select this point smoothing of the input is suggested

§ Example for the Förstner operator

27 28
 Summary Corner Detection Summary Corner Detection

convolutions convolutions
(smoothing (smoothing multiplications
& derivatives) & derivatives)

29 30

Summary Corner Detection Summary Corner Detection

convolutions convolutions
(smoothing multiplications (smoothing multiplications
& derivatives) & derivatives)
convolutions convolutions
(box-summing) (box-summing) multiplications,
sums, sqrt
thresholding
non-max suppression
31 32
Example Harris Corners Example

Image courtesy: Förstner 33 34

Corner Detectors Comparison

§ All three detectors perform similarly
§ Förstner was the first one and
additionally described subpixel estim. Keypoints
§ Harris became the most famous corner “Finding locally distinct points”
detector in the past
§ Shi-Tomasi seems to slightly Part 2: Difference of Gaussians
outperform Harris corners
§ Most libraries use Shi-Tomasi as the
default corner detector (e.g., openCV)
35 36
Difference of Gaussians Keypoints: Difference of Gaussians
Keypoints Over Scale-Space Pyramid

§ A variant of corner detection

§ Provides responses at corners, edges, Procedure
and blobs Over different image pyramid levels
§ Blob = mainly constant region but § Step 1: Gaussian smoothing
different to its surroundings § Step 2: Difference-of-Gaussians: find
extrema (over smoothing scales)
§ Step 3: maxima suppression at edges

37 38

Illustration Illustration

differently differently
blurred images blurred images
Image courtesy: Lowe39 Image courtesy: Lowe40
Illustration Difference of Gaussians

differently
§ Subtract differently blurred images
sized images from each other

§ Increases visibility of corners, edges,

differently and other detail present in the image
blurred images
Image courtesy: Lowe41 42

Scale Space Representation Difference of Gaussians

§ Blurring filters out high-frequencies
(noise)
§ Subtracting differently blurred images
from each other only keeps the
frequencies that lie between the blur
level of both images
§ DoG acts as a band-pass filter

t=0, 1, 4, 16, 64, 265 43 44

Difference of Gaussians Illustration

Keypoints are extrema in the DoG

over different (smoothing) scales
45 Image courtesy: Lowe 46

Extrema Suppression Keypoints

§ The DoG finds blob-like and corner- § Two groups of approaches for finding
like image structures but also leads to locally distinct points:
strong responses along edges § 1. Corners via structure matrix
§ Edges are bad for matching § Harris, Shi-Tomasi, Förstner
§ Eliminate edges via Eigenvalue test § 2. Difference of Gaussians
(similar to Harris corners) § Iterates over scales and blur
§ Finds corners and blobs
§ These approaches are key ingredients
of most hand-designed features
47 48
Summary Slide Information
§ Keypoints and descriptor together § These slides have been created by Cyrill Stachniss as part of
the Photogrammetry courses taught in 2014 and 2019
define common visual features § The slides heavily reply on material by Gil Levi, Alexai Efros,
James Hayes, David Lowe, and Silvio Savarese
§ Keypoint defines the location § I tried to acknowledge all people from whom I used images or
videos. In case I made a mistake or missed someone, please
§ Most keypoints use image gradients let me know.
§ Corners and blobs are good keypoints § If you are a university lecturer, feel free to use the course
material. If you adapt the course material, please make sure
that you keep the acknowledgements to others and please
acknowledge me as well. To satisfy my own curiosity, please
Outlook: Part 2 Feature Descriptors send me email notice if you use my slides.

Cyrill Stachniss, 2014

[email protected]

49 50