Keypoint detection

◼ Many applications benefit from features localized in (x,y)

(image registration, panorama stitching, motion estimation + tracking, recognition …)

◼ Edges well localized only in one direction ➔ detect corners?

◼ Desirable properties of keypoint detector

⚫ Accurate localization
⚫ Invariance against shift, rotation, scale, brightness change
⚫ Robustness against noise, high repeatability

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 1
 Keypoint detection

◼ Laplacian detector
◼ Determinant of Hessian detector
◼ Harris detector
◼ FAST detector

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 2
 Laplacian keypoint detector
LoG convolution Thresholding

keypoints
Input Detect local
f [x,y]
min/max

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 3
 Input images

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 4
 LoG response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 5
 Thresholded LoG response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 6
 Local extrema of thresholded LoG response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 7
 Superimposed LoG keypoints

500 strongest
keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 8
 Determinant of Hessian keypoint detector

Dxx Dyy Dxy

 f [x, y] f [x, y] 
H x, y = 
xx xy

 f xy [x, y] fyy [x, y] 
 
 D x, y  f [x, y] Dxy [x, y] f [x, y] 
 xx   
=
 D [x, y] f [x, y] Dyy [x, y] f [x, y] 
 xy 

det H x, y = fxx [x, y] fyy [x, y] − ( fxy [x, y])

2

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 9
 Input images

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 10
 DoH response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 11
 Thresholded DoH response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 12
 Local maxima of DoH response

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 13
 Superimposed DoH keypoints

500 strongest
keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 14
 What patterns can be localized most accurately?
◼ Local displacement sensitivity (assuming continuous f(x,y))
S (x,y )=   f (x, y )− f (x + x, y + y )
 
2

(x, y)window
◼ Linear approximation for small x, y
( ) ( ) ( ) ( )
fx(x,y) – horizontal image gradient
f x + x, y + y  f x, y + f x x, y x + f y x, y y fy(x,y) – vertical image gradient

  x 
2

(
S x,y  ) 
(x , y )window 
( f (x, y )
x ( ))
f y x, y  
 y 
 
  
)
 ( )
f x2 x, y ( ) ( )
f x x, y f y x, y  x 
= ( x y   

( ) ( )
(x , y)window f x x, y f y x, y ( )2
f y x, y 





y 

  
 x 
= ( x y )
M  y 




◼ Iso-sensitivity curves are ellipses

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 15
 Harris detector
λ2
Based on eigenvalues λ1, λ2 of “structure matrix”
(aka “normal matrix” aka “second-moment matrix”)
“Edge”
λ2 >> λ1 “Corner” 
 f x2 x, y  f x x, y f y x, y 
λ1 and λ2 are large 

M=
x, ywindow x, ywindow

2 
  f x x, y f y x, y 
     f y x, y 
 x, ywindow x, ywindow 
fx[x,y] – horizontal image gradient
fy[x,y] – vertical image gradient

“Flat” “Edge”
region λ1 >> λ2 λ1

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 16
 Harris cornerness
( ( )) ( )
2 2
C = det(M) − k  trace M = 12 − k  1 + 2

5 5

4 4

λ2 λ2
k = 0.2 3 3 k = 0.05
2 2

1 1

0 0
0 1 2 3 4 5 0 1 2 3 4 5
λ1 λ1

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 17
 Input images

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 18
 Harris cornerness

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 19
 Thresholded cornerness

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 20
 Local maxima of cornerness

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 21
 Superimposed Harris keypoints

500 strongest
keypoints

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 22
 Robustness of Harris detector

◼ Invariant to brightness offset: f [x,y] → f [x,y] + c

◼ Invariant to shift and rotation ➔

◼ Not invariant to scaling

Repeatability
➔
edge corner
Scale factor

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 23
 Features from Accelerated Segment Test (FAST)

16 1 2
15 3

14 4
13 P 5
12 6

11 7
10 9 8

◼ Compare “nucleus” p to circle of sixteen pixels

◼ Nucleus is feature point, iff at least n=9 contiguous circle
pixels are either all brighter, or all darker, by θ
◼ Optimize pixel comparisons to reject non-corners early
Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 24
 Input images

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 25
 FAST corners superimposed

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 26
 FAST corner detection on smartphone

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 27
 FAST keypoint tracking on smartphone

Image Processing: Huynh Trung Tru, © 2023 PTIT HCM -- Keypoint Detection 28