Image Detection and

Segmentation
Other Computer Vision Tasks
Semantic Classification Object Instance
Segmentation + Localization Detection Segmentation

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY
No objects, just pixels Single Object Multiple Object This image is CC0 public domain
Semantic Segmentation

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY
No objects, just pixels Single Object Multiple Object This image is CC0 public domain
 This image is CC0 public domain

Semantic Segmentation

Label each pixel in the

image with a category
label

s
Sky Sky

ee
Tr

Tr
e
es
Don’t differentiate
Cat Cow
instances, only care about
pixels
Grass Grass
Semantic Segmentation Idea: Sliding
Window
Classify center
Extract patch
pixel with CNN
Full image
Cow

Cow

Grass

Farabet et al, “Learning Hierarchical Features for Scene Labeling,” TPAMI 2013
Pinheiro and Collobert, “Recurrent Convolutional Neural Networks for Scene Labeling”, ICML 2014
Semantic Segmentation Idea: Sliding
Window
Classify center
Extract patch
pixel with CNN
Full image
Cow

Cow

Grass
Problem: Very inefficient! Not
reusing shared features between
Farabet et al, “Learning Hierarchical Features for Scene Labeling,” TPAMI 2013
overlapping patches Pinheiro and Collobert, “Recurrent Convolutional Neural Networks for Scene Labeling”, ICML 2014
Semantic Segmentation Idea: Fully
Convolutional
Design a network as a bunch of convolutional layers
to make predictions for pixels all at once!

Conv Conv Conv Conv argmax

Input:
Scores: Predictions:
3xHxW
CxHxW HxW
Convolutions:
DxHxW
 Semantic Segmentation Idea: Fully
Convolutional
Design a network as a bunch of convolutional layers
to make predictions for pixels all at once!

Conv Conv Conv Conv argmax

Input:
Scores: Predictions:
3xHxW
CxHxW HxW
Convolutions:
Problem: convolutions at
DxHxW
original image resolution will
be very expensive ...
 Semantic Segmentation Idea: Fully
Convolutional
Design network as a bunch of convolutional layers, with
downsampling and upsampling inside the network!
Med-res: Med-res:
D2 x H/4 x W/4 D2 x H/4 x W/4

Low-res:
D3 x H/4 x W/4
Input: High-res: High-res: Predictions:
3xHxW D1 x H/2 x W/2 D1 x H/2 x W/2 HxW

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015
 Semantic Segmentation Idea: Fully
Convolutional
Downsampling: Design network as a bunch of convolutional layers, with Upsampling:
Pooling, strided downsampling and upsampling inside the network! ???
convolution Med-res: Med-res:
D2 x H/4 x W/4 D2 x H/4 x W/4

Low-res:
D3 x H/4 x W/4
Input: High-res: High-res: Predictions:
3xHxW D1 x H/2 x W/2 D1 x H/2 x W/2 HxW

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015
In-Network upsampling: “Unpooling”

Nearest Neighbor “Bed of Nails”

1 1 2 2 1 0 2 0

1 2 1 1 2 2 1 2 0 0 0 0

3 4 3 3 4 4 3 4 3 0 4 0

3 3 4 4 0 0 0 0

Input: 2 x 2 Output: 4 x 4 Input: 2 x 2 Output: 4 x 4

In-Network
Max Pooling
upsampling: “Max Unpooling”
Max Unpooling
Remember which element was max!
Use positions from
1 2 6 3 pooling layer 0 0 2 0

1 2 0 1 0 0
3 5 2 1 5 6
… 3 4
1 2 2 1 7 8 0 0 0 0
Rest of the network
7 3 4 8 3 0 0 4

Input: 4 x 4 Output: 2 x 2 Input: 2 x 2 Output: 4 x 4

Corresponding pairs of
downsampling and
upsampling layers
Learnable Upsampling: Transpose
Convolution
Recall:Typical 3 x 3 convolution, stride 1 pad 1

Input: 4 x 4 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
Recall: Normal 3 x 3 convolution, stride 1 pad 1

Dot product
between filter
and input

Input: 4 x 4 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
Recall: Normal 3 x 3 convolution, stride 1 pad 1

Dot product
between filter
and input

Input: 4 x 4 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
Recall: Normal 3 x 3 convolution, stride 2 pad 1

Input: 4 x 4 Output: 2 x 2
Learnable Upsampling: Transpose
Convolution
Recall: Normal 3 x 3 convolution, stride 2 pad 1

Dot product
between filter
and input

Input: 4 x 4 Output: 2 x 2
Learnable Upsampling: Transpose
Convolution
Recall: Normal 3 x 3 convolution, stride 2 pad 1

Filter moves 2 pixels in

Dot product the input for every one
between filter pixel in the output
and input
Stride gives ratio between
movement in input and
output
Input: 4 x 4 Output: 2 x 2
Learnable Upsampling: Transpose
Convolution
3 x 3 transpose convolution, stride 2 pad 1

Input: 2 x 2 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
3 x 3 transpose convolution, stride 2 pad 1

Input gives
weight for
filter

Input: 2 x 2 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
Sum where
3 x 3 transpose convolution, stride 2 pad 1
output overlaps

Filter moves 2 pixels in

Input gives the output for every one
weight for pixel in the input
filter
Stride gives ratio between
movement in output and
input
Input: 2 x 2 Output: 4 x 4
Learnable Upsampling: Transpose
Convolution
Sum where
3 x 3 transpose convolution, stride 2 pad 1
output overlaps

Filter moves 2 pixels in

Input gives the output for every one
weight for pixel in the input
filter
Stride gives ratio between
movement in output and
input
Input: 2 x 2 Output: 4 x 4
 Learnable Upsampling: Transpose
Convolution
Sum where
Other names: 3 x 3 transpose convolution, stride 2 pad 1
output overlaps
-Deconvolution (bad)
-Upconvolution
-Fractionally strided
convolution Filter moves 2 pixels in
-Backward strided Input gives the output for every one
convolution weight for pixel in the input
filter
Stride gives ratio between
movement in output and
input
Input: 2 x 2 Output: 4 x 4
Transpose Convolution: 1D Example
Output
Input Filter Output contains
ax copies of the filter
weighted by the
x ay input, summing at
a where at overlaps in
y az + bx the output
b Need to crop one
z by pixel from output to
make output exactly
2x input
bz
 Semantic Segmentation Idea: Fully
Convolutional Upsampling:
Downsampling: Design network as a bunch of convolutional layers, with
Unpooling or strided
Pooling, strided downsampling and upsampling inside the network!
transpose convolution
convolution Med-res: Med-res:
D2 x H/4 x W/4 D2 x H/4 x W/4

Low-res:
D3 x H/4 x W/4
Input: High-res: High-res: Predictions:
3xHxW D1 x H/2 x W/2 D1 x H/2 x W/2 HxW

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015
Classification + Localization

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY

No objects, just pixels Single Object Multiple Object This image is CC0 public domain
 Classification + Localization
Class Scores
Fully Cat: 0.9
Connected:
Dog: 0.05
4096 to 1000
Car: 0.01
...

This image is CC0 public domain Vector: Fully

Connected:
4096 Box
4096 to 4
Coordinates
(x, y, w, h)
Treat localization as a
regression problem!
 Correct label:
Classification + Localization Cat

Class Scores
Fully Cat: 0.9 Softmax
Connected: Dog: 0.05 Loss
4096 to 1000 Car: 0.01
...

This image is CC0 public domain Vector: Fully

Connected:
4096 Box
4096 to 4
Coordinates L2 Loss
(x, y, w, h)
Treat localization as a
Correct box:
regression problem!
(x’, y’, w’, h’)
 Correct label:
Classification + Localization Cat

Class Scores
Fully Cat: 0.9 Softmax
Connected: Dog: 0.05 Loss
4096 to 1000 Car: 0.01
...

Multitask Loss + Loss

This image is CC0 public domain Vector: Fully

Connected:
4096 Box
4096 to 4
Coordinates L2 Loss
(x, y, w, h)
Treat localization as a
Correct box:
regression problem!
(x’, y’, w’, h’)
 Correct label:
Classification + Localization Cat

Class Scores
Fully Cat: 0.9 Softmax
Connected: Dog: 0.05 Loss
4096 to 1000 Car: 0.01
...

+ Loss

This image is CC0 public domain Vector: Fully

Often pretrained on ImageNet Connected:
4096 Box
(Transfer learning) 4096 to 4
Coordinates L2 Loss
(x, y, w, h)
Treat localization as a
Correct box:
regression problem!
(x’, y’, w’, h’)
 Aside: Human Pose Estimation
Represent pose as a
set of 14 joint
positions:

Left / right foot

Left / right knee
Left / right hip
Left / right shoulder
Left / right elbow
Left / right hand
Neck
This image is licensed under CC-BY 2.0.
Head top
Johnson and Everingham, "Clustered Pose and Nonlinear Appearance Models
for Human Pose Estimation", BMVC 2010
 Aside: Human Pose Estimation

Left foot: (x, y)

Right foot: (x, y)

…
Vector:
4096 Head top: (x, y)

Toshev and Szegedy, “DeepPose: Human Pose

Estimation via Deep Neural Networks”, CVPR 2014
 Aside: Human Pose Estimation
Correct left
foot: (x’, y’)

Left foot: (x, y) L2 loss

Right foot: (x, y) L2 loss

… ...
+ Loss
Vector:
4096 Head top: (x, y) L2 loss

Correct head
Toshev and Szegedy, “DeepPose: Human Pose top: (x’, y’)
Estimation via Deep Neural Networks”, CVPR 2014

52
Object Detection

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY

No objects, just pixels Single Object Multiple Object This image is CC0 public domain
Object Detection as Regression?
CAT: (x, y, w, h)

DOG: (x, y, w, h)
DOG: (x, y, w, h)
CAT: (x, y, w, h)

DUCK: (x, y, w, h)
DUCK: (x, y, w, h)
….
Object Detection as Regression?
Each image needs a different number of outputs!
CAT: (x, y, w, h) 4 numbers

DOG: (x, y, w, h)
DOG: (x, y, w, h) 16 numbers
CAT: (x, y, w, h)

DUCK: (x, y, w, h)
Many
DUCK: (x, y, w, h)

numbers!
….
Object Detection as Classification:
Sliding Window
Apply a CNN to many different crops of the
image, CNN classifies each crop as object
or background

Dog? NO
Cat? NO
Background? YES
Object Detection as Classification:
Sliding Window
Apply a CNN to many different crops of the
image, CNN classifies each crop as object
or background

Dog? YES
Cat? NO
Background? NO
Object Detection as Classification:
Sliding Window
Apply a CNN to many different crops of the
image, CNN classifies each crop as object
or background

Dog? YES
Cat? NO
Background? NO
Object Detection as Classification:
Sliding Window
Apply a CNN to many different crops of the
image, CNN classifies each crop as object
or background

Dog? NO
Cat? YES
Background? NO
 Object Detection as Classification:
Sliding Window
Apply a CNN to many different crops of the
image, CNN classifies each crop as object
or background

Dog? NO Cat? YES

Background? NO

Problem: Need to apply CNN to huge number of locations and scales, very
computationally expensive!
 Region Proposals
● Find “blobby” image regions that are likely to contain objects
● Relatively fast to run; e.g. Selective Search gives 1000 region
proposals in a few seconds on CPU

Alexe et al, “Measuring the objectness of image windows”, TPAMI 2012

Uijlings et al, “Selective Search for Object Recognition”, IJCV 2013
Cheng et al, “BING: Binarized normed gradients for objectness estimation at 300fps”, CVPR 2014
Zitnick and Dollar, “Edge boxes: Locating object proposals from edges”, ECCV 2014
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
R-CNN: Problems
• Ad hoc training objectives
• Fine-tune network with softmax classifier (log loss)
• Train post-hoc linear SVMs (hinge loss)
• Train post-hoc bounding-box regressions (least squares)
• Training is slow (84h), takes a lot of disk space
• Inference (detection) is slow
• 47s / image with VGG16 [Simonyan & Zisserman. ICLR15]
• Fixed by SPP-net [He et al. ECCV14]

Girshick et al, “Rich feature hierarchies for accurate object detection and
semantic segmentation”, CVPR 2014.
Slide copyright Ross Girshick, 2015; source. Reproduced with permission.

69
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Fast R-CNN

Girshick, “Fast R-CNN”, ICCV 2015.

Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Girshick, “Fast R-CNN”, ICCV 2015.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
Girshick, “Fast R-CNN”, ICCV 2015.
Figure copyright Ross Girshick, 2015; source. Reproduced with permission.
 R-CNN vs SPP vs Fast R-CNN

Girshick et al, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014.
He et al, “Spatial pyramid pooling in deep convolutional networks for visual recognition”, ECCV 2014
Girshick, “Fast R-CNN”, ICCV 2015
 R-CNN vs SPP vs Fast R-CNN

Problem:
Runtime dominated
by region proposals!

Girshick et al, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014.
He et al, “Spatial pyramid pooling in deep convolutional networks for visual recognition”, ECCV 2014
Girshick, “Fast R-CNN”, ICCV 2015
 Faster R-CNN:
Make CNN do proposals!
Insert Region Proposal
Network (RPN) to predict
proposals from features

Jointly train with 4 losses:

1. RPN classify object / not object
2. RPN regress box coordinates
3. Final classification score (object
classes)
4. Final box coordinates
Ren et al, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, NIPS 2015
Figure copyright 2015, Ross Girshick; reproduced with permission
Faster R-CNN:
Make CNN do proposals!
 Detection without Proposals: YOLO /
SSD
Within each grid cell:
- Regress from each of the B
base boxes to a final box with
5 numbers:
(dx, dy, dh, dw, confidence)
- Predict scores for each of C
classes (including
background as a class)
Input image Divide image into grid Output:
3xHxW 7x7 7 x 7 x (5 * B + C)

Image a set of base boxes

Redmon et al, “You Only Look Once:
Unified, Real-Time Object Detection”, CVPR 2016 centered at each grid cell
Liu et al, “SSD: Single-Shot MultiBox Detector”, ECCV 2016
Here B = 3
 Detection without Proposals: YOLO /
SSD
Go from input image to tensor of scores with one big convolutional network!

Within each grid cell:

- Regress from each of the B
base boxes to a final box with
5 numbers:
(dx, dy, dh, dw, confidence)
- Predict scores for each of C
classes (including
background as a class)
Input image Divide image into grid Output:
3xHxW 7x7 7 x 7 x (5 * B + C)

Image a set of base boxes

Redmon et al, “You Only Look Once:
Unified, Real-Time Object Detection”, CVPR 2016 centered at each grid cell
Liu et al, “SSD: Single-Shot MultiBox Detector”, ECCV 2016
Here B = 3
 Object Detection: Lots of variables ...
Base Network Object Detection Takeaways
VGG16 architecture Faster R-CNN is
ResNet-101 Faster R-CNN slower but more
Inception V2 R-FCN accurate
Inception V3 SSD
Inception SSD is much
ResNet Image Size faster but not as
MobileNet # Region Proposals accurate
…
Huang et al, “Speed/accuracy trade-offs for modern convolutional object detectors”, CVPR 2017
R-FCN: Dai et al, “R-FCN: Object Detection via Region-based Fully Convolutional Networks”, NIPS 2016
Inception-V2: Ioffe and Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, ICML 2015
Inception V3: Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”, arXiv 2016
Inception ResNet: Szegedy et al, “Inception-V4, Inception-ResNet and the Impact of Residual Connections on Learning”, arXiv 2016
MobileNet: Howard et al, “Efficient Convolutional Neural Networks for Mobile Vision Applications”, arXiv 2017
 Aside: Object Detection + Captioning
= Dense Captioning

Johnson, Karpathy, and Fei-Fei, “DenseCap: Fully Convolutional Localization Networks for Dense Captioning”, CVPR 2016
Figure copyright IEEE, 2016. Reproduced for educational purposes.
 Aside: Object Detection + Captioning
= Dense Captioning

Johnson, Karpathy, and Fei-Fei, “DenseCap: Fully Convolutional Localization Networks for Dense Captioning”, CVPR 2016
Figure copyright IEEE, 2016. Reproduced for educational purposes.
Instance Segmentation

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY

No objects, just pixels Single Object Multiple Object This image is CC0 public domain
 Mask R-CNN
Classification Scores: C
Box coordinates (per class): 4 * C

CNN Conv Conv

RoI Align

256 x 14 x 14 256 x 14 x 14 Predict a mask for

each of C classes

C x 14 x 14
He et al, “Mask R-CNN”, arXiv 2017
 Mask R-CNN: Very Good Results!

He et al, “Mask R-CNN”, arXiv 2017

Figures copyright Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, 2017.
Reproduced with permission.
 Mask R-CNN Also does pose
Classification Scores: C
Box coordinates (per class): 4 * C
Joint coordinates

CNN Conv Conv

RoI Align

256 x 14 x 14 256 x 14 x 14 Predict a mask for

each of C classes

C x 14 x 14
He et al, “Mask R-CNN”, arXiv 2017
 Mask R-CNN Also does pose

He et al, “Mask R-CNN”, arXiv 2017

Figures copyright Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, 2017.
Reproduced with permission.

93
Recap:
Semantic Classification Object Instance
Segmentation + Localization Detection Segmentation

GRASS, CAT, CAT DOG, DOG, CAT DOG, DOG, CAT

TREE, SKY

No objects, just pixels Single Object Multiple Object This image is CC0 public domain
Thankyou

Any Questions?