3D Data Processing

3D Data Representation
Alberto Pretto
 Main Paradigms
● Raw data
– Point clouds
– Range images, depth and normal
maps
● Surface-based
– Meshes
– Implicit representations
– Parametric surfaces
● Solid-based
– Voxels
2
Depth Maps and Range Images
● A depth map contains the distances
of points in the scene to a plane
(e.g., the image plane of a camera
in a stereo pair)
● A range image contains the
distances of points in the scene to
another point (e.g., the optical
center of a ToF camera or a LiDAR)
● To remember: z-coordinate of the
point VS Euclidean distance from the
point
3
 Point Clouds
● Set of 3D vertices with coordinates
(x,y,z)
● Common raw output of several 3D
scanners
● Generally unstructured, arbitrary points
order:
– No information about vertex connectivity
– No clue about the underlying surface from
which the points are sampled
– Any permutation of the points yields the
same point cloud.
4
 Surfaces Approximations
● The connectivity of a given
point can be approximated
with its k-nearest neighbors.
● Surface normals can be
estimated by assuming
piecewise planar surfaces
and fitting a plane at one
point and some of its
neighbors
5
 Organized Point Clouds
● Also called structured point clouds are
organized in 2-D matrix of 3D points
– m × n × 3 array and, for each i = 1, . . . , m and
j = 1, . . . , n, there is a corresponding 3D vertex v ij
● Points are organized according to some spatial
relationships between them
● Very efficient memory layout
● Additional 2D arrays of size m × n:
– Presence or absence of 3D data
– “Confidence” values
– Points colors
– ...
6
 Organized Point Clouds
There is an implicit mesh connectivity
between neighbouring points, and single
non-boundary points are always degree 6

7
 Meshes
● Represent or approximate a 2D manifold in
3D space
● A set of vertices connected to each other in
order to form some polygons (e.g., triangles
or quadrilaterals).
● Triangular mesh: very common data structure
usually employed by 3D renderers
– Each triangle has a single surface normal
– For each surface point it is possible to get a
normal vector e.g., from the vertices
normals or directly from the face normal
– Normals can then be used by rendering
algorithms (e.g., to compute lighting
effects).
8
 Footnote: Manifolds
● In mathematics, a manifold is geometrical
space not necessarily equipped with a
metric to measure distances (therefore not
necessarily a Euclidean space) that locally
(i.e., near each point) resembles a
Euclidean space
– Each point of n-dimensional manifold has a
neighborhood that is homeomorphic to an
open subset of n-dimensional Euclidean space
● One-dimensional manifolds include lines
and circles
● Two-dimensional manifolds also called
surfaces, i.e. planes, spheres, and the torus
9
 Triangular Mesh
● Closed: the surface completely encloses a volume
● Open: the mesh contains “boundary” edges which
are used by only one triangle
● Genus of a surface: number of cuts we can make
by means of closed, non-intersecting curves
without disconnecting the manifold

10
[Source: http://mathworld.wolfram.com/]
 Additional Properties
● It is possible to associate a
color (e.g., RGB intensity) to
the vertices of a mesh
● The points that are inside a
triangle can be painted with
a color obtained by
interpolating the colors of the
vertices of the triangle

11
 Texture Coordinates
● Often, meshes are
augmented with texture
coordinates (UV
coordinates) used to map
from the surface to a 2D
planar image
● Texture coordinates range
from 0 to 1 in the x and y [Source: http://www.c-jump.com/]

axis
12
 Implicit Surfaces
● The set of all points which
satisfy the function f(x, y, z) = 0
● Typically, function values
greater than zero indicate a
point which is outside the
object, while negative values
indicate a point which is inside;
this is known as a Signed
Distance Function (SDF)
● In general, obtaining a
function f() which exactly
describes a general surface is
difficult
13
 Voxels
● Voxels ("volumetric pixel") are the 3D
analogue of pixels
● At each voxel, a boolean or a scalar
("points density") is stored which
indicates the presence or absence of
the volume at that location
● Voxels grids are used to represent
properties at discrete spatial locations
over a 3D volume
Source: [ Hoang et al.]
– Magnetic Resonance Imaging (MRI)
– Computed Tomography (CT)
14
 Voxels
● Precision depends on resolution
● Data storage requirements are high
● Can be easily used within deep learning models

[Source: FPNN: Field Probing Neural Networks for 3D Data, Li et al.] 15

 Octrees
● A space-efficient representation for
voxel grids
● It uses adaptive resolution depending
on the complexity of the surface
throughout the volume
● Idea: subdivide voxels which are not
fully occupied by the volume or fully
empty
● Represented by trees:
– The root represents the complete volume
– Nodes have eight children
– Leaves are fully occupied or empty voxels
16
Volumetric Signed Distance Function
● Idea: represents a 3D environment as
a voxel grid in which each voxel
stores the signed distance to the
nearest surface.
● The surface (i.e., the zero level set)
can be recovered by interpolation
● In order to represents non-convex or
[Source: wikipedia.org]
multiple, disconnected surfaces, the
SDF can be truncated between a
minimum and maximum value (TSDF)
17
Depth Fusion with Volumetric SDF

R. A. Newcombe et al., "KinectFusion: Real-time dense surface mapping and tracking",

2011 10th IEEE International Symposium on Mixed and Augmented Reality, 2011, pp. 127-136
18
 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 19

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 20

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 21

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 22

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 23

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 24

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 25

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 26

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 27

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 28

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 29

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 30

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 31

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 32

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 33

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 34

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 35

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 36

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] I.e., take the average SDF for each voxel 37
 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 38

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 39

 Depth Fusion with Volumetric SDF

[Slides from Tanner Schmidt] 40

 Surface Normals
Surface normals are important properties of a
geometric surface used in several problems:
– 3D shape registration
– 3D ego-motion estimation/reconstruction from point-
clouds
– Object pose estimation
– Surface classification
– In computer graphics to apply the correct light sources
that generate shadings and other visual effects
– ...
41
 Surface Normals from Meshes
● If the surface is truly piecewise
planar:
– Associate a face normal nf to
each triangular facet.
● If the surface is assumed
smooth:
– The surface normal is
approximated at each vertex,
e.g., by means of weighted
averages of the adjacent facet
normals
42
Surface Normals from Point Clouds
● No information about vertex connectivity
● Idea: for each point pi select k-nearest
neighbors and fit a plane

43
Surface Normals from Point Clouds
Given a point cloud P = {p1, p2, ..., pn} , pi ∈
R3, for each pi we would like to estimate a
normal vector ni from a set of k points in its
neighborhood plus the point pi itself:

K. Klasing, D. Althoff, D. Wollherr and M. Buss, "Comparison of surface normal estimation methods for range sensing
applications," 2009 IEEE International Conference on Robotics and Automation, 2009, pp. 3206-3211
44
Surface Normals from Point Clouds
Idea 1: fit a local plane to Qi+:

i.e., solve:

with: m × 1 vector of ones

Solved by computing SVD for Qi+ and

taking the last column of V
45
Surface Normals from Point Clouds
Idea 2: minimize the variance by removing
the empirical mean from the data matrix
Qi+:
Matrix containing the
mean of the data
points in every row

Again, we may use SVD to solve this system

and actually...
46
Surface Normals from Point Clouds

This is equivalent to performing a PCA ( Principal Component Analysis) of the original

matrix Qi+ and choosing the principal component with the smallest covariance
47
 PCA - Recup
Orthogonal projection of
data onto a lower
dimension linear space that
– Maximizes variance of
projected data (purple line)
– Minimizes mean squared
distance between original
data points and their
projection (sum of blue lines)
48
 K-D Trees
● Brute force k-nearest neighbor search has
O(k*n2) complexity
– K-dimensional trees can be used for organising
points to speed up this search to O(k*n*log n)
● A K-D tree is a binary tree in which every
leaf node is represented by one (or more)
K-dimensional point.
– K for 3D point cloud is just 3
● A K-D tree is a space-partitioning data
structure for organizing points
[Source: wikipedia.org]
● Partitioning is done by setting planes
orthogonal to one of the axes
49
 Example: K-D Tree Construction
● Construction for 1-NN (just look
for the firs NN): built the K-D
tree from all the 3D points (for
simplicity, we assume here to
deal with 2D points, i.e. K = 2)
● Example points:
{(1,9),(2,3),(4,1),(3,7),(5,4),(6,8),
(7,2),(8,8),(7,9),(9,6)}
● Pick a random dimension, find
the median, split data, repeat
50
 K-D Tree: basic algorithm
Node kdtree (PointList points, int depth)
{
if(points.size == 0) return NULL;

// The points can be grouped inside a leaf or we reached the maximum depth
if(points.size <= MAX_LEAF_SIZE or depth == DEPTH_LIMIT)
return LeafNode(points);

// Select the coordinate axis based on depth so we cyclically select all axes
int coord_id = depth mod k; // (e.g., 0 -> x, 1 -> y, 1 -> z)

// Get the median for the selected coordinate axis

median = getMedian(points, coord_id);

// Partition the points

points_before = getPointsBeforeMedian(points, median, coord_id);
points_after = getPointsAterMedian(points, median, coord_id);

// Create a new node with the median as a splitting threshold and

// recursively construct the subtrees
Node node = new Node();
node->median = median;
node->coord_id = coord_id;
node->left_child := kdtree(points_before, depth+1);
node->right_child := kdtree(points_after, depth+1);

return node;
51
}
 Example: K-D Tree NN Query
● For each query point, find the
NN by navigating the whole tree
up to a leaf.
● E.g., find the NN of the point
(7,2)
– Find the leaf that contains (7,2)
– Compute the distance of all
points in such leaf from (7,2)
and return the point that
corresponds to the second
smallest distance
52
 K-D Trees
● This simiplified implementation provides an
approximated NN, since the actual
NN may not be included in the reached
leaf
● The algorithm should be modified adding
a backtracking step to find the exact
nearest neighbour.
● The tree should be as balanced as
possible.
– Select the better dimension where operate
the split
53