Linear Algebra Basics: Vectors
Foundations of Data Analysis
February 15, 2022
CIFAR-10
32 × 32 × 3 = 3,072 dimensions
10 classes
Uniform Random Images
just kidding!
Manifold Hypothesis
Real data lie near lower-dimensional manifolds
M
Area of a Shrunken Square
(0,1) (1,1)
What is the volume of the unit
square shrunk by some small
amount in each dimension?
ε
A = (1 − 2)2
(0,0) (1,0)
1
Example: = 256
A ≈ 0.9844
Volume in High Dimensions
(1,1,...,1)
What is the volume of the unit
d-cube shrunk by some small
amount in each dimension?
ε
V = (1 − 2)d
Approaches 0 as d → ∞
(0,0,...,0)
1
Example: 256 × 256 × 3 images, = 256
V ≈ 2.0 × 10−670
Algebra
Geometry Statistics
Types of Data
I Categorical (outcomes come from a discrete set)
I Real-valued (outcomes come from R)
I Ordinal (outcomes have an order, e.g., integers)
I Vector (outcomes come from Rd )
Most data is a combination of multiple types!
Vectors
A vector is a list of real numbers:
1
x
x2
x=
...
xd
Notation: x ∈ Rd
Notation: We will use superscripts for coordinates,
subscripts when talking about a collection of vectors,
x 1 , x 2 , . . . , x n ∈ Rd .
Geometry: Direction and Distance
A vector is the difference between two points:
1 1
x2 a b
a2 b2
a=
... ,
b=
... ,
ad bd
b
a
x1
Geometry: Direction and Distance
A vector is the difference between two points:
1 1
x2 a b
a2 b2
a=
... ,
b=
... ,
x ad bd
b
1 1
a x b − a1
x 2 b2 − a2
x=
... = ...
x1 xd bd − ad
Points as Vectors
x2
x1
We will often treat points as vectors, although they are
technically not the same thing. 0
0
Think of a vector being anchored at the origin: 0 =
..
.
0
Vector Addition
x2
x1 + y1
x2 + y2
x+y=
...
y
xd + yd
x
x1
Vector Addition
x2
x1 + y1
y x2 + y2
x+y=
...
xd + yd
x
x1
Vector Addition
x2
x1 + y1
y x2 + y2
x+y x+y=
...
xd + yd
x
x1
Scalar Multiplication
Multiplication between a vector x ∈ Rd and a scalar
s ∈ R:
1 1
x sx
x sx2
2
sx = s
... = ...
xd sxd
Statistics: Vector Mean
Given vector data x1 , x2 , . . . , xn ∈ Rd , the mean is
1 Pn 1
n n Pi=1 xi
1X n1 ni=1 xi2
x̄ = xi = ..
n .
i=1 n
1 d
P
n i=1 xi
Notice that this is a vector of means in each dimension.
Vector Norm
The norm of a vector is its length:
v
u d
uX
kxk = t (xi )2
i=1
Statistics: Total Variance
Remember, the equation for the variance of scalar data,
y 1 , . . . , y n ∈ R:
n
1 X
var(y) = (yi − ȳ)2 .
n−1
i=1
For total variance for vector data, x1 , . . . , xn ∈ Rd , is
n
1 X
var(x) = kxi − x̄k2 .
n−1
i=1
Dot Product
Given two vectors, x, y ∈ Rd , their dot product is
d
X
1 1 2 2 d d
hx, yi = x y + x y + · · · + x y = x i yi .
i=1
Also known as the inner product.
Relation to norm:
p
kxk = hx, xi
Geometry: Angles and Lengths
The dot product tells us the angle θ between two
vectors, x, y ∈ Rd :
x2
�
hx, yi = kxkkyk cos θ.
x
x1
hx,yi
Or, rewriting to solve for θ : θ = arccos kxkkyk .
Geometry: Orthogonality
Two vectors at a 90 degree angle (π/2 radians) are
called orthogonal.
There dot product is zero:
π
hx, yi = kxkkyk cos = kxkkyk0 = 0
2
Geometry: Projection
x2
z x
x1
x
z= hx, yi
kxk2
Equation for a Line
Line passing through the origin along vector x ∈ Rd
L = {tx : t ∈ R}
x2
x1
Linear Independence
Two vectors, x1 , x2 ∈ Rd , are linearly independent if they
aren’t scaled versions of each other:
sx1 6= x2 , for all s ∈ R.
Equation for a Plane
Two linearly independent vectors, x, y ∈ Rd ,
span a plane:
H = {sx + ty : s ∈ R, t ∈ R}
