Vector Spaces
The idea of vectors dates back to the early 1800’s, but the
generality of the concept waited until Peano’s work in 1888. It
took many years to understand the importance and extent of
the ideas involved.
The underlying idea can be used to describe the forces and
accelerations in Newtonian mechanics and the potential
functions of electromagnetism and the states of systems in
quantum mechanics and the least-square fitting of
experimental data and much more.
Vector Spaces
In the study of vector analysis triple of numbers (a1, a2, a3) has been
used to represents a point in space
(a1, a2, a3, a4) = four dimensional Space
(a1, a2, a3, a4, a5) = Five dimensional Space
n
1 Vectors in R
An ordered n-tuple :
a sequence of n real numbers ( x1, x2, , xn )
n
R -space :
the set of all ordered n-tuples
n=1 R1-space = set of all real numbers
(R1-space can be represented geometrically by the x-axis)
n=2 R2-space = set of all ordered pair of real numbers ( x1 , x2 )
(R2-space can be represented geometrically by the xy-plane)
n=3 R3-space = set of all ordered triple of real numbers ( x1 , x2 , x3 )
(R3-space can be represented geometrically by the xyz-space)
n=4 R4-space = set of all ordered quadruple of real numbers ( x1 , x2 , x3 , x4 )
4.3
u u1 , u2 , , un , v v1 , v2 , , vn (two vectors in Rn)
Equality:
u v if and only if u1 v1 , u2 v2 , , un vn
Vector addition (the sum of u and v):
u v u1 v1 , u2 v2 , , un vn
Scalar multiplication (the scalar multiple of u by c):
cu cu1 , cu 2 , , cu n
Difference between u and v:
u v u (1) v (u1 v1 , u2 v2 , u3 v3 ,..., un vn )
4.4
Theorem 1: Properties of vector addition and scalar multiplication
Let u, v, and w be vectors in Rn, and let c and d be scalars
n
(1) u+v is a vector in R (closure under vector addition)
(2) u+v = v+u (commutative property of vector addition)
(3) (u+v)+w = u+(v+w) (associative property of vector addition)
(4) u+0 = u (additive identity property)
(5) u+(–u) = 0 (additive inverse property)
(6) cu is a vector in Rn (closure under scalar multiplication)
(7) c(u+v) = cu+cv (distributive property of scalar multiplication over vector
addition)
(8) (c+d)u = cu+du (distributive property of scalar multiplication over real-
number addition)
(9) c(du) = (cd)u (associative property of multiplication)
(10) 1(u) = u (multiplicative identity property)
4.5
Vector Spaces
General Linear Vector Spaces
General Linear Vector Spaces
General Linear Vector Spaces
Notes:
A vector u (u1 , u2 ,, un ) in R n can be viewed as:
a 1×n row matrix (row vector): u [u1 u2 un ]
or u1
u
a n×1 column matrix (column vector): u 2
u n
4.10
Vector addition Scalar multiplication
u v (u1 , u2 , , un ) (v1 , v2 , , vn ) cu c(u1 , u2 ,, un )
(u1 v1 , u2 v2 , , un vn ) (cu1 , cu 2 , , cu n )
Treated as 1×n row matrix
u v [u1 u2 un ] [v1 v2 vn ] cu c[u1 u2 un ]
[u1 v1 u2 v2 un vn ] [cu1 cu2 cun ]
Treated as n×1 column matrix
u1 v1 u1 v1 u1 cu1
u v u v u cu
u v 2 2 2 2 cu c 2 2
un vn un vn un cu n
4.11
2 Vector Spaces
Vector spaces :
Let V be a set on which two operations (vector addition and
scalar multiplication) are defined. If the following ten axioms
are satisfied for every u, v, and w in V and every scalar (real
number) c and d, then V is called a vector space
Addition:
(1) u+v is in V
(2) u+v=v+u
(3) u+(v+w)=(u+v)+w
(4) V has a zero vector 0 such that for every u in V, u+0=u
(5) For every u in V, there is a vector in V denoted by –u
such that u+(–u)=0
4.12
Scalar multiplication:
(6) cu is in V
(7) c(u v ) cu cv
(8) (c d )u cu du
(9) c(du) (cd )u
(10) 1(u) u
※ Any set V that satisfies these ten properties (or axioms) is called a vector
space, and the objects in the set are called vectors
※ Thus, we can conclude that Rn is of course a vector space
4.13
Four examples of vector spaces are introduced as follows. (It is
straightforward to show that these vector spaces satisfy the above ten axioms)
n
(1) n-tuple space: R
(u1 , u2 ,un ) (v1 , v2 , v2 ) (u1 v1 , u2 v2 ,un vn ) (standard vector addition)
k (u1 , u2 ,un ) (ku1 , ku2 , kun ) (standard scalar multiplication for vectors)
(2) Matrix space : V M mn
(the set of all m×n matrices with real-number entries)
Ex: (m = n = 2)
u11 u12 v11 v12 u11 v11 u12 v12
u u v v u v u v (standard matrix addition)
21 22 21 22 21 21 22 22
u11 u12 ku11 ku12
k (standard scalar multiplication for matrices)
21 22 21
u u ku ku22 4.14
(3) n-th degree or less polynomial space : V Pn
(the set of all real polynomials of degree n or less)
p( x) q( x) (a0 b0 ) (a1 b1 ) x (an bn ) x n
kp( x) ka0 ka1 x kan x n
※ By the fact that the set of real numbers is closed under addition and
multiplication, it is straightforward to show that Pn satisfies the ten
axioms and thus is a vector space
(4) Continuous function space : V C ( , )
(the set of all real-valued continuous functions defined on the
entire real line)
( f g )( x) f ( x) g ( x)
(kf )( x) kf ( x)
※ By the fact that the sum of two continuous function is continuous
and the product of a scalar and a continuous function is still a
continuous function, C ( , ) is a vector space 4.15
Summary of important vector spaces
R set of all real numbers
R 2 set of all ordered pairs
R 3 set of all ordered triples
R n set of all n-tuples
C (, ) set of all continuous functions defined on the real number line
C[a, b] set of all continuous functions defined on a closed interval [a, b]
P set of all polynomials
Pn set of all polynomials of degree n
M m,n set of m n matrices
M n ,n set of n n square matrices
※ Each element in a vector space is called a vector, so a vector can be a real
number, an n-tuple, a matrix, a polynomial, a continuous function, etc.
4.16
Theorem 2: Properties of scalar multiplication
Let v be any element of a vector space V, and let c be any
scalar. Then the following properties are true
(1) 0v 0
(2) c0 0
(3) If cv 0, either c 0 or v 0
(4) (1) v v (the additive inverse of v equals ((–1)v)
4.17
Notes: To show that a set is not a vector space, you need
only find one axiom that is not satisfied
Ex 1: The set of all integers is not a vector space
Pf: 1V , and 12 is a real-number scalar
( 12 )(1) 12 V (it is not closed under scalar multiplication)
scalar noninteger
integer
Ex 2: The set of all (exact) second-degree polynomial functions is
not a vector space
Pf: Let p ( x) x 2 and q( x) x 2 x 1
p( x) q ( x) x 1 V
(it is not closed under vector addition)
4.18
Ex 3: The set of singular matrices is not a subspace of M2×2
Let W be the set of singular (noninvertible) matrices of
order 2. Show that W is not a subspace of M2×2 with the
standard matrix operations
Sol:
1 0 0 0
A W , B W
0 0 0 1
1 0
A B W (W is not closed under vector addition)
0 1
W2 is not a subspace of M 22
4.19
(3) the standard basis matrix space:
Ex: 2 2 matrix space:
1 0 0 1 0 0 0 0
, , ,
0 0 0 0 1 0 0 1
(4) the standard basis for Pn(x):
{1, x, x2, …, xn}
Ex: P3(x) {1, x, x2, x3}
4.20
3.4 Spanning Sets and Linear Independence
Linear combination:
A vector v in a vector space V is called a linear combination of
the vectors u1 , u2 ,L , uk in V if v can be written in the form
v c1u1 c2u 2 K ck u k c1 ,c2 ,L ,ck : scalars
Ex: Given v = (– 1, – 2, – 2), u1 = (0,1,4), u2 = (– 1,1,2), and
u3 = (3,1,2) in R , find a, b, and c such that v = au1 bu 2 cu 3 .
3
Sol: b 3c 1
a b c 2
Thus v u1 2u2 u3
4a 2b 2c 2
a 1, b 2, c 1
4 - 21
Ex: (Finding a linear combination)
v1 (1,2,3) v 2 (0,1,2) v 3 ( 1,0,1)
Prove w (1,1,1) is a linear combination of v1 , v 2 , v 3
Sol: w c1 v1 c2 v 2 c3 v 3
1,1,1 c1 1, 2, 3 c2 0,1, 2 c3 1,0,1
(c1 c3 , 2c1 c2 , 3c1 2c2 c3 )
c1 -c3 1
2c1 c2 1
3c1 2c2 c3 1
4 - 22
1 0 1 1 1 0 1 1
2 1 0 1 Gauss Jordan
0 1 2 1
3 2 1 1 0 0 0 0
c1 1 t , c2 1 2t , c3 t
(this system has infinitely many solutions)
t 1
w 2 v1 3v 2 v 3
4 - 23
Basis
A basis of V is a linearly independent set of vectors
in V which spans V.
Example: Fn the standard basis
V is finite dimensional if there is a finite basis.
Dimension of V is the number of elements of a basis.
(Independent of the choice of basis.)
4.24
AGC
DSP Linear Transformations
A transformation from a vector space X to a vector
space Y with the same scalar field denoted by
L: X Y
is linear when
L(ax) aL( x)
L( x1 x2 ) x1 x2
Where
x, x1 , x2 X
We can think of the transformation as an operator
25
AGC
DSP Linear Transformations …
Example: Mapping a vector space from R n to
R m
can be expressed as a mxn matrix.
Thus the transformation
L( x1 , x2 .x3 ) ( x1 2 x2 ,3x2 4 x3 )
can be written as
x1
y1 1 2 0
y 0 3 4 x2
2
x3
26
AGC
DSP Linear Transformations
Example: Let x x1 x2 x3 ... xm
T
and let the transformation a nxm matrix
Then
Ax x1p1 x2p 2 x3p 3 ... xm p m
Thus, the range of the linear transformation (or
column space of the matrix A ) is the span of the
basis vectors.
The null space is the set which yields Ax 0
27
AGC
DSP A Problem
Given a signal vector x in the vector space S, we
want to find a point v in the subset V of the space ,
nearest to x
W x w0 x v0
w0
V
v1 v0 v2
28
AGC
DSP A Problem …
Let us agree that “nearest to” in the figure is taken in
the Euclidean distance sense.
The projection v orthogonal to the set V gives the
0
desired solution.
Moreover the error of representation is
w0 x v0
This vector is clearly orthogonal to the set V (More
on this later)
29
AGC
DSP Orthogonality Principle
Let {p1 p 2 p 3 ... p m } be a set of
independent vectors in a vector space S.
We wish to express any vector x in S as
x x1p1 x2p 2 x3p 3 ... xm p m
If x is in the span of the independent vectors then
the representation will be exact.
If on the other hand it is not then there will be an
error
30
AGC
DSP Orthogonality Principle
Thus for p x, p1 x, p 2 ... x, p m T
χ x1 x2 ... x m
T
p1 , p1 p1 , p 2 .. p1 , p m
p ,p p2 ,p2 .. p2 ,pm
R 2 1
.. .. .. ..
p ,p
m 1 pm ,p2 .. pm ,pm
Hence p Rχ or χ R 1p
31
AGC
DSP Orthogonalisation
A signal may be projected into any linear space.
The computation of its coefficients in the various
vectors of the selected space is easier when the
vectors in the space are orthogonal in that they are
then non-interacting, ie the evaluation of one such
coefficient will not influence the others
The error norm is easier to compute
Thus it makes sense to use an orthogonal set of
vectors in the space into which we are to project a
signal
32
