Lectures on Numerical Linear Algebra

Yunkai Zhou

Department of Mathematics
Southern Methodist University
Dallas, Texas 75075
[email protected]

Spring, 2011
 Acknowledgements

The lecture slides benefit from the following resources

Prof. Per-Olof Persson’s course materials for MIT 18.335
(Thanks Per-Olof for the tex files of slides)
Prof. Yousef Saad’s book on iterative methods
(Thanks Yousef for the tex files of course materials)
Several well-known textbooks on NLA by J. Demmel, Trefethen
and Bau, G. W. Stewart, C. D. Meyer, Golub and Van Loan
Several other books on matrix analysis and matrix computations
Several books/papers on theory and applications of LA
The Wikipedia website
The open source software: Linux, LATEX, beamer, pstricks
The instructor greatly acknowledges NSF for grants CMMI-0727194
and OCI-0749074.

Y. Zhou Math-6316/CS-7366, SMU 2/209

 Basic Linear Algebra

Spaces: Rn , Cn , Rn×n , Cn×n , Rm×n , Cm×n

(by default, Rn = Rn×1 , Cn = Cn×1 )
(Real: Rn , Rn×n , Rm×n ; Complex: Cn , Cn×n , Cm×n )

Y. Zhou Math-6316/CS-7366, SMU 3/209

 Basic Linear Algebra

Spaces: Rn , Cn , Rn×n , Cn×n , Rm×n , Cm×n

(by default, Rn = Rn×1 , Cn = Cn×1 )
(Real: Rn , Rn×n , Rm×n ; Complex: Cn , Cn×n , Cm×n )
Vectors:
v ∈ Rn (length-n column real vector)
v ∈ Cn (length-n column complex vector)
w ∈ R1×n (length-n row real vector)
w ∈ C1×n (length-n row complex vector)
(We use column vector as the default, so a vector means a
column vector)

Y. Zhou Math-6316/CS-7366, SMU 3/209

 Basic Linear Algebra

Spaces: Rn , Cn , Rn×n , Cn×n , Rm×n , Cm×n

(by default, Rn = Rn×1 , Cn = Cn×1 )
(Real: Rn , Rn×n , Rm×n ; Complex: Cn , Cn×n , Cm×n )
Vectors:
v ∈ Rn (length-n column real vector)
v ∈ Cn (length-n column complex vector)
w ∈ R1×n (length-n row real vector)
w ∈ C1×n (length-n row complex vector)
(We use column vector as the default, so a vector means a
column vector)
Special vectors:
Length-n basis vector: ei
ei : (all elements equal to 0 except the i-th element equals to 1)
n
X
Length-n vector of all-ones: 1 = [1, 1, · · · , 1]T = ei
| {z }
n i=1

Y. Zhou Math-6316/CS-7366, SMU 3/209

 Basic Linear Algebra

Matrices: (element-wise)
An m × n matrix A ∈ Rm×n (or A ∈ Cm×n )
 
A = ai,j

where ai,j ∈ R (or C), i = 1, 2, . . . , m, j = 1, 2, . . . , n.

Y. Zhou Math-6316/CS-7366, SMU 4/209

 Basic Linear Algebra

Matrices: (element-wise)
An m × n matrix A ∈ Rm×n (or A ∈ Cm×n )
 
A = ai,j

where ai,j ∈ R (or C), i = 1, 2, . . . , m, j = 1, 2, . . . , n.

Matrices: (vector-wise)
An m × n matrix A ∈ Rm×n (or A ∈ Cm×n )
 
A = a1 , a2 , · · · , an

where ai ∈ Rm (or Cm ), i = 1, 2, . . . , n.

Y. Zhou Math-6316/CS-7366, SMU 4/209

 Basic Linear Algebra

Transpose:
A = [ai,j ]m×n ⇐⇒ AT = [aj,i ]n×m
Example:
 
a11 a12  
a11 a21 a31
A = a21 a22  ⇐⇒ AT =
a12 a22 a32
a31 a32

Y. Zhou Math-6316/CS-7366, SMU 5/209

 Basic Linear Algebra

Transpose:
A = [ai,j ]m×n ⇐⇒ AT = [aj,i ]n×m
Example:
 
a11 a12  
a11 a21 a31
A = a21 a22  ⇐⇒ AT =
a12 a22 a32
a31 a32

Adjoint (conjugate transpose) :

A = [ai,j ]m×n ⇐⇒ AH = A∗ = [āj,i ]n×m

Example:
 
a11 a12  
ā ā21 ā31
A = a21 a22  ⇐⇒ A = 11 H
ā12 ā22 ā32
a31 a32

Y. Zhou Math-6316/CS-7366, SMU 5/209

 Basic Linear Algebra

A is symmetric: if A = AT
(usually it refers to “real” symmetric,
it can also be “complex” symmetric)
A is hermitian: if A = AH (or A = A∗ )
Vector-wise notation:

a ∈ Cm ⇐⇒ aT ∈ C1×m
 
a1T
  a2T 
A = a1 , a2 , · · · , an ∈ Cm×n ⇐⇒ AT =  .  ∈ Cn×m
 
 .. 
anT

Y. Zhou Math-6316/CS-7366, SMU 6/209

 Basic Linear Algebra

Let b = (bi ) ∈ Rm , A = (ai,j ) ∈ Rm×n , x = (xi ) ∈ Rn

Matrix-vector product b = Ax
n
X
Element-wise bi = ai,j xj , i = 1, 2, . . . , m
j=1
n
X
Vector-wise b= aj xj
j=1

Any A ∈ Cm×n is a linear mapping from Cn to Cm , meaning that

A(x + y) = Ax + Ay, ∀ x, y ∈ Cn
A(αx) = αAx, ∀α∈C

Conversely, any linear mapping in finite dimensional space can

be expressed as a matrix-vector product

Y. Zhou Math-6316/CS-7366, SMU 7/209

 Basic Linear Algebra

Let b = (bi ) ∈ Rm , A = (aj ) ∈ Rm×n , x = (xi ) ∈ Rn

Matrix-vector product b = Ax
Vector-wise
n
X
b = aj xj
j=1

= x1 [a1 ] + x2 [a2 ] + · · · xn [an ]

b is a linear combination of the columns of A

Any column of A can be picked out by choosing a specific x, e.g.

aj = A(:, j) = Aej

Any row of of A can be picked out by matrix-vector product, e.g.

A(i, :) = eiT A

Y. Zhou Math-6316/CS-7366, SMU 8/209

 Basic Linear Algebra

Let A = (aj ) ∈ Rm×n , B = (bj ) ∈ Rn×k , C = (cj ) ∈ Rm×k

Matrix-matrix product C = AB
Vector-wise (compare columns in C = AB)

[c1 , c2 , . . . , ck ] = A[b1 , b2 , . . . , bk ]
n
X
=⇒ cj = Abj = ak bk,j
k=1

Each cj is a linear combination of the columns of A

Y. Zhou Math-6316/CS-7366, SMU 9/209

 Basic Linear Algebra Subroutines (BLAS)

– Standardized interface for simple vector and matrix operations

– The building block of LAPACK (as the one used in Matlab)
– Optimized implementations for specific machines provided by
manufacturers
History:
– BLAS1 (1970s) Vector operations: β = x T y, y = βx + y
– BLAS2 (mid 1980s) Matrix-vector operations: y = Ax + y
– BLAS3 (late 1980s) Matrix-matrix operations: C = AB + C
Careful cache-aware implementations give close to peak
performance for BLAS3 operations
High level algorithms (Gaussian elimination, etc) use BLAS but
no other machine dependent code
– Performance and portability

Y. Zhou Math-6316/CS-7366, SMU 10/209

 Memory Hierarchy and (BLAS)

Modern computers use a memory hierarchy:

From fast/expensive to cheap/slow:
Registers, L1 cache, L2 cache, (L3 cache ...)
local memory, remote memory, secondary memory
Fast algorithms perform many operations on each memory block
to minimize memory access (cache reuse)
Only BLAS3 has potential for very high performance

BLAS Memory Refs Flops Flops/Memory Ref

Level 1 (y = βx + y) 3n 2n 2/3
Level 2 (y = Ax + y) n2 2n2 2
Level 3 (C = AB + C) 4n2 2n3 n/2
√
Flop — floating points operations, here each +, −, ∗, /, counts as one flop, with no distinction between real and complex.

Y. Zhou Math-6316/CS-7366, SMU 11/209

 BLAS implementations

Vendor provided:
— Intel Math Kernel Library (MKL)
— AMD Core Math Library (ACML)
— Sun Performance Library
— SGI Scientific Computing Software Library
Automatically Tuned Linear Algebra Software (ATLAS)
— Analyzes hardware to produce BLAS libraries for any platform
— Used in MATLAB, precompiled libraries freely available
— Sometimes outperforms vendor libraries
GOTO BLAS (mainly for Intel processors)
— Manually optimized assembly code,
(fastest implementation for Intel processors)

Y. Zhou Math-6316/CS-7366, SMU 12/209

 Basic Linear Algebra

Examples of matrix-matrix product:

Outer product: (rank-1)
For a = (ai ) ∈ Cm , b = (bi ) ∈ Cn , (ai , bi ∈ C)

abH = [ab̄1 , ab̄2 , · · · , ab̄n ] = (ai bj ) ∈ Cm×n

Outer product: (rank ≤ k)

For U = [uj ] ∈ Cm×k , V = [vj ] ∈ Cn×k , (uj ∈ Cm , vj ∈ Cn )
 H
v1
v2H  Xk
uj vjH ∈ Cm×n
 
UV H = [u1 , u2 , · · · , uk ]  .  =
 .. 
j=1
vkH

Rank-k SVD is a representative rank-k outer product.

Pk
A = UΣV H = j=1 σj uj vjH
Y. Zhou Math-6316/CS-7366, SMU 13/209
 Basic Linear Algebra

Examples of matrix-matrix product: A ∈ Cm×n

Right multiply by an upper triangular matrix: B = AR
Let R = (rij ) ∈ Cn×n be upper triangular,
 
r11 · · · r1n j
. X

B = AR = [a1 , a2 , · · · , an ]  . . . 
. .  =⇒ bj = aℓ rℓj
rnn ℓ=1

(bj is a linear combination of only the first j columns of A)

Right multiply by a lower triangular matrix: B = AL, L ∈ Cn×n
(bj is a linear combination of only the last n− j + 1 columns of A
Left multiply by an upper triangular matrix: B = RA, R ∈ Cm×m
(i-th row of B is a linear combination of last m− i + 1 rows of A)
Left multiply by a lower triangular matrix: B = LA, L ∈ Cm×m
(i-th row of B is a linear combination of only the first i rows of A)
Y. Zhou Math-6316/CS-7366, SMU 14/209
 Basic Linear Algebra: Range, Nullspace

The range or column space of A = [a1 , a2 , . . . , an ] ∈ Cm×n :

range(A) = span{a1 , a2 , . . . , an }
= All linear combinations of the columns of A
= {Ax | ∀ x ∈ Cn }

The nullspace of A ∈ Cm×n : (also written as kernel space ker(A))

null(A) = {x | Ax = 0}

Relation between range(AH ) and null(A)

null(A) = (range(AH ))⊥

Equivalently,
Rank-nullity theorem: rank(A) + dim(null(A)) = n

Y. Zhou Math-6316/CS-7366, SMU 15/209

 Basic Linear Algebra: Rank

The column rank of A = [a1 , a2 , . . . , an ] ∈ Cm×n is the dimension

of range(A), it is the same as the number of linearly independent
columns in [a1 , a2 , . . . , an ].
Similar definition for row rank
For any m × n matrix A :

rank(A) = column rank of A = row rank of A

Question: How to determine the rank of a given A?

Theorem
An m × n matrix A (m ≥ n) is full rank iff null(A) = {0}.

In other words, a full rank matrix never maps two different vectors to a
same vector.

Y. Zhou Math-6316/CS-7366, SMU 16/209

 Basic Linear Algebra: Rank

Theorem: Let A ∈ Cm×n , (assume operation compatibility)

rank(A) ≤ min(m, n); rank(A) = dim(range(A))
rank(AB) ≤ min(rank(A), rank(B))
rank(AB) = rank(A) if B has full row-rank
rank(CA) = rank(A) if C has full column-rank
Subadditivity: rank(A + B) ≤ rank(A) + rank(B)
(Implication: A rank-k matrix can be the sum of k rank-1
matrices, but not fewer)
rank(AH A) = rank(AAH ) = rank(A) = rank(AH ) = rank(AT )
Rank-nullity theorem: rank(A) + dim(null(A)) = n
Frobenius’ rank-inequality:
rank(AB) + rank(BC) ≤ rank(B) + rank(ABC)
Special case (Sylvester’s rank-inequality):
rank(A) + rank(B) ≤ n + rank(AB)
Y. Zhou Math-6316/CS-7366, SMU 17/209
 Basic Linear Algebra: Inverse

A square (size-n) matrix A is called nonsingular (or invertible or

non-degenerate) if ∃B s.t. AB = BA = In ,
in this case B is called the inverse of A: A-1 = B
If A is nonsingular, then
 -1
A-1 =A
(A ) = (A-1 )T ,
T -1
(AH )-1 = (A-1 )H
-1 -1 -1
(AB) = B A
det(A-1 ) = det(A)-1
Change of basis (view):
x = A-1 b ⇐⇒ x is the solution to Ax = b
x is the linear combination of the columns of A-1 with coefficients b
x is the vector of coefficients of the expansion of b in the basis of
columns of A

Y. Zhou Math-6316/CS-7366, SMU 18/209

 Basic Linear Algebra: Inverse

Theorem: For A ∈ Cn×n , the following statements are equivalent:

A is invertible
rank (A) = n
ker (A) = {0} (or, Ax = b has a unique solution)
n
range (A) = C
det (A) 6= 0
Eigenvalues of A are all non-zero
Singular values of A are all non-zero
The linear mapping x 7→ Ax is a bijection from Cn → Cn
A can be expressed as a product of a finite number of
elementary matrices

Y. Zhou Math-6316/CS-7366, SMU 19/209

 Basic Linear Algebra: Elementary matrices

An elementary matrix is a matrix obtained by only one row operation

(permutation, scaling, addition) of the identity matrix.
There are three types of elementary matrices.
(I) Permutation: E(i, j) exchanges the i-th row with the j-th row of In ,
 
1
 .. 

 . 


 0 1 


E(i, j) =  .. 
 = In − ei eiT − ej ejT + ei ejT + ej eiT .
 . 

 1 0 

 .. 
 . 
1

Properties and applications:

[E(i, j)]-1 = E(i, j) (self-inverse)
E(i, j)A exchanges the i-th row with the j-th row of A
AE(i, j) exchanges the i-th column with the j-th column of A
det (E(i, j)) ≡ −1, det (E(i, j)A) = det (AE(i, j)) = −det (A)
Y. Zhou Math-6316/CS-7366, SMU 20/209
 Basic Linear Algebra: Elementary matrices

(II) Scaling: Es (i, c) scales the i-th row of In by c,

 
1
 .. 

 . 


 1 

Es (i, c) =  c  = In + (c − 1)ei eT .
  i

 1 

 .. 
 . 
1

Properties and applications:

If c 6= 0, then Es (i, c)-1 = Es (i, 1c )
Es (i, c)A scales only the i-th row of A by c
AEs (i, c) scales only the i-th column of A by c
det (Es (i, c)) = c, det (Es (i, c)A) = det (AEs (i, c)) = cdet (A)

Y. Zhou Math-6316/CS-7366, SMU 21/209

 Basic Linear Algebra: Elementary matrices

(III) Addition: Ea (i, j, c) scales the i-th row of In by c, and adds it to

the j-th row of In ,
 
1
 .. 

 . 


 1 


Ea (i, j, c) =  ..  = In + cej eiT .

 . 

 c 1 

 .. 
 . 
1

Properties and applications:

[Ea (i, j, c)]-1 = Ea (i, j, −c)
Ea (i, j, c)A scales i-th row of A by c, and adds it to j-th row
AEa (i, j, c) scales i-th column of A by c, and adds it to j-th
column
det (Ea (i, j, c)) ≡ 1, det (Ea (i, j, c)A) = det (AEa (i, j, c)) = det (A)
Y. Zhou Math-6316/CS-7366, SMU 22/209
 Basic Linear Algebra: Elementary matrices

A general definition of size-n elementary matrices:

Size-n matrices of the form In − uv T , where u, v ∈ Rn with v T u 6= 1,
are called elementary matrices.

It is easy to select u and v for the E, Es , Ea just discussed. E.g.,

For E(i, j), u = v = ei − ej
For Es (i, c), u = (1 − c)ei , v = ei
For Ea (i, j, c), u = cej , v = −ei

Theorem:
An elementary matrix I − uv T is always invertible, its inverse is

uv T
(I − uv T )-1 = I − ,
v u−1
T

which is also an elementary matrix.

Y. Zhou Math-6316/CS-7366, SMU 23/209
 Inverse (block-wise elementary matrix operation)
   
A11 0 A A12
Triangular nonsingular block matrix: and 11
A21 A22 0 A22

 -1  
A11 0 A-1
11 0
=
A21 A22 −A22 A21 A-1
-1
11 A-1
22

 -1  
A11 A12 A-1 −A-1 -1
11 A12 A22
= 11
0 A22 0 A-1
22

In particular,
 -1    -1  
I 0 I 0 I A12 I −A12
= , =
A21 I −A21 I 0 I 0 I

Y. Zhou Math-6316/CS-7366, SMU 24/209

 Inverse (block-wise view)
 
A A12
General block matrix: A = 11
A21 A22
If A11 is invertible,
     
A11 A12 I 0 A11 0 I A-1
11 A12
=
A21 A22 A21 A-1
11 I 0 S 0 I

S = A22 − A21 A-1

11 A12 is called the Schur complement of A11 in A.

If A22 is invertible,
     
A11 A12 I A12 A-1
22 Ŝ 0 I 0
=
A21 A22 0 I 0 A22 A-1
22 A21 I

Ŝ = A11 − A12 A-1

22 A21 is called the Schur complement of A22 in A.

The above decompositions prove a Theorem:

A is nonsingular iff its Schur complement is nonsingular.
Y. Zhou Math-6316/CS-7366, SMU 25/209
 Inverse (block-wise view)

If A is nonsingular, then
 -1    -1  
A11 A12 I −A-111 A12 A11 0 I 0
=
A21 A22 0 I 0 S -1 −A21 A-111 I
 -1 -1 -1 -1 -1 -1

A11 + A11 A12 S A21 A11 −A11 A12 S
=
−S -1 A21 A-1
11 S -1
Similarly,
 -1    
A11 A12 I 0 Ŝ -1 0 I −A12 A-1
22
=
A21 A22 −A-1
22 A21 I 0 A-1
22
0 I
" #
Ŝ -1 −Ŝ -1 A12 A-1
22
=
−A22 A21 Ŝ -1
-1 A-1 22 + A-1 A Ŝ -1 A A-1
22 21 12 22

Comparing the 1-1 block of A-1 =⇒ Ŝ -1 = A-1 -1 -1 -1

11 + A11 A12 S A21 A11 ,
the Binomial Inverse Theorem:
 -1  -1
A11 − A12 A-1
22 A21 = A-1 -1 -1
11 + A11 A12 A22 − A21 A11 A12 A21 A-1
11

Y. Zhou Math-6316/CS-7366, SMU 26/209

 Sherman-Morrison-Woodbury (SMW) formula

Binomial Inverse Theorem: (or SMW)

-1  -1
A + UCV H = A-1 − A-1 U C -1 + V H A-1 U V H A-1 ,

where A, U, C, V are matrices with compatible dimensions, A and


C -1 + V H A-1 U are nonsingular.

Special cases:
(Sherman-Morrison) If A is nonsingular, u, v ∈ Cn , and
1 + v H A-1 u 6= 0, then

-1 A-1 uv H A-1
A + uv H = A-1 −
1 + v H A-1 u
(Sherman-Morrison-Woodbury) If A is nonsingular, U, V ∈ Cn×k ,
and Ik + V H A-1 U is invertible, then

-1  -1
A + UV H = A-1 − A-1 U I + V H A-1 U V H A-1
Y. Zhou Math-6316/CS-7366, SMU 27/209
 Basic Linear Algebra: Vector Norms

Definition:
A vector norm k·k on a vector space X is a real-valued function on X,
which satisfies the following three conditions:
1. kxk ≥ 0, ∀ x ∈ X, and kxk = 0 iff x = 0.
2. kαxk = |α|kxk, ∀ x ∈ X, ∀α ∈ C.
3. (Triangle inequality) kx + yk ≤ kxk + kyk, ∀ x, y ∈ X.

Common vector norms on Cn

kxk1 := |x1 | + |x2 | + · · · + |xn |. (Manhattan norm or taxicab norm)

1/2
kxk2 = |x1 |2 + |x2 |2 + · · · + |xn |2 . (Euclidean norm)
kxk∞ = maxi=1,...,n |xi |. (Chebyshev norm or maximum norm)
All these three norms are special cases of the Lp norm

n
!1/p
X
p
kxkp = |xi | , p ≥ 1. (if p < 1, does kxkp define a norm?)
i=1
Y. Zhou Math-6316/CS-7366, SMU 28/209
 Verification of Norm Conditions

Example 1: Show that kxk∞ = max |xi | defines a norm.

i=1,...,n

Y. Zhou Math-6316/CS-7366, SMU 29/209

 Verification of Norm Conditions

Example 1: Show that kxk∞ = max |xi | defines a norm.

i=1,...,n

1. kxk∞ = max |xi | ≥ 0, and kxk∞ = 0 iff x = 0

i=1,...,n
2. kαxk∞ = max |αxi | = |α| max |xi | = |αkxk∞
i=1,...,n i=1,...,n
3. kx +yk∞ = max |xi +yi | ≤ max |xi |+ max |yi | = kxk∞ +kyk∞
i=1,...,n i=1,...,n i=1,...,n

Y. Zhou Math-6316/CS-7366, SMU 29/209

 Verification of Norm Conditions

Example 1: Show that kxk∞ = max |xi | defines a norm.

i=1,...,n

1. kxk∞ = max |xi | ≥ 0, and kxk∞ = 0 iff x = 0

i=1,...,n
2. kαxk∞ = max |αxi | = |α| max |xi | = |αkxk∞
i=1,...,n i=1,...,n
3. kx +yk∞ = max |xi +yi | ≤ max |xi |+ max |yi | = kxk∞ +kyk∞
i=1,...,n i=1,...,n i=1,...,n

√
Example 2: Show that kxkM = x H Mx, where M is (hermitian) PD,
defines a norm on Cn . (This is called a weighted 2-norm.)

Y. Zhou Math-6316/CS-7366, SMU 29/209

 Verification of Norm Conditions

Example 1: Show that kxk∞ = max |xi | defines a norm.

i=1,...,n

1. kxk∞ = max |xi | ≥ 0, and kxk∞ = 0 iff x = 0

i=1,...,n
2. kαxk∞ = max |αxi | = |α| max |xi | = |αkxk∞
i=1,...,n i=1,...,n
3. kx +yk∞ = max |xi +yi | ≤ max |xi |+ max |yi | = kxk∞ +kyk∞
i=1,...,n i=1,...,n i=1,...,n

√
Example 2: Show that kxkM = x H Mx, where M is (hermitian) PD,
defines a norm on Cn . (This is called a weighted 2-norm.)
√
1. Since M is PD, kxkM = x H Mx ≥ 0, and kxkM = 0 iff x = 0
√
2. kαxkM = ᾱx H Mαx = |α|kxkM
3. kx + yk2M = (x + y)H M(x + y) = x H Mx + x H My + y H Mx + y H My,
Since M is PD, let M = W H W for some W nonsingular, then
x H My + y H Mx = (Wx)H (Wy) + (Wy)H (Wx) ≤ 2 kWxk2 kWyk2 =
2kxkM kykM , therefore kx + yk2M ≤ (kxkM + kykM )2 .
Y. Zhou Math-6316/CS-7366, SMU 29/209
 Basic Linear Algebra: Matrix norms

A matrix norm (k·k) is a vector norm on Fm×n (where F = R or C).

That is,
kAk ≥ 0, and kAk = 0 iff A = 0
kαAk = |α|kAk, ∀ α ∈ F and ∀A ∈ Fm×n
(Triangle inequality) kA + Bk ≤ kAk + kBk, ∀ A, B ∈ Fm×n
In the case of square matrices, if k·k also satisfies
kABk ≤ kAkkBk, ∀ A, B ∈ Fn×n ,
then k·k is called a sub-multiplicative norm (also called a
consistent norm)
Example: Show that kAk = max |aij | defines a matrix norm. (This is
ij
called the max-norm.) Is it sub-multiplicative ?

Y. Zhou Math-6316/CS-7366, SMU 30/209

 Induced Matrix norms

Consider A ∈ Fm×n as an operator from Fn → Fm . Define the

subordinate matrix norm on Fm×n induced by k · kα on Fn and k · kβ
on Fm as:
kAxkβ
kAkα,β = max = max kAxkβ .
x6=0 kxkα kxkα =1

When α = β, it defines the induced matrix norm by vector norm k·kα ,

this norm is also called the operator norm,
kAxkα
kAkα = max = max kAxkα .
x6=0 kxkα kxkα =1

Clearly,
kAxkα ≤ kAkα kxkα .
Property: Every induced matrix norm is sub-multiplicative.
Proof: For any compatible A, B and an induced matrix norm k·kα ,
kABkα = max kABxkα ≤ max kAkα kBxkα ≤ kAkα kBkα .
kxkα =1 kxkα =1

Y. Zhou Math-6316/CS-7366, SMU 31/209

 Examples of induced matrix norms

1-norm:
m
kAxk1 X
kAk1 = max = max |aij |
x6=0 kxk1 1≤j≤n
i=1

2-norm:
kAxk2 p
kAk2 = max = λmax (AH A) = σmax (A)
x6=0 kxk2

(2-norm is also called the spectral norm)

∞-norm:
n
kAxk∞ X
kAk∞ = max = max |aij |.
x6=0 kxk∞ 1≤i≤m
j=1

Y. Zhou Math-6316/CS-7366, SMU 32/209

 Frobenius norm

For A ∈ Fm×n , treat A as a length mn vector and define the Lp vector

norm: (p ≥ 1)
 1/p
Xm Xn
kAkLp =  |aij |p  .
i=1 j=1

p = 2 gives the Frobenius norm

P P 1/2 p p
m n 2
kAkF = i=1 j=1 |aij | = trace (A∗ A) = trace (AA∗ )
Frobenius norm is sub-multiplicative, but it is not an induced
norm (why?)

Y. Zhou Math-6316/CS-7366, SMU 33/209

 Frobenius norm

For A ∈ Fm×n , treat A as a length mn vector and define the Lp vector

norm: (p ≥ 1)
 1/p
Xm Xn
kAkLp =  |aij |p  .
i=1 j=1

p = 2 gives the Frobenius norm

P P 1/2 p p
m n 2
kAkF = i=1 j=1 |aij | = trace (A∗ A) = trace (AA∗ )
Frobenius norm is sub-multiplicative, but it is not an induced
norm (why?) (if it is induced, one would have kIkF = 1)

Y. Zhou Math-6316/CS-7366, SMU 33/209

 Frobenius norm

For A ∈ Fm×n , treat A as a length mn vector and define the Lp vector

norm: (p ≥ 1)
 1/p
Xm Xn
kAkLp =  |aij |p  .
i=1 j=1

p = 2 gives the Frobenius norm

P P 1/2 p p
m n 2
kAkF = i=1 j=1 |aij | = trace (A∗ A) = trace (AA∗ )
Frobenius norm is sub-multiplicative, but it is not an induced
norm (why?) (if it is induced, one would have kIkF = 1)
Both 2-norm and Frobenius norm are unitarily invariant: Given
A ∈ Cm×n , then for any unitary Qm ∈ Cm×m , Qn ∈ Cn×n , (later)
kAkγ = kQm Akγ = kAQn kγ = kQm AQn kγ where γ = 2, F .
 
p = ∞ yields the max-norm max |aij | , also called the uniform
ij
norm.
Y. Zhou Math-6316/CS-7366, SMU 33/209
 Matrix norms defined by singular values

Let X ∈ Cm×n with m ≥ n, denote the singular values of X as

{σi (X )}, i = 1, . . . , n. The Schatten p-norm (p ≥ 1) of X is defined as

n
!1/p
X
kX kSp := σi (X )p .
i=1

Special cases:
Nuclear norm (p = 1), also called the trace norm or Ky-Fan norm:
n
X
kX k∗ = kX ktr := σi (X )
i=1

Frobenius norm (p = 2): kX kF = kX kS2 .

Spectral norm (p = ∞): kX k2 = kX kS∞ .

Y. Zhou Math-6316/CS-7366, SMU 34/209

 Basic Linear Algebra: Inner Products

For x, y ∈ Rn ,
Pn
hx , yi := y T x = x T y = i=1 xi yi
For x, y ∈ Cn ,
Pn
hx , yi := y H x = x H y = i=1 xi ȳi

(An important property) Given A ∈ Cm×n ,

hAx , yi = x , AH y , ∀ x ∈ Cn , y ∈ Cm

hx , xi ≥ 0 ∀ x
Cauchy inequality: (Cauchy-Bunyakowski-Schwarz)

hx , yi ≤ kxk2 kyk2

Let α be the angle between two vectors x, y ∈ Cn , then

hx , yi
cos(α) =
kxk2 kyk2
Y. Zhou Math-6316/CS-7366, SMU 35/209
 Inner Product (general definition)

Definition: (inner product on a linear space V )

A mapping h· , ·i : V × V → F (F = R or C) is called an inner
product if it satisfies
1. Positive-definiteness:
hu , ui ≥ 0, ∀ u ∈ V ; hu , ui = 0 if and only if u = 0,
2. Conjugate symmetry:

hu , v i = hv , ui, ∀ u, v ∈ V

3. Linearity in the first argument:

I.e., the mapping u → hu , v i is linear for each v ∈ V :

hαu , v i = α hu , v i , ∀α ∈ F
hu1 + u2 , v i = hu1 , v i + hu2 , v i , ∀u1 , u2 , v ∈ V

If F = R, then the conjugate symmetry reduces to symmetry.

Y. Zhou Math-6316/CS-7366, SMU 36/209
 Examples of matrix inner products

A very common inner product on the vector space Rn×n is defined as

hX , Y i = trace X T Y = trace Y T X , ∀ X , Y ∈ Rn×n .

The corresponding inner product on Cn×n is defined as

hX , Y i = trace Y H X , ∀ X , Y ∈ Cn×n .

The above defined h· , ·i is known as the Hilbert-Schmidt inner

product.
Frobeniuspnorm is thep
same as the Hilbert-Schmidt norm:
kAkF = hX , X i = trace (X H X ) .

Y. Zhou Math-6316/CS-7366, SMU 37/209

 Orthogonality; Orthonormality

Two vectors x, y in an inner product space (say Rn or Cn ) are

orthogonal if
hx , yi = 0
Two sets of vectors X , Y are orthogonal if

hx , yi = 0, ∀x ∈ X , ∀y ∈ Y

Pairwise orthogonal set of vectors S is defined as a set of

nonzero vectors orthogonal to each other. I.e.,

hx , yi = 0, ∀x, y ∈ S, x 6= y

Pairwise orthonormal set of vectors S is defined as a set of unit

length (in 2-norm) vectors orthogonal to each other.

Y. Zhou Math-6316/CS-7366, SMU 38/209

 Orthogonal matrices; Unitary matrices

A matrix Q ∈ Rn×n is orthogonal if

Q -1 = Q T

A matrix Q ∈ Cn×n is unitary if

Q -1 = Q H

A set of column vectors of a unitary (or orthogonal) matrix is

pairwise orthonormal
A set of row vectors of a unitary (or orthogonal) matrix is pairwise
orthonormal
Inverse reduced to (conjugate) transpose !

Qx = b ⇐⇒ x = QHb

Important class of normal matrices (defined as A∗ A = AA∗ )

Y. Zhou Math-6316/CS-7366, SMU 39/209

 Preservation of length and angle

Q H Q = QQ H = I =⇒ |det (Q) | = 1 , det (Q) = ±1 when Q is

real
Preserves inner product
hQx , Qyi = x , Q H Qy = hx , yi
Therefore, unitary matrix multiplication preserves length of vector
(kQxk2 = kxk2 ) and angle between vectors
cos ∠(Qx, Qy) = cos ∠(x, y)
A (real) orthogonal Q can only be a rigid rotation (det (Q) = 1) or
reflection (det (Q) = −1)
Rotation Reflection
Qv

u Qu u
Qu v Qv v

Y. Zhou Math-6316/CS-7366, SMU 40/209

 Givens rotation in 2-D

−→ −→ −→ −→
Rotating OA anti-clockwise by θ to O Ã. Denote L = kOAk = kO Ãk.

Ã(x̃, ỹ) x = L cos(α), y = L sin(α);

x̃ = L cos(α + θ)
A(x, y ) = x cos(θ) − y sin(θ),
ỹ = L sin(α + θ)
θ
α = y cos(θ) + x sin(θ).
O

      
x̃ x cos(θ) − sin(θ) x
=⇒ = G(θ) :=
ỹ y sin(θ) cos(θ) y
If rotate clockwise by θ, then the Givens rotation matrix is
 
cos(θ) sin(θ)
G(−θ) = .
− sin(θ) cos(θ)

Y. Zhou Math-6316/CS-7366, SMU 41/209

 Givens rotation in 2-D

−→ −→ −→ −→
Rotating OA anti-clockwise by θ to O Ã. Denote L = kOAk = kO Ãk.

Ã(x̃, ỹ) x = L cos(α), y = L sin(α);

x̃ = L cos(α + θ)
A(x, y ) = x cos(θ) − y sin(θ),
ỹ = L sin(α + θ)
θ
α = y cos(θ) + x sin(θ).
O

      
x̃ x cos(θ) − sin(θ) x
=⇒ = G(θ) :=
ỹ y sin(θ) cos(θ) y
If rotate clockwise by θ, then the Givens rotation matrix is
 
cos(θ) sin(θ)
G(−θ) = . G-1 (θ) = G(−θ)
− sin(θ) cos(θ)

Y. Zhou Math-6316/CS-7366, SMU 41/209

 Givens rotation to zero out an element

        
x̃ x cos(θ) − sin(θ) x x cos(θ) − y sin(θ)
= G(θ) := =
ỹ y sin(θ) cos(θ) y y cos(θ) + x sin(θ)

 
x
To zero out the 2nd element in , simply choose a θ s.t. ỹ = 0,
y
−x
i.e., cot(θ) = y
There are more numerically stable ways to compute the
sin(θ), cos(θ) from x, y
To selectively zero out k elements in a length-n vector, apply
corresponding Givens rotation k times sequentially

Y. Zhou Math-6316/CS-7366, SMU 42/209

 Givens rotation in n-D

 
1 ··· 0 ··· 0 ··· 0
 .. . . .. .. .. 
.
 . . . . 
0 · · · cos(θ) ··· − sin(θ) ··· 0
 
 .. .. .. .. ... 
G(i, j, θ) = . . . . 
 
0 · · · sin(θ) ··· cos(θ) ··· 0
 
. .. .. .. .. 
 .. . . . .
0 ··· 0 ··· 0 ··· 1

That is, G(i, j, θ) = In except at the ii, jj, ij, ji positions.

Effect: G(i, j, θ)x rotates x counterclockwise in (i, j) plane by θ
Main use: To introduce zeros in vectors or matrices. E.g.,
for computing QR decompositions
Advantage: Stable (it is unitary!)
Lower operation count for very sparse matrices (only need to
zero out a few nonzero elements)

Y. Zhou Math-6316/CS-7366, SMU 43/209

 Householder reflection

Givens rotation targets to introduce one zero per rotation.

Householder reflection introduces n − 1 zeros to a length-n vector per
reflection: by requiring at that the reflected vector has only one
nonzero (i.e., parallel only to some ei ).
Let x ∈ Cn×n , denote the Householder reflector as H, want Hx to be
parallel to some ei , say e1 :

Hx = αe1
H is unitary =⇒ kHxk2 = kαe1 k2 = |α| = kxk2

Question: How to construct H, which clearly only depends on x, s.t.

the above two requirements are met ?
Essentially there is only one requirement: Construct H to be unitary
such that Hx = αe1 . (The |α| = kxk2 will hold automatically.)

Y. Zhou Math-6316/CS-7366, SMU 44/209

 Constructing a Householder reflector

The hyper-plane to reflect on should be orthogonal to w = Hx − x.

x Orthogonal projection of x on w is

w(w H x)
.
w Hw
w = Hx − x
From x, need to go twice the length
of this projection to reach Hx:
Hx = αe1
ww H x
Hx = x − 2
w Hw

The desired Householder reflector is

ww H
H =I−2
w Hw

where w = αe1 − x, |α| = kxk2 . Choose the sign of α s.t. least

cancellation of αe1 − x is involved (i.e., α = −sign(x1 ) kxk2 )
Y. Zhou Math-6316/CS-7366, SMU 45/209
 More on Householder reflector

H can be compactly written as

H = I − 2vv H , where kv k2 = 1 .

Question: What is det (H)? What are the eigenvalues of H?

Exercise: For a given nonzero v ∈ Cn , construct an H such that

Hv = kv k2 en . Use the constructed H to directly calculate Hv and
verify that it indeed equals to kv k2 en .
Y. Zhou Math-6316/CS-7366, SMU 46/209
 Summary of six major matrix decompositions:

LU decomposition
A = LU
where L is unit lower triangular, U is upper triangular
Cholesky decomposition (for hermitian PSD matrices) :

A = R H R = LDLH

where R is upper triangular, and L is unit upper triangular

QR decomposition (for A ∈ Cm×n , m ≥ n)
   
R R
A = Q̃ := [Q, Q⊥ ] = QR,
0 0

where R ∈ Cn×n is upper triangular, Q ∈ Cm×n , and

Q̃ = [Q, Q⊥ ] ∈ Cm×m is unitary

Y. Zhou Math-6316/CS-7366, SMU 47/209

 Summary of six major matrix decompositions:

Spectral decomposition (for diagonalizable A ∈ Cn×n )

A = X ΛX -1 , Λ = diag(λ1 , · · · , λn ),

where X contains the eigenvectors.

If A is symmetric/hermitian, then

A = QΛQ H ,

where Q is unitary and contains the eigenvectors.

Schur decomposition (for A ∈ Cn×n )

A = USU H ,

where U is unitary, and S is upper triangular. (Questions: What

are on the diag(S)? Can one choose the order of the diagonal
elements?)
Singular value decomposition (SVD) — next few slides
Y. Zhou Math-6316/CS-7366, SMU 48/209
 Some history of SVD

Originally developed independently by differential geometers:

Eugenio Beltrami (1873), Camille Jordan (1874)
Rediscovered independently: James J. Sylvester (1889)
Analog of singular values for compact integral operators:
Erhard Schmidt (1907), Hermann Weyl (1912)
Émile Picard in 1910 seems the first to use the term singular
values
SVD of complex matrices: Léon Autonne (1913)
SVD of rectangular matrices: Carl Eckart and Gale Young
(1936), L. Mirsky (1960)
Computation: Gene Golub and William Kahan (1965),
Gene Golub and Christian Reinsch (1970)
SVD is also known as principal component analysis (PCA), proper
orthogonal decomposition (POD), Hotelling transform, or (discrete)
Karhunen-Loève (KL) transformation.

Y. Zhou Math-6316/CS-7366, SMU 49/209

 Some applications of SVD

Information retrieval and data mining

Data compression; Noise filtering
(Noises tend to correspond to small singular values)
Solving least squares;
Regularization of ill-conditioned (inverse) problems
Image and signal processing: e.g., Image deblurring;
Seismology; Tomography
Graph partition; graph coloring
Bioinformatics and computational biology: Immunology;
Molecular dynamics; Microarray data analysis
Weather prediction
Quantum information, in which SVD is known as the Schmidt
decomposition

Y. Zhou Math-6316/CS-7366, SMU 50/209

 Geometrical motivation of SVD

Fact: Image of a unit sphere S n in Rn under any real m × n matrix is

a hyperellipse AS n in Rm .
For example:  
S2 = (x1 , x2 ) x12 + x22 = 1

   
σ1 2 y12 y22
If A = , then AS = (y1 , y2 ) σ12
+ σ22
= 1 is an
σ2
ellipse in R2
   
a11 a12 2
P 2 2
If A = a21 a22
, then AS = (y1 , y2 ) yi = j aij xj , x1 +x2 = 1
is an ellipse in R2
 
a11 a12
If A = a21 a22 , then AS 2 is a (reduced) ellipsoid in R3
a31 a32
(essentially it is still a 2-d ellipse)

Y. Zhou Math-6316/CS-7366, SMU 51/209

 Geometrical interpretation of SVD

Fact: Image of a unit sphere S in Rn under any A ∈ Rm×n is a

hyperellipse AS in Rm .

Avj = σj uj

v1 σ 2u 2 σ 1u 1
v2
A

S AS

AS = UΣV T S
V T S contains rotations/reflections of S, it is still a unit sphere;
Σ(V T S) contains scaling of the new unit sphere, resulting in a
hyperellipse; and U(ΣV T S) contains rotations/reflections of the
hyperellipse, without changing its shape.
Y. Zhou Math-6316/CS-7366, SMU 52/209
 Geometrical interpretation of SVD

 
Fact: Image of S = x kxk2 = 1, x ∈ Rn under any
A = UΣV T ∈ Rm×n is a hyperellipse AS in Rm .
The σi (A)’s measure how much distortion A applies to S:
U T AS is a hyperellipse in standard position, with k-th semiaxis equal
to σk (A).
 
T T
Note U AS = y y = U Ax, x ∈ S , (assume σi > 0, i = 1, . . . , n)

y := U T Ax = U T UΣV T x = ΣV T x, ∀x ∈ S
y12 y2 y2
kxk2 = V T x 2
= Σ-1 y = 1, =⇒ 2
+ 22 + · · · + n2 = 1
2 σ1 σ2 σn

Since U is unitary, U T AS only applies rotation/reflection to AS without

changing its shape. =⇒ AS is a (reduced) hyperellipse in Rm , with its
k-th semiaxis equal to σk (A).
Y. Zhou Math-6316/CS-7366, SMU 53/209
 Singular value decomposition (main idea)

Let A ∈ Cm×n , assume that m ≥ n.

The idea of SVD may be summarized as to find two sets of
orthonormal bases of A s.t. A appears to be a simple diagonal
matrix:
U = [u1 , . . . , un ] for the column space, i.e., range (A) ⊆ span(U)
V = [v1 , . . . , vn ] for the row space, i.e., range (AH ) ⊆ span(V )
such that Avi is in the direction of ui : Avi = σi ui (σi ≥ 0)

In matrix notation,
 
" # " # σ1
 σ2 
A v1 v2 · · · vn = u1 u2 · · · un 
 ..

 =⇒ AV = UΣ
 . 
σn

The σi ’s are called singular values of A and usually ordered

non-increasingly: σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0.
Y. Zhou Math-6316/CS-7366, SMU 54/209
 Singular value decomposition (main structure)

Singular value decomposition (for A ∈ Cm×n , m ≥ n):

 
Σ H
A = Ũ V
0

where Ũ ∈ Cm×m , V ∈ Cn×n are unitary, Σ is diagonal.

Let Ũ := [U, U⊥ ], U ∈ Cm×n , then
 
Σ H
A = [U, U⊥ ] V = UΣV H
0
 
Σk
Furthermore, if Σ = with k < n, then
0
  
H Σk VkH
A = UΣV = [Uk , Uk ⊥ ] = Uk Σk VkH
0 (Vk ⊥ )H

Y. Zhou Math-6316/CS-7366, SMU 55/209

 Another proof of the rank-nullity theorem

Rank-nullity theorem: ∀ A ∈ Cm×n , rank (A) + dim (ker (A)) = n.

This result is a corollary of a stronger result:
⊥
range (A∗ ) = ker (A) .

This result is straightforward from SVD: Let A = UΣV H , where

Σk
Σ= with σk > 0, k ≤ n, U = [Uk , Uk ⊥ ], V = [Vk , Vk ⊥ ]. Then
0
 
Σk
A[Vk , Vk ⊥ ] = [Uk , Uk ⊥ ] , A∗ = Vk Σk Uk∗ .
0

Therefore, ker (A) = span(Vk ⊥ ), range (A∗ ) = span(Vk ), from which it

follows
⊥
range (A∗ ) = ker (A) .

Y. Zhou Math-6316/CS-7366, SMU 56/209

 SVD (main structure)

Σ VH
Full SVD:
A = U U⊥
A = Ũ Σ̃V H
0

Σ VH
(Thin) SVD:
A = U
A = UΣV H

Σk VkH
Truncated SVD:
A ≈ Uk Σk VkH A ≈ Uk

Y. Zhou Math-6316/CS-7366, SMU 57/209

 SVD and the Eigenvalue Decomposition (EVD)

Assume that A is square and diagonalizable, the eigenvalue

decomposition is
A = X ΛX −1

EVD uses the same basis X for row and column space;
SVD uses two different bases V , U
EVD generally does not maintain an orthonormal basis in X ,
unless A is normal;
SVD always has two orthonormal bases
EVD is defined only for square matrices;
SVD exists for all matrices
For hermitian/symmetric positive definite matrices A, EVD and
SVD are the same (assuming same order in Λ and Σ)
For hermitian/symmetric matrices A, EVD and SVD are the same
except that σi = |λi | (assuming same order in Λ and Σ)

Y. Zhou Math-6316/CS-7366, SMU 58/209

 Matrix properties revealed by SVD

For general matrix A ∈ Cm×n ,


AAH U = UΣ2
A = UΣV H =⇒
AH AV = V Σ2

Nonzero eigenvalues of AH A are nonzero σi2 , eigenvectors are vi

Nonzero eigenvalues of AAH are nonzero σi2 , eigenvectors are ui
The rank of A = the number of nonzero singular values
range(A) = hu1 , . . . , ur i and null(A) = hvr +1 , . . . , vn i,
(r = rank (A))
q
kAk2 = σ1 and kAkF = σ12 + σ22 + . . . + σr2
If A = AH , then σi = |λi | where λi are eigenvalues of A
Qm Qm
For square A, |det(A)| = i=1 σi , (compare det(A) = i=1 λi )
Condition number of A: cond (A) = σσmax min

Y. Zhou Math-6316/CS-7366, SMU 59/209

 Low-Rank Approximations

The SVD of a rank r matrix A ∈ Cm×n (r ≤ min(m, n)) can be written

as a sum of r rank-one matrices
r
X
A = UΣV ∗ = σj uj vj∗ .
j=1

Theorem: (Schmidt-Weyl / Eckart-Young-Mirsky)

The best rank k approximation of a rank r A in the 2- and F-norm is
Pk
Ak = j=1 σj uj vj∗ .
q
The errors are kA − Ak k2 = σk+1 and kA − Ak kF = σk+1 2 + · · · + σr2 .

In other words,
r
X 2
σk+1 = min kA − Bk2 , σi2 = min kA − BkF .
rank(B)=k rank(B)=k
i=k+1

Y. Zhou Math-6316/CS-7366, SMU 60/209

 Proof of the Schmidt-Weyl Theorem

We prove the general result: For any A ∈ Cm×n ,

min kA − Bk2 = σk+1 (A) .

B∈Cm×n
rank(B)≤k

The proof uses a standard technique in linear algebra which may be called
dimensionality argument.
Proof. By contradiction, if ∃B ∈ Cm×n , rank (B) ≤ k s.t.
kA − Bk2 < σk+1 (A). Then ∀w ∈ ker (B), w 6= 0,
kAwk2 = k(A − B)wk2 ≤ kA − Bk2 kwk2 < σk+1 (A) kwk2 .
Note that dim (ker (B)) ≥ n − k, and
dim(span{v1 , v2 , . . . , vk+1 }) = k + 1, therefore
Pk+1
∃w0 ∈ ker (B) ∩ span{v1 , v2 , . . . , vk+1 }, where w0 = i=1 ci vi 6= 0, for
which it must
Pbe true that
k+1
kAw0 k2 = i=1 ci σi (A)ui ≥ σk+1 (A) kw0 k2 . A contradiction.
2

Y. Zhou Math-6316/CS-7366, SMU 61/209

 Another interpretation of SVD

The SVD of a rank r matrix A ∈ Cm×n (r ≤ min(m, n)) can be written

as a sum of r rank-one matrices
r
X r
X
A= σj uj vj∗ = σj Zj , where Zj := uj vj∗ .
j=1 j=1

The {Zj }rj=1 construct part of an orthonormal basis of the Cm×n space:
 
Zi , Zj = trace Zj∗ Zi = δij

Therefore, SVD can be considered as a (partial) Fourier expansion of

A in the partial orthonormal basis {Zj }rj=1 ,

σj = A , Zj

can be interpreted as the Fourier coefficient of A in the Zj “direction”.

Y. Zhou Math-6316/CS-7366, SMU 62/209

 Why SVD is so fundamental

Provide fundamental matrix properties:

σ
Numerical Rank of matrix (counting σ j ’s > tolerance)
1
Bases for range and nullspace (in U and V )
Define matrix norms (e.g., k · k2 , k · k∗ , k · kSp )
U and V are unitary — best numerical stability (best conditioning)
Least squares fitting; Regularization of ill-conditioned problems
U and V unitary/orthogonal provide useful geometric insight
Very stable — small changes in A causes only small changes in the
SVD
Large singular values correspond to the principal components;
Small singular values correspond to noises (can be truncated)
Optimal low-rank approximations (in k · kSp such as k · k2 , k · kF )
conveniently obtained via truncated SVD
In most applications, the principal components are essential and
noise better be discarded

Y. Zhou Math-6316/CS-7366, SMU 63/209

 Why SVD does denoising well

Random noise (non-directional, e.g., Gaussian white noise) exist

almost universally
Non-directional noise distributes more or less uniformly across
each orthonormal basis Zi
Each σi Zi contains approximately the same level of noise
Signal-to-noise ratio (SNR) in σi Zi improves with larger σi
For σi ’s below some threshold, the noise level basically dominate
the signal level in σi Zi (i.e., SNR(σi Zi ) is too small). In this case,
truncating σi Zi loses only a small amount of signal, but removes
disproportionately large amount of noise.

Y. Zhou Math-6316/CS-7366, SMU 64/209

 Application of SVD in Image Compression

View m × n image as a (real) matrix A, find best rank k

approximation by SVD
Storage k(m + n) instead of mn
(When m, n are really large, more economical methods than SVD
are needed)

Y. Zhou Math-6316/CS-7366, SMU 65/209

 signular values semilogy singular values
3
180 10

160

140 2
10

120

100
1
10
80

60

0
10
40

20

−1
0 10
0 50 100 150 200 0 50 100 150 200

Singular values of the clown image, the horizontal lines plot the 1st,
5th, 10th, 15th, 20th, · · · , 65th, 70th singular values.
Original (Rank 200) Rank 1
[ 200 x 320 ] image, svd rank = 1
Rank 2
[ 200 x 320 ] image, svd rank = 2
[ 200 x 320 ] original image

20 20
20

40 40
40
60 60
60
80 80
80
100 100
100
120 120
120
140 140
140
160 160
160
180 180
180
200 200
50 100 150 200 250 300 50 100 150 200 250 300
200
50 100 150 200 250 300 Truncation error: sigma(2)/sigma(1)=2.315e−01 Truncation error: sigma(3)/sigma(1)=2.006e−01

Rank 3
[ 200 x 320 ] image, svd rank = 3
Rank 5
[ 200 x 320 ] image, svd rank = 5
Rank 10
[ 200 x 320 ] image, svd rank = 10

20 20 20

40 40 40

60 60 60

80 80 80

100 100 100

120 120 120

140 140 140

160 160 160

180 180 180

200 200 200

50 100 150 200 250 300 50 100 150 200 250 300 50 100 150 200 250 300
Truncation error: sigma(4)/sigma(1)=1.375e−01 Truncation error: sigma(6)/sigma(1)=1.229e−01 Truncation error: sigma(11)/sigma(1)=6.738e−02

Rank 15
[ 200 x 320 ] image, svd rank = 15
Rank 25
[ 200 x 320 ] image, svd rank = 25
Rank 50
[ 200 x 320 ] image, svd rank = 50

20 20 20

40 40 40

60 60 60

80 80 80

100 100 100

120 120 120

140 140 140

160 160 160

180 180 180

200 200 200

50 100 150 200 250 300 50 100 150 200 250 300 50 100 150 200 250 300
Truncation error: sigma(16)/sigma(1)=5.042e−02 Truncation error: sigma(26)/sigma(1)=3.160e−02 Truncation error: sigma(51)/sigma(1)=1.795e−02
 signular values semilogy singular values
3
500 10

450

2
400 10

350

1
300 10

250

0
200 10

150

−1
100 10

50

−2
0 10
0 200 400 600 0 200 400 600

Singular values of the lena image, the horizontal lines plot the 1st,
5th, 10th, 15th, 20th, · · · , 65th, 70th singular values.
 Original (Rank 512) Rank 3 Rank 5
[ 512 x 512 ] image, svd rank = 3 [ 512 x 512 ] image, svd rank = 5
[ 512 x 512 ] original image
50 50

50
100 100
100
150 150
150
200 200
200
250 250
250
300 300
300

350 350
350

400 400
400

450 450
450

500 500
500
50 100 150 200 250 300 350 400 450 500 50 100 150 200 250 300 350 400 450 500
50 100 150 200 250 300 350 400 450 500
Truncation error: sigma(4)/sigma(1)=9.206e−02 Truncation error: sigma(6)/sigma(1)=7.938e−02

Rank 10 Rank 15 Rank 20

[ 512 x 512 ] image, svd rank = 10 [ 512 x 512 ] image, svd rank = 15 [ 512 x 512 ] image, svd rank = 20

50 50 50

100 100 100

150 150 150

200 200 200

250 250 250

300 300 300

350 350 350

400 400 400

450 450 450

500 500 500

50 100 150 200 250 300 350 400 450 500 50 100 150 200 250 300 350 400 450 500 50 100 150 200 250 300 350 400 450 500
Truncation error: sigma(11)/sigma(1)=3.655e−02 Truncation error: sigma(16)/sigma(1)=3.030e−02 Truncation error: sigma(21)/sigma(1)=2.292e−02

Rank 30 Rank 50 Rank 70

[ 512 x 512 ] image, svd rank = 30 [ 512 x 512 ] image, svd rank = 50 [ 512 x 512 ] image, svd rank = 70

50 50 50

100 100 100

150 150 150

200 200 200

250 250 250

300 300 300

350 350 350

400 400 400

450 450 450

500 500 500

50 100 150 200 250 300 350 400 450 500 50 100 150 200 250 300 350 400 450 500 50 100 150 200 250 300 350 400 450 500
Truncation error: sigma(31)/sigma(1)=1.519e−02 Truncation error: sigma(51)/sigma(1)=9.987e−03 Truncation error: sigma(71)/sigma(1)=7.237e−03
 signular values semilogy singular values
4
200 10

2
180 10

0
160 10

−2
140 10

−4
120 10

−6
100 10

−8
80 10

−10
60 10

−12
40 10

−14
20 10

−16
0 10
0 50 100 150 200 250 0 50 100 150 200 250

Singular values of the sand image (from webshots), the horizontal

lines plot the 1st, 5th, 10th, 15th, 20th, · · · , 65th, 70th singular values.
 Original (Rank 512) Rank 3 Rank 5
[ 210 x 280 ] image, svd rank = 3 [ 210 x 280 ] image, svd rank = 5
[ 210 x 280 ] original image
20 20
20
40 40
40
60 60
60
80 80
80
100 100
100
120 120
120
140 140
140
160 160
160

180 180
180

200 200
200
50 100 150 200 250 50 100 150 200 250
50 100 150 200 250
Truncation error: sigma(4)/sigma(1)=9.982e−02 Truncation error: sigma(6)/sigma(1)=5.124e−02

Rank 10 Rank 15 Rank 20

[ 210 x 280 ] image, svd rank = 10 [ 210 x 280 ] image, svd rank = 15 [ 210 x 280 ] image, svd rank = 20

20 20 20

40 40 40

60 60 60

80 80 80

100 100 100

120 120 120

140 140 140

160 160 160

180 180 180

200 200 200

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250
Truncation error: sigma(11)/sigma(1)=2.690e−02 Truncation error: sigma(16)/sigma(1)=1.869e−02 Truncation error: sigma(21)/sigma(1)=1.390e−02

Rank 30 Rank 40 Rank 50

[ 210 x 280 ] image, svd rank = 30 [ 210 x 280 ] image, svd rank = 40 [ 210 x 280 ] image, svd rank = 50

20 20 20

40 40 40

60 60 60

80 80 80

100 100 100

120 120 120

140 140 140

160 160 160

180 180 180

200 200 200

50 100 150 200 250 50 100 150 200 250 50 100 150 200 250
Truncation error: sigma(31)/sigma(1)=9.384e−03 Truncation error: sigma(41)/sigma(1)=6.457e−03 Truncation error: sigma(51)/sigma(1)=5.271e−03
 SVD: Proof of existence

Theorem: (Any matrix has a SVD decomposition)

For any A ∈ Cm×n , there exist unitary matrices U ∈ Cm×m , V ∈ Cn×n ,
and a nonnegative diagonal matrix Σ ∈ Cm×n such that A = UΣV H .

Y. Zhou Math-6316/CS-7366, SMU 72/209

 SVD: Proof of existence

Theorem: (Any matrix has a SVD decomposition)

For any A ∈ Cm×n , there exist unitary matrices U ∈ Cm×m , V ∈ Cn×n ,
and a nonnegative diagonal matrix Σ ∈ Cm×n such that A = UΣV H .

Outline of proof: Let v1 = arg max kAxk2 . Let Av1 = σ1 u1 with

kxk2 =1
σ1 ≥ 0, ku1 k2 = 1 Then clearly σ1 = kAk2 .
Extend u1 and v1 into unitary matrices Û = [u1 , U2 ], V̂ = [v1 , V2 ], then
 
H σ1 w
Û AV̂ = , where A2 = U2H AV2 .
A2

Show that w = 0. Then apply induction to A2 .

Y. Zhou Math-6316/CS-7366, SMU 72/209

 SVD: Proof of existence

Theorem: (Any matrix has a SVD decomposition)

For any A ∈ Cm×n , there exist unitary matrices U ∈ Cm×m , V ∈ Cn×n ,
and a nonnegative diagonal matrix Σ ∈ Cm×n such that A = UΣV H .

Outline of proof: Let v1 = arg max kAxk2 . Let Av1 = σ1 u1 with

kxk2 =1
σ1 ≥ 0, ku1 k2 = 1 Then clearly σ1 = kAk2 .
Extend u1 and v1 into unitary matrices Û = [u1 , U2 ], V̂ = [v1 , V2 ], then
 
H σ1 w
Û AV̂ = , where A2 = U2H AV2 .
A2

Show that w = 0. Then apply induction to A2 .

(Uniqueness: Assume σi ’s are in nonincreasing order. If A is square and σj
are distinct, then left/right singular vectors uj , vj are uniquely determined up
to complex signs.)

Y. Zhou Math-6316/CS-7366, SMU 72/209

 Proof of Schur Decomposition

Theorem: (Schur decomposition)

Any A ∈ Cn×n can be decomposed as A = QSQ ∗ , where Q is unitary
and S is upper triangular.

Y. Zhou Math-6316/CS-7366, SMU 73/209

 Proof of Schur Decomposition

Theorem: (Schur decomposition)

Any A ∈ Cn×n can be decomposed as A = QSQ ∗ , where Q is unitary
and S is upper triangular.

Proof. Pick an eigenpair (λ1 , x) of A, with kxk2 = 1. Augment x into a

unitary U1 := [x, U2 ], then
 ∗   
x Ax x ∗ AU2 λ1 x ∗ AU2
U1∗ AU1 = = .
U2∗ Ax U2∗ AU2 0 U2∗ AU2

Apply induction: Assume that U2∗ AU2 has Schur decomposition

Q2 S2 Q2∗ . Then
     
λ x ∗ AU2 1 0 λ1 x ∗ AU2 Q2 1 0
U1∗ AU1 = 1 = .
0 Q2 S2 Q2∗ 0 Q2 0 S2 0 Q2∗

Multiply U1 , U1∗ on both sizes to obtain the Q, S as in A = QSQ ∗ .

Y. Zhou Math-6316/CS-7366, SMU 73/209
 A Corollary of Schur Decomposition

Corollary: Any normal matrix is unitarily diagonalizable.

Y. Zhou Math-6316/CS-7366, SMU 74/209

 A Corollary of Schur Decomposition

Corollary: Any normal matrix is unitarily diagonalizable.

Proof. Let A be a normal matrix with A = QSQ ∗ . Since AA∗ = A∗ A,

one must have SS ∗ = S ∗ S. It now remains to show that a triangular
normal matrix must be diagonal. Let
 
s t∗
S = 11 .
S2

Then |s11 |2 = |s11 |2 + t ∗ t =⇒ t = 0. Since S2 is also normal and

upper triangular, one can use the same trick to show that S2 must be
diagonal.

Y. Zhou Math-6316/CS-7366, SMU 74/209

 Jordan-Wielandt Theorem

m×n
This theorem can be stated as an exercise: Let the SVD
H
 of A ∈ C
0 A
be A = UΣV H . Find the eigendecomposition of .
A 0

Y. Zhou Math-6316/CS-7366, SMU 75/209

 Variational Principle (VP) for σi ’s

Characterization of σi ’s (based√on the VP of eigenvalues of hermitian

matrices: notice that kAxk2 = x ∗ A∗ Ax) . Let A ∈ Cm×n with SVD
A = UΣV ∗ , Σ = diag(σi ), V = [v1 , v2 , · · · , vn ],

with σi ’s in nonincreasing order. Let Vk = span{v1 , · · · , vk }.

Then
σ1 = max{kAxk2 : kxk2 = 1, x ∈ Cn }
σ2 = max{kAxk2 : kxk2 = 1, x ∈ Cn , x ⊥ V1 }
···
σk+1 = max {kAxk2 : kxk2 = 1, x ∈ Cn , x ⊥ Vk } .
More generally,
σk+1 = min max {kAxk2 } .
Wk ⊂Cn x∈Cn , kxk2 =1
dim(Wk )=k x⊥Wk

for k = 0, 1, · · · , n − 1 .
Y. Zhou Math-6316/CS-7366, SMU 76/209
 (semi-) Variational Principle

Theorem: For any A ∈ Cm×n , u ∈ Cm , v ∈ Cn ,

|u ∗ Av |
σmax (A) = max max = max max |u ∗ Av |.
u6=0 v 6=0 kuk2 kv k2 kuk2 =1 kv k2 =1

The following generalization is often used to prove the triangular

inequalities for the norms defined by various sum of σi ’s.

Theorem: For any A ∈ Cm×n , with nonincreasing σi ’s,

k
X
σi (A) = max max |trace (U ∗ AV )|.
U∈Cm×k V ∈Cn×k
i=1 U ∗ U=Ik V ∗ V =Ik

Proof. Apply SVD and Cauchy inequality.

Y. Zhou Math-6316/CS-7366, SMU 77/209

 Pseudo-inverse (Generalized Inverse)

Given A ∈ Cm×n , A+ ∈ Cn×m is called pseudo-inverse of A if

1. AA+ A = A
2. A+ AA+ = A+
Such A+ always exists, but uniqueness is not guaranteed. A+ is
called the Moore-Penrose pseudoinverse of A if a further condition is
added
3. Both AA+ and A+ A are hermitian
This condition guarantees uniqueness. In practice pseu-doinverse A+
mainly refers to the Moore-Penrose pseudoinverse.
 
Σk
If the full SVD of A ∈ Cm×n is A = UΣV ∗ , where ∈ Rm×n ,
0
with σk > 0 and σj = 0, j > k, then A+ can be easily obtained by
 -1 
Σk
A+ = V Σ+ U ∗ , where Σ+ = ∈ Rn×m .
0
Y. Zhou Math-6316/CS-7366, SMU 78/209
 Some properties of pseudo-inverse

ker (A+ ) = ker (A∗ ), range (A+ ) = range (A∗ )

(A+ )+ = A
+
(AT )+ = (A+ )T , A = A+ , (A∗ )+ = (A+ )∗

A = AA∗ A∗+ = A∗+ A∗ A

+
A = A+ A+∗ A∗ = A∗ A+∗ A+

A+ = (A∗ A)+ A∗ , If A has full column-rank, A+ = (A∗ A)-1 A∗

A+ = A∗ (AA∗ )+ , If A has full row-rank, A+ = A∗ (AA∗ )-1

A+ = lim (A∗ A + δI)−1 A∗ = lim A∗ (AA∗ + δI)−1

δց0 δց0

(AA+ )2 = AA+ , (A+ A)2 = A+ A

(important, related to orthogonal-projectors (later))

Y. Zhou Math-6316/CS-7366, SMU 79/209

 On QR decompositions
Projectors, orthogonal projectors, reflectors
Computing QR factorization (GS, MGS, Householder, Givens)
Solving least squares by QR and SVD

Y. Zhou Math-6316/CS-7366, SMU 80/209

 Projectors

A projector is a square matrix P that satisfies

P2 = P

If v ∈ range(P), then Pv = v
Since with v = Px, v b

Pv = P 2 x = Px = v
Projection along the line b

Pv − v ∈ null(P) Pv ra n g e
Since P(Pv − v ) = (P )
P 2 v − Pv = 0

P 2 = P is not enough to introduce an orthogonal projector (later)

Y. Zhou Math-6316/CS-7366, SMU 81/209

 Complementary Projectors, Complementary Subspaces

For projector P, the matrix I − P is its complementary projector

I − P projects on the nullspace of P:
If Pv = 0, then (I − P)v = v , so null(P) ⊆ range(I − P)
For any y ∈ range(I − P), y = (I − P)v , then Py = (P − P 2 )v = 0;
so range(I − P) ⊆ null(P)
Therefore
range(I − P) = null(P), null(I − P) = range(P)
That is,
null(I − P) ∩ null(P) = {0}
or, range(P) ∩ null(P) = {0}

A projector separates Cm into two spaces S1 , S2⊥ , with

range(P) = S1 and null(P) = S2⊥ .
That is, P is the projector along null(P) onto range(P).
Any x ∈ Cm can be decomposed as x = x1 + x2 , where
x1 ∈ range(P), x2 ∈ null(P): x = Px + (I − P)x.
Y. Zhou Math-6316/CS-7366, SMU 82/209
 A more general view of a projector

Lemma: Given any two dimension n (n < m) subspaces S1 and S2 , if

S1 and S2 are not orthogonal (i.e., S1 ∩ S2⊥ = {0}), then for any
x ∈ Cm , there exists a projector P such that Px ∈ S1 , x − Px ⊥ S2 .
(Px is the unique projection of x onto S1 along S2⊥ .
And the projector is called a projector onto S1 along S2⊥ .)

b x

S2
b

S1
b

Px

When S1 = S2 , the projector is called an orthogonal projector,

otherwise it is called an oblique projector.
Y. Zhou Math-6316/CS-7366, SMU 83/209
 Orthogonal Projectors

v
b

ra n g e
(P )
b

Pv

Definition (geometric): A projector P is orthogonal if

range(P) = (null(P))⊥

(More generally, an orthogonal projector projects onto a

subspace S1 along a subspace S2⊥ which is orthogonal to S1 .)
Definition (algebraic): A projector P is orthogonal if P ∗ = P
Definition (analytic): A projector P is orthogonal if kPk2 = 1
Y. Zhou Math-6316/CS-7366, SMU 84/209
 Equivalence of the definitions

Theorem: For any projector P,

range(P) = (null(P))⊥ ⇐⇒ P = P ∗.

Proof. The (⇐=) part is straightforward by the known fact (related to

the Rank-nullity theorem) that
range(P ∗ ) = (null(P))⊥ .

For the (=⇒) part: Given any x ∈ Cm , let y = Px ∈ range (P). Since
range(P) = (null(P))⊥ = range(P ∗ ), y ∈ range (P ∗ ). Now apply the
properties of a projector,
y = Py = P 2 x = Px
y = P ∗ y = P ∗ Px,
which lead to Px = P ∗ Px, or (P − P ∗ P)x = 0, for all x ∈ Cm . This is
only possible when P = P ∗ P, taking conjugate transpose gives
P = P ∗ = P ∗ P.
Y. Zhou Math-6316/CS-7366, SMU 85/209
 Equivalence of the definitions

Theorem: For any projector P,

kPk2 = 1 ⇐⇒ P = P ∗.

Proof. The (⇐=) part is straightforward and can be proved in several

different ways, we list two here:
(1) P = P ∗ =⇒ P is unitarily diagonalizable, let P = QΛQ ∗ , then
P 2 = P =⇒ Λ2 = Λ =⇒ Λ can only have 1 or 0 on its diagonal
=⇒ kPk2 = 1.
(2) hPx , Pxi = hx , P ∗ Pxi = x , P 2 x = hx , Pxi
=⇒ kPxk22 ≤ kxk2 kPxk2 =⇒ kPxk2 ≤ kxk2 =⇒ kPk2 ≤ 1.
But since kPk2 ≥ 1 for all P 2 = P, it must be kPk2 = 1.
The (=⇒) part is more involved but can also be proved in several different
1
ways. One of them is to use the fact that sin(θ) = kPk , where θ is the angle
2
between range (P) and null(P). Therefore kPk2 = 1 implies that
range (P) ⊥ null(P), which is equivalent to P being orthogonal, from previous
equivalence proof we get P = P ∗ .
Two other proofs based on matrix decompositions are listed below.
Y. Zhou Math-6316/CS-7366, SMU 86/209
 Prove that P 2 = P, kPk2 = 1 =⇒ P = P ∗

By SVD of P: Assume rank (P) = k ≤ m. Let P = Uk Σk Vk∗ be the

TSVD of P, with Σk nonsingular. Then
P 2 = P =⇒ Σk Vk∗ Uk Σk = Σk =⇒ Vk∗ Uk = Σ-1
k .

Therefore Vk∗ Uk is diagonal. In addition, since Uk , Vk are columns of

unitary matrices, the diagonal elements of Vk∗ Uk are all ≤ 1 by Cauchy
inequality. But since kPk2 = 1, we have σi (P) ≤ 1. Hence it must be
that Vk∗ Uk = Σ-1 ∗
k = Ik , therefore Uk = Vk , and P = Uk Σk Uk = P .
∗

(Comment: This proof shows that the singular values, as well as eigenvalues, of
an orthogonal projector must be 1 or 0.)
By Schur-decomposition of P: Let P = QSQ ∗ , then
P 2 = P =⇒ S 2 = S. Let diag(S) = (sii ), comparing diagonal elements
of S 2= S we have
 sii = 1 or 0 for all i. Assume S is ordered as
S S12
S = 11 , where diag(S11 ) = Ik , diag(S22 ) = 0m−k . Then clearly
S22
2
S22 = S22 =⇒ S22 = (0)m−k . Now use the condition kSk2 = 1 to show
that S12 = (0) and S11 = Ik : Let si: = S(i, :), i = 1 : k , by variational
e∗ Ss∗ s s∗
principal, σ1 (S) = 1 ≥ ke ki ksi: k = ksi: ki: = ksi: k2 .
i 2 i: 2 i: 2

Y. Zhou Math-6316/CS-7366, SMU 87/209

 Equivalence of the definitions

Theorem: For any projector P,

range(P) = (null(P))⊥ ⇐⇒ kPk2 = 1.

(straightforward from the previous two equivalences, however, it is a good

exercise to show the equivalence directly)

Y. Zhou Math-6316/CS-7366, SMU 88/209

 Equivalence of the definitions

Theorem: For any projector P,

range(P) = (null(P))⊥ ⇐⇒ kPk2 = 1.

Proof. For the (⇐=) part, kPk2 ≥ 1 easily follows from P 2 = P. Now
show kPk2 ≤ 1: Since range(P) ⊥ null(P), and (I − P)x ∈ null(P) for
any x, x = Px + (I − P)x is an orthogonal decomposition, by the
Pythagorean theorem kxk2 ≥ kPxk2 , hence kPk2 ≤ 1.
For the (=⇒) part: Given any nonzero x, y, with x ∈ range(P),
y ∈ null(P), need to show x ⊥ y:
Decompose x as x = αy + r where r ⊥ y and α ∈ C, then by the
2 2 2
Pythagorean theorem, kxk2 = |α|2 kyk2 + kr k2 . However, P is a
projector with kPk2 = 1,

x = Px = αPy + Pr = Pr =⇒ kxk2 = kPr k2 ≤ kPk2 kr k2 = kr k2 .

This is only possible when α = 0, i.e., x = r =⇒ x ⊥ y. Therefore

range(P) ⊥ null(P) =⇒ P is orthogonal.
Y. Zhou Math-6316/CS-7366, SMU 88/209
 Projection with orthonormal basis

Given V ∈ Cm×k with orthonormal columns, (i.e., V ∗ V = Ik ), find the

orthogonal projectors PV and PV⊥ that projects onto range (V ) and
(range (V ))⊥ respectively.

Y. Zhou Math-6316/CS-7366, SMU 89/209

 Projection with orthonormal basis

Given V ∈ Cm×k with orthonormal columns, (i.e., V ∗ V = Ik ), find the

orthogonal projectors PV and PV⊥ that projects onto range (V ) and
(range (V ))⊥ respectively.
Note that an orthogonal P needs to satisfy P 2 = P = P ∗
Since range (PV ) = range (V ),
PV = VV ∗ .
The complement I − PV is the PV⊥ , (note [V , V⊥ ] is unitary)
PV⊥ = V⊥ V⊥ ∗ = I − VV ∗ .
Special cases
Rank-1 orthogonal projector (project onto a unit direction q)
Pq = qq ∗
Rank m − 1 orthogonal projector (eliminate component in a unit
direction q)
Pq⊥ = I − qq ∗ (also written as P⊥q )
Y. Zhou Math-6316/CS-7366, SMU 89/209
 Projection with arbitrary basis

Given A ∈ Cm×k with rank (A) = k, for the orthogonal projectors PA

and PA⊥ that projects onto range (A) and (range (A))⊥ respectively.

Y. Zhou Math-6316/CS-7366, SMU 90/209

 Projection with arbitrary basis

Given A ∈ Cm×k with rank (A) = k, for the orthogonal projectors PA

and PA⊥ that projects onto range (A) and (range (A))⊥ respectively.
Easily done if QR decomposition of A is available.
Can do without QR of A:

Y. Zhou Math-6316/CS-7366, SMU 90/209

 Projection with arbitrary basis

Given A ∈ Cm×k with rank (A) = k, for the orthogonal projectors PA

and PA⊥ that projects onto range (A) and (range (A))⊥ respectively.
Easily done if QR decomposition of A is available.
Can do without QR of A:

For any v ∈ Cm , PA v ∈ range(A). Then

PA v − v ⊥ range(A), or A∗ (PA v − v ) = 0,

Set PA v = Ax, then

A∗ (Ax − v ) = 0 ⇐⇒ A∗ Ax = A∗ v

Since A∗ A is nonsingular,

x = (A∗ A)−1 A∗ v

Finally, PA v = Ax = A(A∗ A)−1 A∗ v , giving the orthogonal projector

PA = A(A∗ A)−1 A∗ ; by complement PA⊥ = I − PA .

Y. Zhou Math-6316/CS-7366, SMU 90/209
 Projection with arbitrary basis

Given A ∈ Cm×k with rank (A) = k, for the orthogonal projectors PA

and PA⊥ that projects onto range (A) and (range (A))⊥ respectively.
Easily done if QR decomposition of A is available.
Can do without QR of A:

Another way to look at it:

Since range(PA ) ⊆ range(A) and P ∗ = P, we have PA = AMA∗ for
some M = M ∗ ∈ Ck×k
Since P 2 = P, we have AMA∗ AMA∗ = AMA∗
Notice that A∗ A is nonsingular, we pick M = (A∗ A)−1 , which readily
makes PA = AMA∗ = A(A∗ A)−1 A∗ an orthogonal projector (since
P 2 = P = P ∗ ) to range(A).

Y. Zhou Math-6316/CS-7366, SMU 90/209

 Relation to pseudo-inverse

Recall that
A+ = (A∗ A)+ A∗ = A∗ (AA∗ )+
If A has full column rank,

A+ = (A∗ A)-1 A∗

So the orthogonal projector that projects onto range (A) (column

space of A) is
PA = A(A∗ A)+ A∗ = AA+ .

Similarly, the orthogonal projector that projects onto range (A∗ ) (row
space of A) is
PA∗ = A∗ (AA∗ )+ A = A+ A .

Y. Zhou Math-6316/CS-7366, SMU 91/209

 The QR Factorization (main idea)

Find orthonormal vectors that span the successive spaces

spanned by the columns of A:

ha1 i ⊆ ha1 , a2 i ⊆ ha1 , a2 , a3 i ⊆ . . .

This means that (for full rank A),

q1 , q2 , . . . , qj = a1 , a2 , . . . , aj , for j = 1, . . . , n

Y. Zhou Math-6316/CS-7366, SMU 92/209

 The QR Factorization (matrix structure)

In matrix form, q1 , q2 , . . . , qj = a1 , a2 , . . . , aj becomes

 
r11 r12 ··· r1n
" # " # .. 
 r22 . 
a1 a2 ··· an = q1 q2 ··· qn  
.. 
 ..
 . . 
rnn

or

A = QR

This is the thin QR factorization (also called reduced QR)

Orthogonal extension from Q ∈ Cm×n to Q̃ = [Q, Q⊥ ] ∈ Cm×m ,
and adding zero rows to R gives the full QR factorization .

Y. Zhou Math-6316/CS-7366, SMU 93/209

 The structure of full and thin QR Factorizations

Let A be an m × n matrix (for m ≥ n)

The full QR factorization is A = Q̃ R̃, where Q̃ is m × m unitary, R̃
is m × n upper-triangular.

A = Q̃ R̃

The thin QR Factorization is more compact, A = QR, where Q is

the first m × n part of Q̃, R is the top n × n upper-triangular part
of R̃

A = Q

Y. Zhou Math-6316/CS-7366, SMU 94/209

 Gram-Schmidt Orthogonalization

Find new qj orthogonal to q1 , . . . , qj−1 by subtracting

components along previous vectors

vj = aj − (q1∗ aj )q1 − (q2∗ aj )q2 − · · · − (qj−1

∗
aj )qj−1

Normalize to get qj = vj /kvj k

We then obtain a reduced QR factorization A = QR, with

rij = qi∗ aj , (i 6= j)

and
j−1
X
|rjj | = kaj − rij qi k2
i=1

Y. Zhou Math-6316/CS-7366, SMU 95/209

 Classical Gram-Schmidt

Straight-forward application of Gram-Schmidt orthogonalization

Numerically unstable
Algorithm: Classical Gram-Schmidt
for j = 1 to n
vj = aj
for i = 1 to j − 1
rij = qi∗ aj
vj = vj − rij qi
rjj = kvj k2
qj = vj /rjj

Matlab implementation (uses BLAS-2)

R ( 1 , 1 ) = norm ( A ( : , 1 ) ) ; Q ( : , 1 ) = A ( : , 1 ) / R ( 1 , 1 ) ;
for j = 2 : n ,
R ( 1 : j−1,j ) = Q ( : , 1 : j−1) ' * A ( : , j ) ;
Q ( : , j ) = A ( : , j ) − Q ( : , 1 : j−1) * R ( 1 : j−1,j ) ;
R ( j , j ) = norm ( Q ( : , j ) ) ;
i f ( R ( j , j ) == 0 ) , e r r o r ( [ ' columns l i n e a r l y dependent ' ] ) ; end
Q (: , j) = Q (: , j) /R(j ,j) ;
end

Y. Zhou Math-6316/CS-7366, SMU 96/209

 Existence and Uniqueness

Every A ∈ Cm×n (m ≥ n) has a full QR factorization and a thin

QR factorization.
Proof. For full rank A, Gram-Schmidt process proves existence
of thin A = QR. Otherwise, when vj = 0 choose arbitrary vector
orthogonal to previous qi .
For full QR, add orthogonal extension to Q and zero rows to R.

Each A ∈ Cm×n (m ≥ n) of full rank has a unique thin QR

decomposition A = QR, with rjj > 0.
Proof. Again Gram-Schmidt, rjj > 0 uniquely determines the
sign.

Y. Zhou Math-6316/CS-7366, SMU 97/209

 Gram-Schmidt Projections

The orthogonal vectors produced by Gram-Schmidt can be

written in terms of orthogonal projectors

P1 a1 P2 a2 Pn an
q1 = , q2 = , ..., qn =
kP1 a1 k kP2 a2 k kPn an k

where
" #
∗
Pj = I − Q̂j−1 Q̂j−1 with Q̂j−1 = q1 q2 ··· qj−1

Pj projects orthogonally onto the space orthogonal to

q1 , . . . , qj−1 , and rank(Pj ) = m − (j − 1)

Y. Zhou Math-6316/CS-7366, SMU 98/209

 The Modified Gram-Schmidt (MGS) Algorithm

The projection Pj can equivalently be written as

Pj = P⊥qj−1 · · · P⊥q2 P⊥q1

where

P⊥q = I − qq ∗

P⊥q projects orthogonally onto the space orthogonal to q, and

rank(P⊥q ) = m − 1
The Classical Gram-Schmidt algorithm computes an orthogonal
vector by

vj = Pj aj

while the Modified Gram-Schmidt algorithm uses

vj = P⊥qj−1 · · · P⊥q2 P⊥q1 aj

Y. Zhou Math-6316/CS-7366, SMU 99/209
 Classical vs. Modified Gram-Schmidt

MGS is only a simple modification of CGS: use the most current

vector for projection (e.g., orth. A = [a1 , · · · , an ] ∈ Cm×n )

Classical GS (CGS) Modified GS (MGS)

1. For j = 1, ..., n Do: 1. For j = 1, ..., n Do:
2. qj := aj 2. qj := aj
3. For i = 1, . . . , j − 1 Do 3. For i = 1, . . . , j − 1 Do
rij = aj , qi rij = qj , qi
qj := qj − rij qi qj := qj − rij qi
4. EndDo 4. EndDo
5. rjj = kqj k2 . If rjj = 0 exit 5. rjj = kqj k2 . If rjj = 0 exit
6. qj := qj /rjj 6. qj := qj /rjj
7. EndDo 7. EndDo

The above MGS partially uses P 2 = P for any orthogonal

projector P (theoretically equivalent, numerically not equivalent)

Question: For this version of MGS, is there a BLAS-2 implementation

of steps 3–4, such as that for CGS?
Y. Zhou Math-6316/CS-7366, SMU 100/209
 MGS (BLAS-2 version)

Can reorganize MGS s.t. inner loops use BLAS-2 operations, as in

CGS. Compute R row by row instead of column by column.
Modified GS (MGS2)
For j = 1, ..., n Do:
qj := aj Q = A ; R = zeros ( n , n ) ;
EndDo for j = 1 : n ,
R ( j , j ) = norm ( Q ( : , j ) ) ;
For j = 1, . . . , n Do i f ( R ( j , j ) == 0 ) ,
rjj = kqi k2 e r r o r ( ' l i n e a r l y dependent columns ' ) ;
If rjj = 0 exit end
qj := qj /rjj Q (: , j) = Q (: , j) / R(j ,j) ;
For i = j + 1, ..., n Do: R ( j , j +1: n ) =Q ( : , j ) ' * Q ( : , j +1: n ) ;
Q ( : , j +1: n ) =Q ( : , j +1: n )−Q ( : , j ) * R ( j , j +1: n ) ;
rji = qi , qj end
qi := qi − rji qj
EndDo
EndDo

This version of MGS essentially uses the relation

Pj = P⊥qj−1 · · · P⊥q2 P⊥q1 and do individual projection one by one,
while CGS apply Pj at once.
Y. Zhou Math-6316/CS-7366, SMU 101/209
Compare CGS with MGS for the vectors (choose ǫ s.t. 1 + ǫ2 ≈ 1)

a1 = (1, ǫ, 0, 0)T , a2 = (1, 0, ǫ, 0)T , a3 = (1, 0, 0, ǫ)T

Classical:
p
v1 ← (1, ǫ, 0, 0)T , r11 = 1 + ǫ2 ≈ 1, q1 = v1 /1 = (1, ǫ, 0, 0)T
v2 ← (1, 0, ǫ, 0)T , r12 = q1T a2 = 1, v2 ← v2 − 1q1 = (0, −ǫ, ǫ, 0)T
√ √
r22 = 2ǫ, q2 = v2 /r22 = (0, −1, 1, 0)T / 2
v3 ← (1, 0, 0, ǫ)T , r13 = q1T a3 = 1, v3 ← v3 − 1q1 = (0, −ǫ, 0, ǫ)T
r23 = q2T a3 = 0, v3 ← v3 − 0q2 = (0, −ǫ, 0, ǫ)T
√ √
r33 = 2ǫ, q3 = v3 /r33 = (0, −1, 0, 1)T / 2

Modified:
p
v1 ← (1, ǫ, 0, 0)T , r11 = 1 + ǫ2 ≈ 1, q1 = v1 /1 = (1, ǫ, 0, 0)T
T
v2 ← (1, 0, ǫ, 0) , r12 = = 1, v2 ← v2 − 1q1 = (0, −ǫ, ǫ, 0)T
q1T v2
√ √
r22 = 2ǫ, q2 = v2 /r22 = (0, −1, 1, 0)T / 2
v3 ← (1, 0, 0, ǫ)T , r13 = q1T v3 = 1, v3 ← v3 − 1q1 = (0, −ǫ, 0, ǫ)T
√
r23 = = ǫ/ 2, v3 ← v3 − r23 q2 = (0, −ǫ/2, −ǫ/2, ǫ)T
q2T v3
√ √
r33 = 6ǫ/2, q3 = v3 /r33 = (0, −1, −1, 2)T / 6

Check Orthogonality:
Classical: q2T q3 = (0, −1, 1, 0)(0, −1, 0, 1)T /2√
= 1/2
Modified: q2T q3 = (0, −1, 1, 0)(0, −1, −1, 2)T / 12 = 0
MGS is numerically stable (less sensitive to rounding errors)
 Flops counts of Gram-Schmidt QR

√
Count each +, −, ∗, /, · as one flop,
only look at the higher order terms
Orthonormalize A ∈ Rm×n , (m ≥ n)

Each rij = qj , qi , qj := qj − rij qi step

Modified GS (MGS) needs about 4m flops
1. For j = 1, ..., n Do: Need to do it approximately this many
2. qj := aj times
3. For i = 1, . . . , j − 1 Do j−1
n X Z n Z j−1
X n2
rij = qj , qi 1≈ 1didj ≈
2
qj := qj − rij qi j=1 i=1 1 1

4. EndDo
Approximate total flops for MGS (same for
5. rjj = kqj k2 . If rjj = 0 exit
CGS)
6. qj := qj /rjj n2
7. EndDo 4m = 2mn2
2

Y. Zhou Math-6316/CS-7366, SMU 103/209

 Gram-Schmidt as Triangular Orthogonalization

Gram-Schmidt can be considered as multiplying with triangular

matrices to make orthogonal columns. E.g., at 1st step:
 1 −r12 −r13 
" # r11 r11 r11 ··· " #

 1 
 (2) (2)
a1 a2 · · · an  1  = q1 q2 · · · qn
 
..
.

After n steps we get a product of triangular matrices

A R1 R2 · · · Rn = Q
| {z }
R −1

“Triangular orthogonalization”

Y. Zhou Math-6316/CS-7366, SMU 104/209

 Householder Orthogonal Triangularization

The Householder method multiplies by unitary matrices to make

a triangular matrix. E.g., at 1st step:
 
r11 X · · · X
 0 X · · · X
 
Q1 A =  0 X · · · X
 
 .. .. . . .. 
 . . . .
0 X ··· X

After all the steps we get a product of orthogonal matrices

Qn · · · Q2 Q1 A = R
| {z }
Q∗

“Orthogonal triangularization”

Y. Zhou Math-6316/CS-7366, SMU 105/209

 Introducing Zeros by Householder Reflectors

Qk introduces zeros below the diagonal in column k

Preserves all the zeros previously introduced

× × × X X X × × × × × ×
       
 × × ×  Q1
 0 X X  Q2
 X X  Q3
 × × 
 × × ×  0 X X  0 X  X 
       
 −→ −→  −→ 
 × × ×   0 X X   0 X   0 
× × × 0 X X 0 X 0
A(0) := A A(1) := Q1 A A(2) := Q2 Q1 A A(3) := Q3 Q2 Q1 A

Question: what shape is Qk ?

Y. Zhou Math-6316/CS-7366, SMU 106/209

 Introducing Zeros by Householder Reflectors

Qk introduces zeros below the diagonal in column k

Preserves all the zeros previously introduced

× × × X X X × × × × × ×
       
 × × ×  Q1
 0 X X  Q2
 X X  Q3
 × × 
 × × ×  0 X X  0 X  X 
       
 −→ −→  −→ 
 × × ×   0 X X   0 X   0 
× × × 0 X X 0 X 0
A(0) := A A(1) := Q1 A A(2) := Q2 Q1 A A(3) := Q3 Q2 Q1 A

Question: what shape is Qk ?

 
I 0
Qk = k−1 ∈ Cm×m , Hk = Im−k+1 − 2vk vk∗ , vk ∈ Cm−k+1 .
0 Hk

Y. Zhou Math-6316/CS-7366, SMU 106/209

 Introducing Zeros by Householder Reflectors

Qk introduces zeros below the diagonal in column k

Preserves all the zeros previously introduced

× × × X X X × × × × × ×
       
 × × ×  Q1
 0 X X  Q2
 X X  Q3
 × × 
 × × ×  0 X X  0 X  X 
       
 −→ −→  −→ 
 × × ×   0 X X   0 X   0 
× × × 0 X X 0 X 0
A(0) := A A(1) := Q1 A A(2) := Q2 Q1 A A(3) := Q3 Q2 Q1 A

Question: what shape is Qk ?

 
I 0
Qk = k−1 ∈ Cm×m , Hk = Im−k+1 − 2vk vk∗ , vk ∈ Cm−k+1 .
0 Hk

Question: what is vk ?

Y. Zhou Math-6316/CS-7366, SMU 106/209

 Introducing Zeros by Householder Reflectors

Qk introduces zeros below the diagonal in column k

Preserves all the zeros previously introduced

× × × X X X × × × × × ×
       
 × × ×  Q1
 0 X X  Q2
 X X  Q3
 × × 
 × × ×  0 X X  0 X  X 
       
 −→ −→  −→ 
 × × ×   0 X X   0 X   0 
× × × 0 X X 0 X 0
A(0) := A A(1) := Q1 A A(2) := Q2 Q1 A A(3) := Q3 Q2 Q1 A

Question: what shape is Qk ?

 
I 0
Qk = k−1 ∈ Cm×m , Hk = Im−k+1 − 2vk vk∗ , vk ∈ Cm−k+1 .
0 Hk

Question: what is vk ?
vk
ṽk = A(k−1) (k : m, k), vk ← α kṽk k2 e1 −ṽk , (α =?), vk ←
kvk k2
Y. Zhou Math-6316/CS-7366, SMU 106/209
 The Householder Algorithm

Choice of reflector: vk = α kṽk k2 e1 − ṽk ,

To minimize cancellation error, choose α = −sign(ṽk (1)) .
Equivalently, vk = sign(ṽk (1)) kṽk k2 e1 + ṽk .
Compute the factor R of a QR factorization of A ∈ Cm×n , (m ≥ n)
Leave result in place of A, (i.e., overwrite A by R).
Store reflection vectors vk for later use.

Algorithm: (QR by Householder reflectors)

For k = 1 to n
x = Ak:m,k
vk = sign(x(1))kxk2 e1 + x
vk = vk /kvk k2
Ak:m,k:n = Ak:m,k:n − 2vk (vk∗ Ak:m,k:n )

Y. Zhou Math-6316/CS-7366, SMU 107/209

 Implicit application of Q

The idea is that Qk w for any w ∈ Cm is only about 4(m − k + 1)

operation due to the special structure of Qk
Compute Q ∗ b = Qn · · · Q2 Q1 b implicitly:

Algorithm: Implicit Calculation of Q ∗ b

For k = 1 to n
bk:m = bk:m − 2vk (vk∗ bk:m )

Compute Qx = Q1 Q2 · · · Qn x implicitly:

Algorithm: Implicit Calculation of Qx

For k = n downto 1
xk:m = xk:m − 2vk (vk∗ xk:m )

To create Q explicitly, apply the calculation of Qx to x = I

Y. Zhou Math-6316/CS-7366, SMU 108/209

 Flop counts of Householder QR

Algorithm: (QR by Householder reflectors)

For k = 1 to n
x = Ak:m,k
vk = sign(x(1))kxk2 e1 + x
vk = vk /kvk k2
Ak:m,k:n = Ak:m,k:n − 2vk (vk∗ Ak:m,k:n )

Look at the highest order: Most work done by

Ak:m,k:n = Ak:m,k:n − 2vk (vk∗ Ak:m,k:n )

2(m − k )(n − k ) for the dot products vk∗ Ak:m,k:n

(m − k )(n − k ) for the outer product 2vk (· · · )
(m − k )(n − k ) for the subtraction Ak:m,k:n − · · ·
4(m − k)(n − k) major work per iteration
Including the outer loop, the total becomes
n
X n
X
4(m − k)(n − k) = 4 (mn − k(m + n) + k 2 )
k=1 k=1

≈ 4(mn2 − (m + n)n2 /2 + n3 /3) = 2mn2 − 2n3 /3

Y. Zhou Math-6316/CS-7366, SMU 109/209
 QR via Givens Rotations

 
cos θ − sin θ
Recall Givens rotation G(θ) = rotates x ∈ R2
sin θ cos θ
anticlockwisely by θ
To set an element to zero, choose cos θ and sin θ so that
    "q 2 #
cos θ − sin θ xi xi + xj2
=
sin θ cos θ xj 0

or
xi −xj
cos θ = q , sin θ = q
xi2 + xj2 xi2 + xj2

“Orthogonal Triangularization”

Y. Zhou Math-6316/CS-7366, SMU 110/209

 QR via Givens Rotations

Introduce zeros in column from bottom and up

     
× × × × × × × × ×
 × × × 
 (3,4)
 × × × 
 (2,3)
 X X X 
 (1,2)
 −→  −→  −→
 × × ×   X X X   0 X X 
× × × 0 X X × ×
     
X X X × × × × × ×
 0 X X  (3,4)  × ×  (2,3)  X X 
 (3,4)
 
 −→  −→  −→ R
 × ×   X X   0 X 
× × 0 X ×

Flop count 3mn2 − n3 (or 50% more than Householder QR)

Y. Zhou Math-6316/CS-7366, SMU 111/209

 Linear Least Squares Problems

In general, an over-determined system such as Ax = b,

(A ∈ Cm×n , with m > n), has no solution
A least square solution tries to minimize the 2-norm of the
residual r = b − Ax:

Least Square problem:

Given A ∈ Cm×n , m ≥ n, b ∈ Cm , solve minn kAx − bk2 .
x∈C

Geometric Interpretation
b
For any x ∈ Cn , Ax ∈ range(A)
Minimizing kb − Axk2 means finding r = b − Ax
the shortest distance from b to
b
range(A) y = Pb
= Ax ra n g e
Need Ax = Pb where P is an (A)
orthogonal projector onto range(A),
i.e., r ⊥ range(A)
Y. Zhou Math-6316/CS-7366, SMU 112/209
 Solving Least Squares Problems

Essentially, we solve Ax = Pb, which always has a solution.

Different ways representing P leads to different methods.
If A = QR, then P = QQ ∗

Ax = Pb =⇒ QRx = QQ ∗ b =⇒ Rx = Q ∗ b

If A = UΣV ∗ , than P = UU ∗

Ax = Pb =⇒ UΣV ∗ x = UU ∗ b =⇒ ΣV ∗ x = U ∗ b

(Most stable but also most expensive among the three)

If A is full rank, then P = A(A∗ A)-1 A∗ (note PP ∗ = P ∗ P, P 2 = P)

Ax = Pb =⇒ Ax = A(A∗ A)-1 A∗ b =⇒ A∗ Ax = A∗ b.

This is called the normal equations. (Least expensive, but also

least accurate among the three if A has close to linearly
dependent columns.)
Y. Zhou Math-6316/CS-7366, SMU 113/209
 Solving LS: via thin QR decomposition

Using thin QR: A = QR, Q ∈ Cm×n , R ∈ Cn×n .

Project b onto range(A) as Pb = QQ ∗ b
Insert into Ax = Pb to get QRx = QQ ∗ b, or Rx = Q ∗ b

Algorithm: Least Squares via QR Factorization

1. Compute the thin QR factorization A = QR
2. Compute the vector Q ∗ b (without forming Q)
3. Solve the upper-triangular system Rx = Q ∗ b for x

Major cost: thin QR Factorization ∼ 2mn2 − 2n3 /3 flops

Good stability, relatively fast. (Used in MATLAB’s “backslash” \)

Y. Zhou Math-6316/CS-7366, SMU 114/209

 Solving LS: via SVD

Compute SVD of A: A = UΣV ∗ , Q ∈ Cm×n , Σ ∈ Rn×n , V ∈ Cn×n .

Project b onto range(A) as Pb = UU ∗ b
Insert into Ax = Pb to get UΣV ∗ x = UU ∗ b, or ΣV ∗ x = U ∗ b

Algorithm: Least Squares via SVD

1. Compute the reduced SVD A = UΣV ∗
2. Compute the vector U ∗ b
3. Solve the diagonal system Σw = U ∗ b for w
4. Set x = Vw

Major cost: SVD of A ∼ 2mn2 + 11n3 flops

Very good stability properties, use if A is close to rank-deficient

Y. Zhou Math-6316/CS-7366, SMU 115/209

 Solving LS: via Normal Equations

If A has full rank, A∗ A is square, (hermitian) positive definite

Solve by Cholesky factorization (Gaussian elimination)

Algorithm: Least Squares via Normal Equations

1. Form the matrix A∗ A and the vector A∗ b
2. Compute the Cholesky factorization A∗ A = R ∗ R
3. Solve the lower-triangular system R ∗ w = A∗ b for w
4. Solve the upper-triangular system Rx = w for x

Major cost: Forming A∗ A and Cholesky ∼ mn2 + n3 /3 flops

Fast, but sensitive to rounding errors (particularly so when A is
close to rank deficient)

Y. Zhou Math-6316/CS-7366, SMU 116/209

 LS by normal equations

In fact, the normal equation A∗ Ax = A∗ b is a necessary condition for

minn kAx − bk2 (no need to assume A full rank).
x∈C
The is readily seen from the geometric view:

r ⊥ range (A) =⇒ A∗ r = A∗ (Ax − b) = 0 =⇒ A∗ Ax = A∗ b.

2
It can also be obtained by expanding minn kAx − bk2 as
x∈C

f (x) = x ∗ A∗ Ax − b∗ Ax − x ∗ A∗ b + b∗ b,

then set the first order derivative of f (x) w.r.t. x to 0. This also leads
to the normal equation A∗ Ax = A∗ b.

Y. Zhou Math-6316/CS-7366, SMU 117/209

 On solving linear equations Ax = b, A ∈ Cm×n (with m = n)
Gaussian Elimination via LU and pivoted LU
Cholesky decomposition for A SPD or (H)PD
Conditioning and stability

Y. Zhou Math-6316/CS-7366, SMU 118/209

 The LU Factorization

Compute A = LU, where L, U ∈ Cm×m , L is unit lower triangular,

U is upper triangular
Obtain U by sequentially subtracting multiples of rows:
Left multiply by elementary matrices, each Li introduces zeros
below diagonal of column i.

       
×××× ×××× ×××× ××××
 ×××× L1 L2 L3
  →  0 XXX
  → 
 ×××  → 
 ××× 
 ××××  0 XXX  0 XX  ××
×××× 0 XXX 0 XX 0 X
A L1 A L2 L1 A L3 L2 L1 A

Lm−1 · · · L2 L1 A = U =⇒ A = LU where L = L−1 −1 −1

1 L2 · · · Lm−1
| {z }
L−1

“Triangular triangularization”
Y. Zhou Math-6316/CS-7366, SMU 119/209
 The Matrices Lk

At step k, eliminate elements below Akk :

 ∗
xk = x1k ··· xkk xk+1,k ··· xmk
 ∗
Lk xk = x1k ··· xkk 0 ········· 0

xjk
The multipliers ℓjk = xkk appear in Lk :
 
1
 .. 

 . 
 m
 1  Y
Lk =  = Ea (k, j, −ℓjk )

 −ℓk+1,k 1 
 j=k+1
 .. .. 
 . . 
−ℓmk 1

Recall: Ea (k, j, c) = I + cej ekT , Ea-1 (k, j, c) = Ea (k, j, −c)

Y. Zhou Math-6316/CS-7366, SMU 120/209
 Structure of L in A = LU

Each Lk is an elementary matrix:

Let ℓk = [0, · · · , 0, ℓk+1,k , · · · , ℓm,n ]∗ , then Lk = I − ℓk ek∗ .
By Sherman-Morison, (or direct verification)

ℓk ek∗
L-1
k =I − ∗ = I + ℓk ek∗
ek ℓk − 1

L−1 −1 ∗ ∗ ∗ ∗
k Lk+1 = (I + ℓk ek )(I + ℓk+1 ek+1 ) = I + ℓk ek + ℓk+1 ek+1
The product L = L−1 −1 −1
1 L2 · · · Lm−1 is obtained by inserting ℓk into
the k-th column of I
 
1
 ℓ21 1 
 
L = L−1 −1 −1  ℓ31 ℓ32 1 
1 L2 · · · Lm−1 = 
 ..

.. .. .. 
 . . . . 
ℓm1 ℓm2 ··· ℓm,m−1 1

Y. Zhou Math-6316/CS-7366, SMU 121/209

 Gaussian Elimination (without pivoting)

Algorithm: Factorize A ∈ Cm×m into A = LU, (no pivoting)

L = I, U = A (can overwrite A by L and U to avoid using L, U)
For k = 1 to m − 1
for j = k + 1 to m
ℓjk = ujk /ukk
uj,k:m = uj,k:m − ℓjk uk,k:m

Inner loop can use matrix operations, e.g., (overwrite A)

f o r k = 1 : m−1
i f ( A ( k , k ) == 0 ) ,
e r r o r ( ' w i t h o u t p i v o t i n g , LU decomposition f a i l s ' )
else
A ( k +1: m , k ) = A ( k +1: m , k ) / A ( k , k ) ;
A ( k +1: m , k +1: m ) = A ( k +1: m , k +1: m )−A ( k +1: m , k ) * A ( k , k +1: m ) ;
end
end

Pm Pm
Operation count ∼ k=1 2(m − k)(m − k) ∼ 2 k=1 k 2 ∼ 2m3 /3
Y. Zhou Math-6316/CS-7366, SMU 122/209
 Pivoting

At step k of no pivoting LU, the (k, k) element is used (as pivot)

to introduce zeros in k-column below the (k, k) element
   
× × ××× × × ×××

 xkk X X X  

 xkk × × × 

 × ××× →   0 XXX  
 × ×××   0 XXX 
× ××× 0 XXX

But any nonzero element i ≥ k in column k can be used as pivot:

   
× × ××× × × ×××

 × ×××  

 0 XXX  

 × ××× →   0 XXX  
 xik X X X   xik × × × 
× ××× 0 XXX

Y. Zhou Math-6316/CS-7366, SMU 123/209

 Pivoting

Also, any nonzero row element j ≥ k can be used as pivot:

   
×× × ×× ×××××

 × × ××  

 X 0 XX  

 × × ×× →  X 0 XX  
 X xij X X   × xij × × 
× × ×× X 0 XX

Choose different pivots to avoid zero or very small pivots

(reduce instability) !
Pivoting means first exchanging rows (or columns) s.t. the
diagonal pivot has larger magnitude, then applying the standard
(no-pivot) LU
A computer code might account for the pivoting indirectly instead
of actually moving the data

Y. Zhou Math-6316/CS-7366, SMU 124/209

 Partial (row) Pivoting

Full pivoting searches among all valid pivots, i.e., at k-th step,
choose maxi≥k,j≥k |aij | as pivot, (interchange rows and columns),
expensive
Partial pivoting considers a pivot in column k only, i.e., choose
maxi≥k |aik | as pivot, (interchange rows)
     
× × ××× × × ××× × × ×××

 × ×××   P1 
 xik X X X  L1 
 xik × × ×  

 × × × ×  −→ 
  × × × ×  −→ 
  0 XXX  
 xik X X X   X XXX   0 XXX 
× ××× × ××× 0 XXX
Pivot selection Row interchange Elimination
In terms of matrices:
Lm−1 Pm−1 · · · L2 P2 L1 P1 A = U,
where Pi ’s are the elementary matrices, each used to switch two
rows when a pivot is necessary.
Y. Zhou Math-6316/CS-7366, SMU 125/209
 The PA = LU Factorization

To combine all Lk and all Pk into matrices, rewrite as

Lm−1 Pm−1 · · · L2 P2 L1 P1 A = U
(L′m−1 · · · L′2 L′1 )(Pm−1 · · · P2 P1 )A = U

where
−1 −1
L′k = Pm−1 · · · Pk+1 Lk Pk+1 · · · Pm−1

This gives the LU factorization of A

PA = LU

Y. Zhou Math-6316/CS-7366, SMU 126/209

 Gaussian Elimination with Partial Pivoting

Algorithm: Gaussian Elimination for PA = LU

U = A, L = I, P = I
for k = 1 to m − 1
Select i ≥ k to maximize |uik |
uk,k:m ↔ ui,k:m (interchange two rows)
ℓk,1:k−1 ↔ ℓi,1:k−1
pk,: ↔ pi,:
for j = k + 1 to m
ℓjk = ujk /ukk
uj,k:m = uj,k:m − ℓjk uk,k:m

Can overwrite A by L and U (saves the memory for storing L, U)

When used to solve Ax = b, no need to store P either, can apply
P directly to b and solve PAx = Pb =⇒ LUx = Pb.
Flops: similar to no pivoting, ∼ 2m3 /3.

Y. Zhou Math-6316/CS-7366, SMU 127/209

 Gaussian Elimination with Partial Pivoting

Matlab code using PPGE to solve Ax = b. Overwrite A by L and U,

P is not stored but directly applied to A and b.

f o r j = 1 : n−1
% choose t h e one w i t h l a r g e s t magnitude from A( j : n , j ) as p i v o t
[ amax , ip ] = max ( abs ( A ( j : n , j ) ) ) ;
% i p from above i s i n [ 1 : n−j + 1 ] , p o i n t i t t o t r u e row number i n A
ip = ip + j−1;
i f ( ip ˜= j ) ,
% a p p l y P j t o both A and b , t h i s i s n o t h i n g b u t row swamping
tmp=A ( ip , j : n ) ; A ( ip , j : n ) =A ( j , j : n ) ; A ( j , j : n ) =tmp ;
tmp = b ( ip ) ; b ( ip ) = b ( j ) ; b ( j ) = tmp ;
end
i f ( A ( j , j ) ˜=0) ,
% a p p l y t h e s t a n d a r d gauss e l i m i n a t i o n
A ( j +1: n , j ) = A ( j +1: n , j ) / A ( j , j ) ;
A ( j +1: n , j +1: n ) = A ( j +1: n , j +1: n ) − A ( j +1: n , j ) * A ( j , j +1: n ) ;
b ( j +1: n ) = b ( j +1: n ) − A ( j +1: n , j ) * b ( j ) ;
else
error ( ' singular matrix ' ) ;
end
end
x = t r i u ( A ) \b ;

Y. Zhou Math-6316/CS-7366, SMU 128/209

 Full Pivoting

If pivots are selected from a different column, permutation

matrices Qk for the columns are required:

Lm−1 Pm−1 · · · L2 P2 L1 P1 AQ1 Q2 · · · Qm−1 = U

(L′m−1 · · · L′2 L′1 )(Pm−1 · · · P2 P1 )A(Q1 Q2 · · · Qm−1 ) = U

Set

L = (L′m−1 · · · L′2 L′1 )−1

P = Pm−1 · · · P2 P1
Q = Q1 Q2 · · · Qm−1

to obtain

PAQ = LU

Y. Zhou Math-6316/CS-7366, SMU 129/209

 Cholesky Factorization

Compute with R upper triangular;

or A = LDL∗ for L unit lower triangular
Need A to be symmetric/hermitian;
need positive definiteness1 of A for A = R ∗ R
Utilize symmetry, complexity is ∼ m3 /3
(reduced by half that of general LU)
Some applications: Solve Ax = b when A is SPD, such as in the
Hessian matrices (quasi-Newton methods for nonlinear
optimization), and covariance matrices (Monte Carlo simulation,
and Kalman filters)

1 For A ∈ Cn×n , A is PD if x ∗ Ax > 0, ∀x ∈ Cn 6= 0; this condition implicitly

guarantees A∗ = A. While for A ∈ Rn×n , A is PD if x T Ax > 0, ∀x ∈ Rn 6= 0; but this

does not guarantee AT = A, hence one needs A to be SPD to guarantee existence of
A = R T R.
Y. Zhou Math-6316/CS-7366, SMU 130/209
 Computing Cholesky Factorization A = R ∗ R

√
Let α = a11 . The first step for A = R ∗ R is
    
a11 w ∗ α 0 α w ∗ /α
A := =
w A(1) w/α I 0 A(1) − ww ∗ /a11
   
α 0 1 0 α w ∗ /α
= =: R1∗ A1 R1
w/α I 0 A(1) − ww ∗ /a11 0 I
p
That is, R(1,1) = A(1,1) , R(1,2:n) = A∗(2:n,1) /R(1,1) .

Can apply the same to A(2) := A(1) − ww ∗ /a11 (also PD, why?)
     
1 0 1 0 1 0
A= R1∗ R = R1∗ R2∗ A2 R2 R1 , R2 = , A2 =
0 R̃2∗ Ã2 R̃2 1 0 R̃2 0 Ã2
q ∗
(2) (2)
Note R(2,2) = A(1,1) , R(2,2:n) = A(2:n,1) /R(2,2) .
Apply the same recursively to diagonal block of A(k)
Y. Zhou Math-6316/CS-7366, SMU 131/209
 Computing A = R ∗ R (A is PD, two versions)

R = A;
for k = 1 : n
f o r j = k+1 : n % o n l y update upper t r i a n g u l a r p a r t ( symmetry )
R(j , j : n) = R(j , j : n) − R(k , j : n) *R(k , j) ' / R(k , k) ;
end
i f ( R ( k , k ) <= 0 ) ,
e r r o r ( ' A i s n o t HPD, t r y ' ' A=Rˆ * DR ' ' i n s t e a d ' ) ,
end
R(k , k : n) = R(k , k : n) / sqrt (R(k , k) ) ;
end
R = triu (R) ;

R = zeros ( n , n ) ;
for i = 1 : n ,
tmp = A ( i , i ) − R ( 1 : i−1,i ) ' * R ( 1 : i−1,i ) ;
i f ( tmp <= 0 ) ,
e r r o r ( ' A i s n o t HPD, t r y ' ' A=Rˆ * DR ' ' i n s t e a d ' ) ,
end
R ( i , i ) = s q r t ( tmp ) ;
f o r j = i+1 : n
R ( i , j ) = ( A ( i , j ) − R ( 1 : i−1,i ) ' * R ( 1 : i−1,j ) ) / R ( i , i ) ;
end
end

Y. Zhou Math-6316/CS-7366, SMU 132/209

 Computing A = R ∗ DR (two of several versions)

R = eye ( n ) ; % t h e r e t u r n e d R i s u n i t upper t r i a n g u l a r
f o r j = 1 : n−1,
dv ( j ) = r e a l ( A ( j , j ) ) ;
R ( j , j +1: n ) = A ( j , j +1: n ) / dv ( j ) ;
f o r i = j+1 : n % o n l y update upper t r i a n g u l a r row elements
A ( i , i : n ) = A ( i , i : n ) − R ( j , i ) ' * dv ( j ) * R ( j , i : n ) ;
end
end
dv ( n ) = A ( n , n ) ; % D= d i a g ( dv ( 1 : n ) )

R = eye ( n ) ;
f o r j = 1 : n−1,
dv ( j ) = r e a l ( A ( j , j ) ) ;
f o r i = j +1: n
R ( j , i ) = A ( j , i ) / dv ( j ) ;
f o r k = j+1 : i %o n l y update l o w e r t r i a n g u l a r column elements
A ( k , i ) = A ( k , i ) − R ( j , i ) * dv ( j ) * R ( j , k ) ' ;
end
end
end
dv ( n ) = A ( n , n ) ;

Y. Zhou Math-6316/CS-7366, SMU 133/209

 Solving nonsingular triangular systems

Solving Ux = b: (backward substitution)

n
X
uik xk = bi , i = 1, · · · , n
k=i
Pn
bi − k=i+1 uik xk
=⇒ xi = , i = n, · · · , 1
uii

Solving Lx = b: (forward substitution)

i
X
lik xk = bi , i = 1, · · · , n
k=1
Pi−1
bi − k=1 lik xk
=⇒ xi = , i = 1, · · · , n
lii

Complexity for triangular solves: ∼ O(n2 )

Y. Zhou Math-6316/CS-7366, SMU 134/209
 On Conditioning and Stabilities
General definition of Condition Numbers
Accuracy of (numerical) solutions
Stability
Forward stability
Backward stability
Mixed stability
Main picture: Accuracy depend on two things
1. Conditioning of the underlying problem
2. Stability of the algorithm used to solve the problem

Y. Zhou Math-6316/CS-7366, SMU 135/209

 Conditioning, Condition number

Absolute Condition Number of a function f : X → Y at x:

kf (x + δx) − f (x)k
κ̂(f , x) = sup
δx6=0 kδxk

If f is differentiable,
κ̂(f , x) = kJf (x)k
where the Jacobian (Jf )ij = ∂fi /∂xj , and the matrix norm is
induced by the norms on X and Y .
Relative Condition Number
  
κ̂ kf (x + δx) − f (x)k kδxk
κ(f , x) = = sup
kf (x)k / kxk δx6=0 kf (x)k kxk

If f is differentiable,
kJf (x)k
κ(f , x) =
kf (x)k/kxk
Y. Zhou Math-6316/CS-7366, SMU 136/209
 Conditioning, Condition number

Example: The function f (x) = αx

Absolute condition number κ̂ = kJf k = α
kJf k α
Relative condition number κ = kf (x)k/kxk = αx/x
=1
√
Example: The function f (x) = x
1
Absolute condition number κ̂ = kJf k = √
2 x √
kJf k 1/(2 x) 1
Relative condition number κ = kf (x)k/kxk
= √
x/x
= 2

Example: The function f (x) = x1 − x2 (with ∞-norms)

Absolute condition number κ̂ = kJf k = k(1, −1)k = 2
Relative condition number
kJf k 2
κ= =
kf (x)k/kxk |x1 − x2 |/ max{|x1 |, |x2 |}

Ill-conditioned (in the relative sense) when x1 ≈ x2

(This is the well-known cancellation problem when subtracting two
close numbers)

Y. Zhou Math-6316/CS-7366, SMU 137/209

 Conditioning, Condition number

Consider f (x) = Ax, with A ∈ Cm×n

kJf k kxk
κ= = kAk
kf (x)k/kxk kAxk

For A square and nonsingular, use kxk/kAxk ≤ kA−1 k:

κ ≤ kAkkA−1 k

(equality achieved for the last right singular vector x = vn )

κ = kAkkA−1 k is also the condition number for f (b) = A−1 b
(solution of linear system)
Condition number of matrix A:
σ1
κ(A) := kAk2 kA−1 k2 =
σn

Y. Zhou Math-6316/CS-7366, SMU 138/209

 Condition of System of Equations

For fixed b, consider f (A) = A−1 b

Perturb A by δA and find perturbation δx:

(A + δA)(x + δx) = b

Use Ax = b and assume (δA)(δx) ≈ 0:

(δA)x + A(δx) = 0 =⇒ δx = −A−1 (δA)x

Condition number of problem f :

 
kδxk kδAk kA−1 kkδAkkxk kδAk
κ= ≤ = kA−1 kkAk = κ(A)
kxk kAk kxk kAk

Y. Zhou Math-6316/CS-7366, SMU 139/209

 O(ǫmachine ) notation

The notation ϕ(t) = O(ψ(t)) means there is a constant C such

that, for t close to a limit (often 0 or ∞), |ϕ(t)| ≤ Cψ(t)
Example: sin2 t = O(t 2 ) as t → 0 means | sin2 t| ≤ Ct 2 for some
C
If ϕ depends on additional variables, the notation

ϕ(s, t) = O(ψ(t)) uniformly in s

means there is a constant C such that |ϕ(s, t)| ≤ Cψ(t) for any s
Example: (sin2 t)(sin2 s) = O(t 2 ) uniformly as t → 0, but not if
sin2 s is replaced by s2
In bounds such as kx̃ − xk ≤ Cκ(A)ǫmachine kxk, C does not
depend on A or b, but it might depend on the dimension m

Y. Zhou Math-6316/CS-7366, SMU 140/209

 Accuracy of an algorithm

For a problem described as f : X → Y , which is assumed

differentiable,
Apply (discrete) approximation and solve by an algorithm,
described as f̃ : X → Y . (f̃ (x) is the computed value of f (x))
f̃ (x) has absolute error kf̃ (x) − f (x)k and relative error
kf̃ (x) − f (x)k
kf (x)k
Algorithm is accurate if (for all x ∈ X )
kf̃ (x) − f (x)k
= O(ǫmachine )
kf (x)k
where O(ǫmachine ) is “on the order of ǫmachine ”
Constant in O(ǫmachine ) is likely to be large in many problems
(rounding error exists for x)
More realistic to compare f̃ (x) with f (x̃), where x̃ is an
approximation of the exact x
Y. Zhou Math-6316/CS-7366, SMU 141/209
 Stability of an algorithm

An algorithm f̃ : X → Y for a problem f : X → Y is stable if (for all

x ∈ X)

kf̃ (x) − f (x̃)k

= O(ǫmachine )
kf (x̃)k

for some x̃ with

kx̃ − xk
= O(ǫmachine )
kxk

“Nearly the right answer to nearly right data/problem”

An algorithm f̃ for a problem f is backward stable if (for all x ∈ X )

kx̃ − xk
f̃ (x) = f (x̃) for some x̃ with = O(ǫmachine )
kxk

“Exactly the right answer to nearly the right data/problem”

Y. Zhou Math-6316/CS-7366, SMU 142/209
 Stability, Backward Stability

f (x̃)
b f̃ (x)
f
forward error
x̃
b kf (x)−f̃ (x)k
f̃
backward error b

kx−x̃k f f (x)
b

x
f̃ is stable (in the mixed forward-backward sense): Nearly right solution to a
nearly right problem.

f (x̃) = f̃ (x)
b

f forward error
x̃
b kf (x)−f̃ (x)k
f̃
backward error b

kx−x̃k f f (x)
b

x
f̃ is backward stable: Exactly right solution to a nearly right problem.
Y. Zhou Math-6316/CS-7366, SMU 143/209
 Linking forward error with backward error

Assume that forward error, backward error, and condition number are
defined mutually consistently, then a rule of thumb in error analysis is

(forward error) ≤ C ∗ (condition number) ∗ (backward error)

That is,
f (x) − f̃ (x) ≤ Cκ̂(f , x) kx − x̃k ,

which may be considered as an approximation of the 1st order Taylor

expansion.
If f is backward stable, then by the definition of κ̂(f , x) we see the
constant C can be set to 1.
Idea of backward error analysis: Backward error reveals the
stability of the algorithm, isolated from the conditioning of the
underlying problem. (While forward error depends on both
stability of algorithm and conditioning of problem.)
Y. Zhou Math-6316/CS-7366, SMU 144/209
 Three types of stability

forward error kf̃ (x)−f (x)k

Small condition number , i.e., ( κ(f ,x) )
=⇒ forward stable algorithm
Small mixed error ( f̃ (x) − f (x̃) )
=⇒ stable algorithm (in mixed forward-backward sense)
Small backward error (kx̃ − xk)
=⇒ backward stable algorithm

Backward stability is the strongest among the three:

Backward stable =⇒ stable
Backward stable =⇒ forward stable
Comment: However, the above definition for forward stability is not
universally accepted. It is also possible to require small “forward error” for
forward stability. In this case “backward stability” does not imply “forward
stability”. An example is the QR factorization by GS, which may be
considered “forward unstable” (Q factor may not be orthogonal), though it is
backward stable.
Y. Zhou Math-6316/CS-7366, SMU 145/209
 Accuracy of a Backward Stable Algorithm

Theorem: If a backward stable algorithm is used to solve a problem f

with condition number κ, then the relative errors satisfy

kf̃ (x) − f (x)k

= O(κ(f , x)ǫmachine ) .
kf (x)k

Proof. The definition of condition number gives

kf (x̃) − f (x)k kx̃ − xk

≤ (κ(f , x) + o(1))
kf (x)k kxk

where o(1) → 0 as ǫmachine → 0.

Backward stability of f̃ means f̃ (x) = f (x̃) for x̃ such that

kx̃ − xk
= O(ǫmachine )
kxk

Combining these gives the desired result.

Y. Zhou Math-6316/CS-7366, SMU 146/209
 Backward Stability of Householder QR

For a QR factorization A = QR computed by Householder

triangularization, the factors Q̃ and R̃ satisfy

kδAk
Q̃ R̃ = A + δA, = O(ǫmachine )
kAk

Exactly the right QR factorization of a slightly perturbed A

Here R̃ is the R computed by the algorithm using floating points
However, Q̃ is a product of exactly unitary reflectors:

Q̃ = Q̃1 Q̃2 · · · Q̃n

where Q̃k is implicitly given by the computed ṽk (since Q is

generally not formed explicitly)

Y. Zhou Math-6316/CS-7366, SMU 147/209

 Backward Stability of Solving Ax = b with QR

Algorithm: Solving Ax = b by QR Factorization

1. QR = A using Householder, represent Q by reflectors
2. y = Q ∗ b implicitly using reflectors
3. x = R −1 y by back substitution

Step 1 is backward stable (from previous slide)

Step 2 can be shown to be backward stable:

(Q̃ + δQ)ỹ = b, kδQk = O(ǫmachine )

Step 3 is backward stable (will be shown later):

kδRk
(R̃ + δR)x̃ = ỹ, = O(ǫmachine )
kR̃k

Y. Zhou Math-6316/CS-7366, SMU 148/209

 Backward Stability of Solving Ax = b with QR

Put the three steps together to show backward stability of the

algorithm:

k∆Ak
(A + ∆A)x̃ = b, = O(ǫmachine )
kAk

Proof. Steps 2 and 3 give

h i
b = (Q̃ + δQ)(R̃ + δR)x̃ = Q̃ R̃ + (δQ)R̃ + Q̃(δR) + (δQ)(δR) x̃

Step 1 (backward stability of Householder) gives

h i
b = A + δA + (δQ)R̃ + Q̃(δR) + (δQ)(δR) x̃
= (A + ∆A)x̃

Y. Zhou Math-6316/CS-7366, SMU 149/209

 Backward Stability of Solving Ax = b with QR

δA is small compared to A, therefore

kR̃k kA + δAk
≤ kQ̃ ∗ k = O(1)
kAk kAk

Now show that each term in ∆A is small:

k(δQ)R̃k kR̃k
≤ kδQk = O(ǫmachine )
kAk kAk
kQ̃(δR)k kδRk kR̃k
≤ kQ̃k = O(ǫmachine )
kAk kR̃k kAk
k(δQ)(δR)k kδRk
≤ kδQk = O(ǫ2machine )
kAk kAk

Y. Zhou Math-6316/CS-7366, SMU 150/209

 Backward Stability of Solving Ax = b with QR

Add the terms to show that ∆A is small:

k∆Ak kδAk k(δQ)R̃k kQ̃(δR)k k(δQ)(δR)k

≤ + + +
kAk kAk kAk kAk kAk
= O(ǫmachine )

Since the algorithm is backward stable, it is also accurate:

kx̃ − xk
= O(κ(A)ǫmachine )
kxk

Y. Zhou Math-6316/CS-7366, SMU 151/209

 On Floating Points
Representation
Precision (or size of Significand, or significant digits):
an integer p ≥ 1
Exponent size:
two bounds emin and emax , with an integer e ∈ [emin , emax ]
Base (or Radix):
an integer β ≥ 2
β = 2 — binary format (most common in computers)
β = 10 — decimal format
β = 16 — hexadecimal
IEEE single and double precision floating point data type
Floating point arithmetic

Y. Zhou Math-6316/CS-7366, SMU 152/209

 Floating Point Representations

A floating point (number) system is a subset of the real numbers R,

with elements represented by

m e
±mβ e−p = ± β
βp

The β is the base (also called radix)

The p is the precision
The e is the exponent — an integer bounded by [emin , emax ]
The m is the significand — an integer satisfying 0 ≤ m ≤ β p − 1
An equivalent form of the floating point (number) is
 
e d1 d2 dp
±0.d1 d2 · · · dp × β = ± + 2 + · · · + p βe,
β β β

0 ≤ di ≤ β − 1, and d1 6= 0 for normalized numbers.

Y. Zhou Math-6316/CS-7366, SMU 153/209
 Floating Point Representations (continued)

Two advantages of normalized representation:

Uniqueness of representation
For β = 2, d1 ≡ 1, which does not need to be stored (saved one
extra bit for a longer significand (also called mantissa))
For normalized floating points: To represent 0, use e = emin − 1.
For nonzero normalized floating points, β p−1 ≤ m ≤ β p − 1
(uniqueness of representation)
Range of nonzero normalized floating points (symmetric w.r.t. 0)

β emin −1 ≤ |fl(y)| ≤ β emax (1 − β −p )

1 emin
Minimum when d1 = 1, di = 0 (i > 1), e = emin , i.e., β
β .
! di = β − 1 (i ≥ 1), e = emax , i.e.,
Maximum when
p
X β−1
β emax = β emax (1 − β −p ).
βi
i=1
Or, by using mβ e−p : β p−1 β emin −p ≤ |fl(y )| ≤ (β p − 1)β emax −p .
Y. Zhou Math-6316/CS-7366, SMU 154/209
 Machine epsilon and unit roundoff

Machine epsilon (ǫmachine ), is sometimes called unit roundoff (µ),

(while some authors uses µ = ǫmachine /2 for a good reason)
The IEEE standard does not define the terms “machine epsilon”
and unit roundoff
ǫmachine provides an upper bound on the relative error due to
rounding. That is, for any non-zero real number y within the
normalized range of a floating point system,

fl(y ) − y
≤ ǫmachine
y

A few (essentially) equivalent (but slightly different) definitions exist:

E.g., ǫmachine measures the distance from 1 to the adjacent larger
floating point, i.e., from 1
β
β to ( β1 + β
1
p )β, ǫmachine = β 1−p
1−p
The definition ǫmachine = β assumes “rounding to zero” (i.e.,
directed rounding towards zero with truncation)
If “rounding to nearest” is used, then ǫmachine = 21 β 1−p , which is the
unit roundoff as is (quite often) used

Y. Zhou Math-6316/CS-7366, SMU 155/209

 Floating Point Numbers

The gaps between adjacent numbers scale with the size of the
numbers
For all x ∈ R in the range of a floating point system, there exists
a floating point number fl(x) such that |x − fl(x)| ≤ ǫmachine |x|
Example: β = 2, p = 3, emin = −1, emax = 3, normalized
 
d1 d2 d3
+ 2 + 3 2e , e ∈ {−1, 0, 1, 2, 3},
2 2 2
d1 ≡ 1, d2 , d3 ∈ {0, 1}, (essentially only need two bits for p = 3)

1 1
0 4 2 1 2 3 4 5 6 7
Number of floating points between adjacent powers of 2: 2p−1 − 1.
(# of floating points between adjacent powers of β: (β − 1)β p−1 − 1)
Y. Zhou Math-6316/CS-7366, SMU 156/209
 Denormalized (or Subnormal) Numbers

With normalized significand, ∃ a “gap” between 0 and β emin −1

This can result in x − y = 0 even though x 6= y, and code
fragments like if x 6= y then z = 1/(x − y) might break
Solution: Allow non-normalized significand when the exponent is
emin (i.e, d1 can be 0 when e = emin )
This gradual underflow garantees that
x = y ⇐⇒ x − y = 0
Subnormal numbers have lower relative precision than
normalized numbers
Example: β = 2, p = 3, emin = −1, emax = 3


d1 /2 + d2 2−2 + d3 2−3 2e , e ∈ {−1, 0, 1, 2, 3}, di ∈ {0, 1}.

1 1
0 4 2 1 2 3 4 5 6 7
Y. Zhou Math-6316/CS-7366, SMU 157/209
 Two equivalent floating point representations

The (normalized) representation just discussed uses

m e
±mβ e−p = ± β , where β p−1 ≤ m ≤ β p − 1
βp

The range of m implies that this representation is essentially

 
d1 d2 dp
±0.d1 d2 · · · dp × β e = ± + 2 + · · · + p βe,
β β β

where 0 ≤ di ≤ β − 1, and d1 6= 0.
Another equivalent representation (more often used, as used in IEEE) is
 
d2 dp
±d1 .d2 · · · dp × β e−1 = ± d1 + 1 + · · · + p−1 β e−1 ,
β β

where 0 ≤ di ≤ β − 1, and d1 6= 0.
No essential difference at all, except that in order to represent the same floating
point numbers, the emin and emax of the first representation should be 1 greater
than that of the second representation. (which can cause some confusion.)
For example, the previous example using the second representation should be
β = 2, p = 3, emin = −2, emax = 2.
Y. Zhou Math-6316/CS-7366, SMU 158/209
 An exercise

The following shows a portion of a floating point system

0 β emin β emin +1 β emin +2 β emin +3

0 β emin β emin +1 β emin +2 β emin +3

The top one contains the normalized, while the bottom one contains
both the normalized and the subnormal, floating points.
1. Which representation is the system using,
the 0.d1 d2 · · · dp × β e or the d1 .d2 · · · dp × β e ?
2. Determine the possible values of β and p for this system.

Y. Zhou Math-6316/CS-7366, SMU 159/209

 An exercise

The following shows a portion of a floating point system

0 β emin β emin +1 β emin +2 β emin +3

0 β emin β emin +1 β emin +2 β emin +3

The top one contains the normalized, while the bottom one contains
both the normalized and the subnormal, floating points.
1. Which representation is the system using,
the 0.d1 d2 · · · dp × β e or the d1 .d2 · · · dp × β e ?
2. Determine the possible values of β and p for this system.
Answer: To solve this problem, apply the formula that determines the number of floating points between adjacent powers of β,
which is (β − 1)β p−1 − 1. (This formula can be obtained in several ways.)
Here, since (β − 1)β p−1 − 1 = 11, the only two integer solution pairs are (β, p) = (4, 2) and (13, 1). (Note the proportion
of gap is not drawn correctly to reveal the value of β.)

Y. Zhou Math-6316/CS-7366, SMU 159/209

 Special Quantities

±∞ is returned when an operation overflows

x/ ± ∞ = 0 for any number x, x/0 = ±∞ for any nonzero
number x
Operations with infinity are defined as limits, e.g.

4 − ∞ = lim 4 − x = −∞
x→∞

NaN (Not a Number) is returned when the an operation has no

well-defined finite or infinite result .
— Examples: ∞ − ∞, ∞/∞, 0/0, NaN ⊙ x

Y. Zhou Math-6316/CS-7366, SMU 160/209

 IEEE 754 binary formats

Floating-point numbers are stored in computer data as three parts

(from left to right): 1. the sign bit, 2. the exponents, 3. the significand.
Sign bit (S) Exponent bits (E) Significand bits (M)
The IEEE 754 standard was created in the early 1980s (published version
IEEE 754-1985), which has been followed by almost all modern machines.
Current version is IEEE 754-2008, which is a revision of IEEE 754-1985 and
adds the half-precision type.
IEEE 754 standard represent floating point data using bit sizes as
Exponent bits, Significand bits,
Precision Type Sign Total bits Exponent bias
[emin , emax ] (bits precision)
Half 1 5, [-14,15] 10, (11) 16 15
Single 1 8, [-126,127] 23, (24) 32 127
Double 1 11, [-1022,1023] 52, (53) 64 1023
Quadruple 1 15, [-16382, 16383] 112, (113) 128 16383

In binary formats the exponent is stored as an unsigned number,

with a fixed ”bias” to account for the ± sign of an exponent.
The listed [emin , emax ] assume the 1.d1 d2 d3 · · · dp × 2e format.
Y. Zhou Math-6316/CS-7366, SMU 161/209
 IEEE Single Precision

1 sign bit, 8 exponent bits, 23 significand bits:

0 00000000 0000000000000000000000000000000
S E (8 bits) M (23 physical bits, effective 24 bits)
emin =1−127=−126 223 −1 # of floating reals in (2e , 2e+1 ),
0/1 emax =28 −2−127=127 for every integer e∈[emin ,emax ]
Represented number:

(−1)S × 1.M × 2E−127

Special cases:
E =0 0 < E < 255 E = 255
M=0 ±0 Powers of 2 ±∞
M 6= 0 Denormalized Ordinary numbers NaN

Y. Zhou Math-6316/CS-7366, SMU 162/209

 IEEE Single Precision

1 sign bit, 8 exponent bits, 23 significand bits:

0 00000000 0000000000000000000000000000000
S E (8 bits) M (23 physical bits, effective 24 bits)
emin =1−127=−126 223 −1 # of floating reals in (2e , 2e+1 ),
0/1 emax =28 −2−127=127 for every integer e∈[emin ,emax ]
Represented number:

(−1)S × 1.M × 2E−127

Special cases:
E =0 0 < E < 255 E = 255
M=0 ±0 Powers of 2 ±∞
M 6= 0 Denormalized Ordinary numbers NaN
Comment: Giving up two strings for exponents (representing E = 0 and E = 255) to store the special 0 and ∞.
That is why emin = −126 and emax = 127.

Y. Zhou Math-6316/CS-7366, SMU 162/209

 IEEE Single Precision, Examples

S E M Quantity
0 11111111 00000100000000000000000 NaN
1 11111111 00100010001001010101010 NaN
0 11111111 00000000000000000000000 ∞
0 10000001 10100000000000000000000 +1 · 2129−127 · 1.101 = 6.5
0 10000000 00000000000000000000000 +1 · 2128−127 · 1.0 = 2
0 00000001 00000000000000000000000 +1 · 21−127 · 1.0 = 2−126
0 00000000 10000000000000000000000 +1 · 2−126 · 0.1 = 2−127
0 00000000 00000000000000000000001 +1 · 2−126 · 2−23 = 2−149
0 00000000 00000000000000000000000 0
1 00000000 00000000000000000000000 −0
1 10000001 10100000000000000000000 −1 · 2129−127 · 1.101 = −6.5
1 11111111 00000000000000000000000 −∞

Y. Zhou Math-6316/CS-7366, SMU 163/209

 IEEE Single and Double Precision binary data type

Single precision Double precision

Significand size (p) 24 bits 53 bits
Exponent size 8 bits 11
Exponent bias 27 − 1 = 127 210 − 1 = 1023
Total size 32 bits 64 bits
emax +127 +1023
emin -126 -1022
Smallest normalized 2−126 ≈ 10−38 2−1022 ≈ 10−308
Largest normalized 2127 ≈ 1038 21023 ≈ 10308
unit roundoff (β −p ) 2−24 ≈ 6 · 10−8 2−53 ≈ 10−16

Y. Zhou Math-6316/CS-7366, SMU 164/209

 Floating Point Arithmetic

Define fl(x) as the closest floating point approximation to x

By the definition of ǫmachine , we have for the relative error:
For all x ∈ R in the range of a floating point system,
there exists ǫ with |ǫ| ≤ ǫmachine such that fl(x) = x(1 + ǫ)
The result of an operation ⊛ using floating point numbers is
fl(a ⊛ b)
The arithmetic is said to rounds correctly if fl(a ⊛ b) is the
nearest floating point number to a ⊛ b. In a floating point system
that rounds correctly (IEEE standard does), the following
property holds:
For all floating point x, y, there exists ǫ with |ǫ| ≤ ǫmachine such
that x ⊛ y = (x ∗ y)(1 + ǫ)
Tie-breaking rule: Round to nearest even (i.e., set the least
significant bit to 0)

Y. Zhou Math-6316/CS-7366, SMU 165/209

 A few examples (In Matlab, with IEEE single precision)

>> single ( 2 ˆ 2 3 + [ 1 : 2 2 ] ) − single ( 2 ˆ 2 3 )

ans = 1 2 3 4 5 6 7 8 9 10 11 ←֓
12 13 14 15 16 17 18 19 20 21 22
>> single ( 2 ˆ 2 4 + [ 1 : 2 2 ] ) − single ( 2 ˆ 2 4 )
ans = 0 2 4 4 4 6 8 8 8 10 12 ←֓
12 12 14 16 16 16 18 20 20 20 22
>> single ( 2 ˆ 2 5 + [ 1 : 2 2 ] ) − single ( 2 ˆ 2 5 )
ans = 0 0 4 4 4 8 8 8 8 8 12 ←֓
12 12 16 16 16 16 16 20 20 20 24
>> single ( 2 ˆ 2 6 + [ 1 : 2 2 ] ) − single ( 2 ˆ 2 6 )
ans = 0 0 0 0 8 8 8 8 8 8 8 ←֓
16 16 16 16 16 16 16 16 16 24 24
>> single ( 2 ˆ 2 7 ) + [ 1 : 2 2 ] − single ( 2 ˆ 2 7 )
ans = 0 0 0 0 0 0 0 0 16 16 16 ←֓
16 16 16 16 16 16 16 16 16 16 16
>> single ( 2 ˆ 2 8 ) + [ 1 : 2 2 ] − single ( 2 ˆ 2 8 )
ans = 0 0 0 0 0 0 0 0 0 0 0 ←֓
0 0 0 0 0 32 32 32 32 32 32
>> single ( 2 ˆ 2 9 ) + [ 1 : 2 2 ] − single ( 2 ˆ 2 9 )
ans = 0 0 0 0 0 0 0 0 0 0 0 ←֓
0 0 0 0 0 0 0 0 0 0 0

Y. Zhou Math-6316/CS-7366, SMU 166/209

 A few examples (In Matlab, with IEEE double precision)

>> 2ˆ52 + [1:22] − 2ˆ52

ans = 1 2 3 4 5 6 7 8 9 10 11 ←֓
12 13 14 15 16 17 18 19 20 21 22
>> 2ˆ53 + [1:22] − 2ˆ53
ans = 0 2 4 4 4 6 8 8 8 10 12 ←֓
12 12 14 16 16 16 18 20 20 20 22
>> 2ˆ54 + [1:22] − 2ˆ54
ans = 0 0 4 4 4 8 8 8 8 8 12 ←֓
12 12 16 16 16 16 16 20 20 20 24
>> 2ˆ55 + [1:22] − 2ˆ55
ans = 0 0 0 0 8 8 8 8 8 8 8 ←֓
16 16 16 16 16 16 16 16 16 24 24
>> 2ˆ56 + [1:22] − 2ˆ56
ans = 0 0 0 0 0 0 0 0 16 16 16 ←֓
16 16 16 16 16 16 16 16 16 16 16
>> 2ˆ57 + [1:22] − 2ˆ57
ans = 0 0 0 0 0 0 0 0 0 0 0 ←֓
0 0 0 0 0 32 32 32 32 32 32
>> 2ˆ58 + [ 1 : 2 2 ] − 2ˆ58
ans = 0 0 0 0 0 0 0 0 0 0 0 ←֓
0 0 0 0 0 0 0 0 0 0 0

Y. Zhou Math-6316/CS-7366, SMU 167/209

 A few examples (In Matlab, with IEEE double precision)

>> f o r m a t long e

>> eps / 2
ans = 1.110223024625157e−16
>> 1 . + eps / 2 − 1 .
ans = 0

>> eps / 1 . 5
ans = 1.480297366166875e−16
>> 1 . + eps / 1 . 5 − 1 .
ans = 2.220446049250313e−16

>> 2 . + eps − 2 .
ans = 0
>> 2 . + 1 . 1 * eps − 2 .
ans = 4.440892098500626e−16
>> 2 . + 2 * eps − 2 .
ans = 4.440892098500626e−16

>> 4 . + 2 * eps − 4 .
ans = 0
>> 4 . + 3 * eps − 4 .
ans = 8.881784197001252e−16
>> 4 . + 4 * eps − 4 .
ans = 8.881784197001252e−16

Y. Zhou Math-6316/CS-7366, SMU 168/209

 A few examples (In Matlab, with IEEE double precision)

>> 2 ˆ 9 * eps
ans = 1.136868377216160e−13
>> 1024. + 2 ˆ 9 * eps − 1024.
ans = 0
>> 1024. + ( 1 + 1 . e−16) * 2 ˆ 9 * eps − 1024.
ans = 0
>> 1024. + (1+ eps ) * 2 ˆ 9 * eps − 1024.
ans = 2.273736754432321e−13
>> 1024. + 2 ˆ 1 0 * eps − 1024.
ans = 2.273736754432321e−13

>> 2 ˆ 1 1 . + 2 ˆ 1 0 * eps − 2 ˆ 1 1 .
ans = 0
>> 3 * 2 ˆ 1 0 * eps
ans = 6.821210263296962e−13
>> [ 2ˆ11 + 3 * 2 ˆ 1 0 * eps − 2 ˆ 1 1 , 2ˆ11 + 5 * 2 ˆ 1 0 * eps − 2ˆ11 ]
ans = 9.094947017729282e−13 9.094947017729282e−13

>> 2ˆ1000 * eps

ans = 2.379227053564453e+285
>> 2ˆ1001+ 2ˆ1000 * eps − 2ˆ1001
ans = 0

>> [ 2ˆ1022 * eps , 2ˆ1023 + 2ˆ1022 * eps − 2ˆ1023 ]

ans = 9.979201547673599e+291 0

Y. Zhou Math-6316/CS-7366, SMU 169/209

 On eigenvalue problems and related algorithms
Properties related to eigen-problems
A few representative algorithms
Power method, inverse iteration, shift-inverse iteration
RQI
The QR algorithm
Jacobi iteration, Divide-and-Conquer
Computing SVD

Y. Zhou Math-6316/CS-7366, SMU 170/209

 The Eigenvalue Problem

The standard eigenvalue problem for m × m matrix A is

Ax = λx

with eigenvalues λ and eigenvectors x (x 6= 0)

In the direction of an eigenvector, A is “condensed” into a scalar λ
Eigenvalue decomposition of A: (assume A has complete
eigenvectors)

A = X ΛX −1 or AX = X Λ

Columns of X are eigenvectors, with corresponding eigenvalues

on diagonal of Λ
In “eigenvector coordinates”, A is diagonal:

Ax = b → (X −1 b) = Λ(X −1 x)

Y. Zhou Math-6316/CS-7366, SMU 171/209

 Eigen-subspace, invariant subspace, multiplicity

The span of eigenvectors corresponding to an eigenvalue λ form

an eigen-subspace Eλ
Dimension of Eλ = dim(null(A − λI)) = geometric multiplicity of λ
The span of k linearly independent eigenvectors (corresponding
to eigenvalues) form a dimension-k eigen-subspace Yk , which is
invariant under A
AYk = Yk Sk , with Sk ∈ Ck×k
The characteristic polynomial of A is
pA (z) = det(zI − A) = (z − λ1 )(z − λ2 ) · · · (z − λm )
λ is eigenvalue of A ⇐⇒ pA (λ) = 0
If λ is eigenvalue, then ∃x 6= 0, λx − Ax = 0. Hence λI − A is
singular, det(λI − A) = 0.
Multiplicity of a root λ to pA = algebraic multiplicity of λ
Any A ∈ Cm×m has m eigenvalues, counted with algebraic
multiplicity
Y. Zhou Math-6316/CS-7366, SMU 172/209
 Similarity Transformations

The map A 7→ X −1 AX is a similarity transformation of A

A, B ∈ Cm×m are called similar if there is a similarity
transformation B = X −1 AX
A and X −1 AX have the same characteristic polynomials,
eigenvalues, and multiplicities:
The characteristic polynomials are the same:

pX −1 AX (z) = det(zI − X −1 AX ) = det(X −1 (zI − A)X )

= det(X −1 )det(zI − A)det(X ) = det(zI − A) = pA (z)

Therefore, the algebraic multiplicities are the same

If Eλ is eigenspace for A, then X −1 Eλ is eigenspace for X −1 AX , so
geometric multiplicities are the same

Y. Zhou Math-6316/CS-7366, SMU 173/209

 Algebraic Multiplicity ≥ Geometric Multiplicity

Let n first columns of V̂ be orthonormal basis of the eigenspace

for λ
Extend V̂ to square unitary V , and form
 
λI C
B = V ∗ AV =
0 D

Since

det(zI − B) = det(zI − λI)det(zI − D) = (z − λ)n det(zI − D)

the algebraic multiplicity of λ (as eigenvalue of B) is ≥ n

A and B are similar; so the same is true for λ of A

Y. Zhou Math-6316/CS-7366, SMU 174/209

 Defective and Diagonalizable Matrices

An eigenvalue is called defective if its algebraic multiplicity > its

geometric multiplicity
A defective matrix is any matrix with at least one defective
eigenvalue
A nondefective or diagonalizable matrix has equal algebraic and
geometric multiplicities for all eigenvalues
A is nondefective ⇐⇒
A is diagonalizable (i.e., ∃X nonsingular, s.t. A = X ΛX −1 )
(⇐=) If A = X ΛX −1 , A is similar to Λ and has the same eigenvalues
and multiplicities. But Λ is diagonal and thus nondefective.
(=⇒) Nondefective A has m linearly independent eigenvectors.
Take these as the columns of X , then A = X ΛX −1 .

Y. Zhou Math-6316/CS-7366, SMU 175/209

 Eigenvalue-Revealing Factorizations

Three common Eigenvalue-Revealing Factorizations:

Diagonalization A = X ΛX −1 (any nondefective A)
Unitary diagonalization A = QΛQ ∗ (any normal A)
Unitary triangularization (Schur factorization) A = QSQ ∗ (any A)
A few direct consequences of these decompositions:
Pm
trace (A) = trace (QSQ ∗ ) = trace (S) = j=1 λj
Qm
det (A) = det (QSQ ∗ ) = det (S) = j=1 λj
Qm
Since it is known (by SVD) that |det (A) | = j=1 σj , we get

m
Y m
Y
|λj | = σj
j=1 j=1

Y. Zhou Math-6316/CS-7366, SMU 176/209

 Eigenvalues and roots of polynomials

Well-known: Roots of a polynomial lead to eigenvalues:

Eigenvalues of A are the roots of pA (λ) = 0
Conversely: Eigenvalues lead to roots of a given polynomial.
For any given p(z) = z m + am−1 z m−1 + · · · + a1 z + a0 , it can be
shown that the roots of p are the eigenvalues of its companion
matrix
 0 −a0 
 1 0 −a1 

 1 0 −a2  
A=  . .. .. 
 1 . 

 .. 
. 0 −am−2
1 −am−1

Conclusion: Finding eigenvalues of a matrix is equivalent to

solving for roots of a polynomial

Y. Zhou Math-6316/CS-7366, SMU 177/209

 Eigenvalue Algorithms

The obvious method: Find roots of pA (λ), is ill-conditioned

Instead, compute Schur factorization A = QSQ ∗ by introducing
zeros
This can not be done in a finite number of steps. In fact
Any eigenvalue solver for A ∈ Cm×m with m ≥ 5 must be iterative
Reason: Consider a general polynomial of degree m

p(z) = z m + am−1 z m−1 + · · · + a1 z + a0

There is no closed-form expression for the roots of p: (Abel, 1842)

In general, the roots of polynomial equations higher than
fourth degree cannot be written in terms of a finite number of
operations
Schur factorization is utilized for computing all eigenvalues
Next we first look at iterative algorithms for computing only one
eigenvalue

Y. Zhou Math-6316/CS-7366, SMU 178/209

 Eigenvalue Algorithms (compute 1 eigenvalue)

The Power Iteration (Power method)

Arguably “the mother of most eigenvalue algorithms”
Reveals the “essential ratio” that determines convergence rate
The QR algorithm, as well as sparse eigen-algorithms such as
Arnoldi/Lanczos/Davidson are all variations of power method
(including its block and shift-inverse versions)
The Shift-Inverse Iteration
Essentially “power iteration”, but applied to a shift-inverse matrix
The Rayleigh-Quotient Iteration (RQI)
Essentially “power iteration”, but applied to a shift-inverse matrix,
where the shift is the current Rayleigh-quotient

x ∗ Ax
r (x) = , x ∈ Cm , x 6= 0
x ∗x

Y. Zhou Math-6316/CS-7366, SMU 179/209

 The Power Iteration

Algorithm: The simple Power Iteration

Choose v (0) = a unit length (random) vector
for k = 0, 1, 2, . . .
w = Av (k) (apply A)
λ(k) = (v (k) )∗ w (Rayleigh quotient, note v (k) 2
≡ 1)
v (k+1) = w/kwk2 (normalize)

Questions:
1. Under what condition does it converge?

Y. Zhou Math-6316/CS-7366, SMU 180/209

 The Power Iteration

Algorithm: The simple Power Iteration

Choose v (0) = a unit length (random) vector
for k = 0, 1, 2, . . .
w = Av (k) (apply A)
λ(k) = (v (k) )∗ w (Rayleigh quotient, note v (k) 2
≡ 1)
v (k+1) = w/kwk2 (normalize)

Questions:
1. Under what condition does it converge?
2. How to determine convergence?

Y. Zhou Math-6316/CS-7366, SMU 180/209

 The Power Iteration

Algorithm: The simple Power Iteration

Choose v (0) = a unit length (random) vector
for k = 0, 1, 2, . . .
w = Av (k) (apply A)
λ(k) = (v (k) )∗ w (Rayleigh quotient, note v (k) 2
≡ 1)
v (k+1) = w/kwk2 (normalize)

Questions:
1. Under what condition does it converge?
2. How to determine convergence?
Convergence may be determined from |λ(k+1) − λ(k) |, or from the
angle between v (k+1) and v (k) , or by the residual norm
Av (k) − λ(k) v (k)
3. If it converges, what does it converge to?

Y. Zhou Math-6316/CS-7366, SMU 180/209

 Convergence of Power Iteration

Assume diagonalizable. Expand initial v (0) in the eigenvector

basis {qi }, and apply Ak :

v (0) = a1 q1 + a2 q2 + · · · + am qm
v (k) = ck Ak v (0) = ck (a1 λk1 q1 + a2 λk2 q2 + · · · + am λkm qm )
= ck λk1 (a1 q1 + a2 (λ2 /λ1 )k q2 + · · · + am (λm /λ1 )k qm )

If |λ1 | > |λ2 | ≥ · · · ≥ |λm | and q1T v (0) 6= 0, then


2k
λ2
if A = A∗
k 
(k) λ2 (k)
O(
λ1 ),
kv − (±q1 )k = O( ), |λ − λ1 | = k
λ1 
O( λ2
), if A 6= A∗
λ1

Converges to the largest eigen-pair, unless eigenvector

q1 ⊥ v (0) , which is unlikely if v (0) is (uniformly/Gaussian) random
Linear convergence, factor ≈ |λ2 /λ1 | (the gap-ratio), at each
iteration
Y. Zhou Math-6316/CS-7366, SMU 181/209
 The Shift-Inverse Iteration

Power method converges to maxi |λi | only, and if gap-ratio

|λ2 /λ1 | ≈ 1− , then very slow convergence
Apply power iteration on (A − µI)−1 , (eigenvalues (λj − µ)−1 ,
converges to a λ closest to µ, with potentially much improved
gap-ratio)

Algorithm: Shift-Inverse Iteration

Choose a shift µ, and set v (0) = some unit length (random) vector
for k = 1, 2, . . .
Solve (A − µI)w = v (k−1) for w apply (A − µI)−1
v (k) = w/kwk normalize
(k) (k) ∗ (k)
λ = (v ) Av Rayleigh quotient

Converges to eigenvector qJ if the shift µ is closest to a simple λJ

(and second closest to λL 6= λJ ):
! ! (
k k̂
(k) µ − λJ (k) µ − λJ 2k if A = A∗
kv − (±qj )k = O ; |λ − λJ | = O , k̂ =
µ − λL µ − λL k if A 6= A∗
Y. Zhou Math-6316/CS-7366, SMU 182/209
 The Rayleigh-Quotient Iteration (RQI)

The shift µ is constant in shift-inverse iteration, (better

convergence if µ is updated to be closer to an eigenvalue)
Improvement: Set µ as the most current Rayleigh quotient

Algorithm: RQI
Choose v (0) = some unit length (random) vector
Compute λ(0) = (v (0) )∗ Av (0)
for k = 1, 2, . . .
Solve (A − λ(k−1) I)w = v (k−1) for w (shift-inverse)
v (k) = w/kwk (normalize)
λ(k) = (v (k) )∗ Av (k) (current Rayleigh quotient)

Convergence rate:
(locally) Square in v and λ when A is not hermitian
(locally) Cubic in v and 6th order in λ when A is hermitian
Y. Zhou Math-6316/CS-7366, SMU 183/209
 Block Power Method

Also called simultaneous iteration, or subspace iteration, or

orthogonal iteration
Can be used to compute more than 1 eigenpairs
Simultaneously apply Power method to a block of linearly
independent vectors
(0) (0) (0)
V (0) = [v1 , v2 , · · · , vn ],
(0) (0) (0)
V (k) = Ak V (0) = [Ak v1 , Ak v2 , · · · , Ak vn ]

Intrinsically ill-conditioned, since from the Power method we

(0)
know all Ak vi will converge to the dominant eigenvector
Rescue: Find an orthonormal basis of V (k) at each step of
iteration to enforce linear independence of columns

Y. Zhou Math-6316/CS-7366, SMU 184/209

 Block Power Method

Algorithm: The simple Block Power Iteration

Choose V (0) ∈ Cm×n with n orthonormal column vectors
for k = 0, 1, 2, . . .
W = AV (k) (apply A)
Λ(k) = (V (k) )∗ W (block Rayleigh quotient, for convergence test)
V (k+1) R = W (compute QR of W , orthonormalization)

Under suitable conditions, V (k) converges to an orthonormal

basis of the invariant subspace of A spanned by the first n
dominant eigenvectors
Assume |λ1 | ≥ · · · ≥ |λn | > |λn+1 | ≥ · · · ≥ |λm |, then the rate of
convergence is linear with factor |λn+1 /λn |. With an acceleration
scheme by Stewart (1976),
k
(k) λn+1
|λi − λi | = O( ), i = 1 : n.
λi
Y. Zhou Math-6316/CS-7366, SMU 185/209
 Computing all eigenvalues

The previously discussed methods compute only partial

eigenvalues, and they only require matrix-vector products, i.e., A
need not be explicitly available, only a subroutine that generates
Ax for any x is necessary (the basic requirement of many sparse
eigen algorithms)
Now we turn to eigen algorithms that compute all eigenvalues,
they are based on matrix decompositions and usually require A
to be explicitly available
Based on unitary similarity transformation
Based on QR decomposition
In essence, they are variants of (shift-inverse) power method, the
choice of shift is quite important

Y. Zhou Math-6316/CS-7366, SMU 186/209

 Schur Factorization and Diagonalization

Compute Schur factorization A = QSQ ∗ by “unitary

triangularization”:
Transforming A with similarity transformations

Qj∗ · · · Q2∗ Q1∗ A Q1 Q2 · · · Qj

| {z } | {z }
Q∗ Q

which converge to S as j → ∞
For practical reason, an eigen algorithm should converge with a
reasonably small j
For hermitian A, the sequence converges to a diagonal matrix
Since a real matrix may have complex eigenvalues (and they
always appear in conjugate pairs), the Q and S in its Schur form
can be complex.
When only real Q and S are desired, then one uses a real Schur
factorization, in which S may have 2 × 2 blocks on its diagonal.
Y. Zhou Math-6316/CS-7366, SMU 187/209
 Unitary similarity triangularization

Goal: Compute a Schur factorization A = QSQ ∗ . Can apply

Householder reflectors from left and right to introduce zeros. But
directly targeting at upper-triangular form is too ambitious
××××× XXXXX XXXXX
××××× left mult. Q ∗  0 X X X X  right mult. Q1  X X X X X 
××××× 1 
0XXXX
 XXXXX

×××××
 −→ 0XXXX −→ 
XXXXX

××××× 0XXXX XXXXX
A Q1∗ A Q1∗ AQ1

The right multiplication destroys the zeros previously introduced

We already knew similarity transformation to triangular form in
finite steps would not work (because of Abel’s theorem)
Will need iteration to reach the goal (A = QSQ ∗ )
Need two phases, so that the iterative phase can be done as
inexpensive (per iteration) as possible

Y. Zhou Math-6316/CS-7366, SMU 188/209

 Two Phases of (dense) Eigenvalues Computations

General A: First to upper-Hessenberg form, then to

upper-triangular
     
××××× ××××× ×××××
××××× ×××××  ××××
  Phase 1   Phase 2  
××××× −→  ××××
−→  ×××
×××××  ×××   ××
××××× (finite steps) ×× (iterative) ×
A 6= A∗ H S

Hermitian A: First to tridiagonal form, then to diagonal (both

because of symmetry)
     
××××× ×× ×
××××× ×××   × 
  Phase 1   Phase 2  
××××× −→  ××× 
−→  × 
×××××  ×××   × 
××××× (finite steps) ×× (iterative) ×
∗
A=A S D
Y. Zhou Math-6316/CS-7366, SMU 189/209
 First phase: To Hessenberg form

Try to introduce as many zeros in the (finite steps) first phase

Need similarity transform: An (upper) Hessenberg form is the
best possible form without destroying zeros previously introduced
First step unitary similarity transform:
   

×××××
 ××××× ×XXXX
XXXXX ×XXXX
×××××  Q∗  0 X X X X   Q1  X X X X 
××××× 1 
××××× −→  0 X X X X  −→  X X X X 
××××× 0 XXXX XXXX
A Q1∗ A Q1∗ AQ1
(zeros introduced by left-mult-Q ∗ are kept after right-mult-Q)
Continue in a similar way with column 2:
×××××  

×××××
 ××XXX
×××××
×××××  Q∗   Q ××XXX
 ×××× 2  X X X X  2  ×XXX
×××× −→ 0 XXX −→  XXX

×××× 0 XXX XXX
Q1∗ AQ1 Q2∗ Q1∗ AQ1 Q2∗ Q1∗ AQ1 Q2
Y. Zhou Math-6316/CS-7366, SMU 190/209
 First phase: To Hessenberg form

Reach the (upper) Hessenberg form in m − 2 (finite) steps:

 
×××××
×××××
∗
· · · Q2∗ Q1∗ A Q1 Q2 · · · Qm−2 = H =  ××××
 
Qm−2
| {z } | {z } ×××
Q∗ Q ××
For hermitian A, Hessenberg reduces to tridiagonal (due to
symmetry)
   

×××××
 ××××× ×X0 0 0 
××

××××× ∗ X X X X X ×XXXX ···  ×××
××××× Q1  0 X X X X  Q1 
   
XXXX  ××× 
××××× −→  0 X X X X  −→  XXXX −→
 
×××
××××× 0 XXXX XXXX ××
A Q∗A Q ∗ AQ1 T
1 1

Producing a hermitian tridiagonal matrix T after m − 2 steps

∗
Qm−2 · · · Q2∗ Q1∗ A Q1 Q2 · · · Qm−2 = T
| {z } | {z }
Q∗ Q
Y. Zhou Math-6316/CS-7366, SMU 191/209
 Reduction to Hessenberg by Householder reflectors

Algorithm: Hessenberg by Householder reflectors

for k = 1 to m − 2
x = Ak+1:m,k
vk = sign(x1 )kxk2 e1 + x
vk = vk /kvk k2
Ak+1:m,k:m = Ak+1:m,k:m − 2vk (vk∗ Ak+1:m,k:m )
A1:m,k+1:m = A1:m,k+1:m − 2(A1:m,k+1:m vk )vk∗
Matlab code:
function [ H , Q ] = hessen ( A )
[ m , n ]= size ( A ) ; H = A ;
i f ( nargout >1) , Q = eye ( n ) ; end
f o r k = 1 : n−2
u = H ( k +1: n , k ) ;
u ( 1 ) = s i g n ( u ( 1 ) ) * norm ( u ) +u ( 1 ) ; u = u / norm ( u ) ;
H ( k +1: n , k : n ) = H ( k +1: n , k : n ) − 2 * u * ( u ' * H ( k +1: n , k : n ) ) ;
H ( 1 : n , k +1: n ) = H ( 1 : n , k +1: n ) − 2 * ( H ( 1 : n , k +1: n ) * u ) * u ' ;
i f ( nargout >1) , % accumulate Q s . t . A = QHQ' ;
% f o r w a r d a c c u m u l a t i o n ( backward would use l e s s f l o p s )
Q ( 1 : n , k +1: n ) = Q ( 1 : n , k +1: n ) − 2 * ( Q ( 1 : n , k +1: n ) * u ) * u ' ;
end
end
Y. Zhou Math-6316/CS-7366, SMU 192/209
 Reduction to Hessenberg (Another implementation)

function [ u , tau ] = house_gen ( x )

% generates a householder r e f l e c t o r H = I − uu ' s t H * x = t a u * e 1 ,
% where | t a u |= norm ( x ) , ( note here norm ( u , 2 ) = s q r t ( 2 ) )
u = x; tau = norm ( x ) ; i f tau == 0 , u ( 1 ) = s q r t ( 2 ) ; r e t u r n , end
u = x / tau ;
i f u ( 1 ) >= 0 , u ( 1 ) = u ( 1 ) +1; tau = −tau ; else , u ( 1 ) = u ( 1 ) −1; end
u = u / s q r t ( abs ( u ( 1 ) ) ) ;

function [ H , Q ] = hessen2 ( A )
[ m , n ] = s i z e ( A ) ; H=A ;
Q = eye ( n ) ;
f o r k = 1 : n−2
[ Q ( k +1: n , k ) , H ( k +1 , k ) ] = house_gen ( H ( k +1: n , k ) ) ;
% p r e m u l t i p l y by ( I − uu ' ) , u = Q( k +1: n , k ) ;
H ( k +1: n , k +1: n ) =H ( k +1: n , k +1: n ) − . . .
Q ( k +1: n , k ) * ( Q ( k +1: n , k ) ' * H ( k +1: n , k +1: n ) ) ;
H ( k +2: n , k ) = zeros ( n−k −1 ,1) ;
% p o s t m u l t i p l y by ( I − uu ' )
H ( 1 : n , k +1: n ) =H ( 1 : n , k +1: n ) −(H ( 1 : n , k +1: n ) * Q ( k +1: n , k ) ) * Q ( k +1: n , k ) ' ;
end
% accumulate Q, use backward a c c u m u l a t i o n ( l e s s f l o p s )
f o r k = n−2 : −1 : 1
u = Q ( k +1: n , k ) ;
Q ( k +1: n , k +1: n ) = Q ( k +1: n , k +1: n ) − u * ( u ' * Q ( k +1: n , k +1: n ) ) ;
Q ( : , k ) = zeros ( n , 1 ) ; Q ( k , k ) =1;
end
Y. Zhou Math-6316/CS-7366, SMU 193/209
 Operation counts and stability

Operation count (not twice Householder QR):

Main operations:

Ak+1:m,k:m = Ak+1:m,k:m − 2vk (vk∗ Ak+1:m,k:m )

A1:m,k+1:m = A1:m,k+1:m − 2(A1:m,k+1:m vk )vk∗
m
X
4(m − k)2 + 4m(m − k) = 4m3 /3 +4m3 − 4m3 /2 = 10m3 /3
| {z }
k=1
QR

For hermitian A, flop count is twice QR divided by two = 4m3 /3

The Householder Hessenberg reduction algorithm is backward
stable:
kδAk
Q̃ H̃ Q̃ ∗ = A + δA, = O(ǫmachine )
kAk

where Q̃ is an exactly unitary matrix based on ṽk

Y. Zhou Math-6316/CS-7366, SMU 194/209
 Main picture of the QR algorithm

Change notation a bit, use V to denote the unitary matrix that transforms A
into H, i.e., V ∗ AV = H
i. reduce A to upper Hessenberg form: AV = VH
ii. while not convergent Do :
1. select a shift µ
2. QR factorization of the shifted H: QR = H − µI
3. update V : V ← VQ
4. update H: H ← RQ + µI (= Q ∗ HQ)

Denote V + = VQ the updated matrix with columns [v1+ , v2+ , · · · , vm+ ]:

AV = VH = V (QR + µI) ⇒ (A − µI)V = VQR

⇒ (A − µI)v1 = v1+ r11
(shifted A power iteration on the first column of V )

V ∗ (A − µI)−1 = R −1 (VQ)∗ ⇒ RV ∗ = (VQ)∗ (A − µI)

⇒ VR ∗ = (A − µI)∗ (VQ) ⇒ vm rmm ∗
= (A − µI)∗ vm+
∗
(shifted A inverse iteration on the last column of V )
Y. Zhou Math-6316/CS-7366, SMU 195/209
 Understanding the QR algorithm

A step further: If we look at a block of V (e.g., the full V ) instead of

just one single vector, then

(A − µI)V = VQR = V + R
⇒ At each iteration, QR is block power iteration with shift µ
⇒ In total, QR is subspace iteration with variable shifts

VR ∗ = (A − µI)∗ (VQ) = (A − µI)∗ V +

⇒ At each iteration, QR is inverse block power iteration with shift µ
⇒ In total, QR is inverse subspace iteration with variable shifts
(guaranteed convergence with suitably chosen shifts)

That is, QR algorithm does both subspace iteration and shift-inverse

subspace iteration on each column of V at the same time.
Y. Zhou Math-6316/CS-7366, SMU 196/209
 Second phase: From Hessenberg to Triangular

This iterative phase essentially contains two steps

QR factorization of a shifted H: QR = H − µI
Reverse multiplication of the QR factors, plus shift:
H ← RQ + µI
A is pre-processed into a Hessenberg form (V ∗ AV = H) because QR
decomposition of H is only of O(m2 ) complexity, instead of the O(m3 ) for a
general A. Can use either of two approaches to reduce H to R:
By Givens rotator (only 1 non-zero to zero out per step)
By Householder reflector of length-2 (instead of length-m) per
step . (for real A using real arithmetic, use length-3 reflectors)
The other two key properties:
Each update of H + = RQ + µI is a similarity transform of the
previous H: H + = Q ∗ HQ
Each updated H still maintains upper Hessenberg form (why?)
Y. Zhou Math-6316/CS-7366, SMU 197/209
 Choices of shifts

 a 2 × 2 diagonal block of the

At a certain iteration step, obtain a shift from
hk−1,k−1 hk−1,k
current H, say, H2 (k ) := . Usually obtain a shift from H in
hk−1,k hk,k
a bottom-to-top manner. That is, k from m downto 2.
Rayleigh-quotient shift: (mainly for hermitian matrix)
Set µ = hk,k , note that hk,k = vk∗ Avk is a readily available RQ.
Questions: Why RQ shift can fail to converge for real
nonsymmetric matrix with complex eigenvalues?
Wilkinson shift: Set µ as the eigenvalue of the 2 × 2 matrix
H2 (k) that is closer to hk,k .
Convergence rate: Quadratic for A 6= A∗ , cubic for A = A∗ .
Needs on average two QR iterations to converge an eigenvalue,
which makes the QR algorithm behave like a “direct” method.
Francis double shifts: Use (implicitly) both of the eigenvalues of
the 2 × 2 matrix H2 (k) as the double shifts. (For real A using real
arithmetic)

Y. Zhou Math-6316/CS-7366, SMU 198/209

 Choices of shifts

Need “exceptional shift” for a small set of matrices, e.g.

   
0 0 0 h 0 1 0 0
h 0 0 0  1 0 10−13 0
0 h 0 0 ,
   
0 −10−13 0 1
0 0 h 0 0 0 1 0

Y. Zhou Math-6316/CS-7366, SMU 199/209

 Deflation of converged eigenvalues in QR algorithm

Mainly utilize the “shift-inverse power” property of the QR

algorithm for fast convergence:
Recall that QR algorithm performs “shift-inverse” iteration on the
last column of V
With Wilkinson shift, the convergence rate is at least quadratic,
and the last column in V typically converges first
Therefore, deflate converged columns in V from the last column
to the first
That is, check convergence in H from bottom up. Typically, the
last subdiagonal elements in H decreases to 0 fastest

Y. Zhou Math-6316/CS-7366, SMU 200/209

 A very simplified sample code of QR algorithm

f u n c t i o n [ H , V ] = qrschur ( A , tol ) ;
% compute A=VHV' , where H converges t o upper t r i a n g u l a r

[ m , n ] = s i z e ( A ) ; H = zeros ( n , n ) ;
i f ( n a r g o u t > 2 ) , [ H , V ] = hessen2 ( A ) ; else , [ H ] = hessen ( A ) ; end
k = n ; it = 1 ; itmax = n ˆ 2 ;
w h i l e ( k > 1 & it <=itmax )
% compute t h e W i l k i n s o n s h i f t
mu = e i g ( H ( k−1:k , k−1:k ) ) ;
i f abs ( mu ( 1 )−H ( k , k ) )<=abs ( mu ( 2 )−H ( k , k ) ) , mu = mu ( 1 ) ;
else , mu = mu ( 2 ) ; end

% compute QR ( should use Givens o r l e n g t h −2 Householder i n s t e a d ,

% should use i m p l i c i t s h i f t i n s t e a d o f e x p l i c i t s h i f t )
[ Q , R ] = q r ( H ( 1 : k , 1 : k ) − mu * eye ( k ) ) ;
H ( : , 1 : k) = H ( : , 1 : k) *Q ; H(1:k , : ) = Q ' * H(1:k , : ) ;

i f ( nargout > 2) , V ( : , 1 : k ) = V ( : , 1 : k ) * Q ; end %update V

% d e f l a t e i f a s u b d i a g o n a l i s s m a l l enough
i f abs ( H ( k , k−1) ) < tol * ( abs ( H ( k−1,k−1) ) +abs ( H ( k , k ) ) ) ,
H ( k , k−1) = 0 ; k = k−1;
end
it = it + 1 ;
end

Y. Zhou Math-6316/CS-7366, SMU 201/209

 A few demos of the convergence of the QR algorithm

Click the following links for some online demos

A symmetric, with Wilkinson shift 2

3
A symmetric, with Rayleigh quotient shift

A nonsymmetric, with Wilkinson shift 4

2 http://faculty.smu.edu/yzhou/Teach/demo/sym_wilks.gif
3 http://faculty.smu.edu/yzhou/Teach/demo/sym_RQshifts.gif
4 http://faculty.smu.edu/yzhou/Teach/demo/nonsym_wilks.gif
Y. Zhou Math-6316/CS-7366, SMU 202/209
 Quite a few details left out

When A ∈ Rm×m , do not want to use complex arithmetic, instead,

using real arithmetic, perform implicit double shifts to compute
“real” Schur form.
For the iterative 2nd phase, exploit the “implicit Q theorem” to get
the QR decomposition of the shifted matrix (either H − µI or a
double shifted H 2 − sH + tI) without using explicit shift
Using Givens rotator or length-2/3 Householder reflectors for the
iterative process to go from Hessenberg to triangular
Details in Golub and Van Loan’s “matrix computations”,
or, J. Demmel’s “Applied Numerical Linear Algebra”,
or, G. W. Stewart’s “Matrix algorithms, Vol 2”.

Y. Zhou Math-6316/CS-7366, SMU 203/209

 The implicit Q theorem

Theorem: Given A ∈ Cm×m . Let U and V be two unitary matrices,

with U ∗ AU = H, V ∗ AV = G, where H, G are of unreduced upper
Hessenberg form. If u1 = v1 , then ui = ci vi with |ci | = 1, and
|hij | = |gij |.
In words, if A is unitarily transformed into unreduced Hessenberg form by
similarity transformation, and the first columns of the unitary matrices are
identical, then the remaining columns are identical upto a complex sign.

Proof: Quite straightforward noting that uk only depends on

u1 , · · · uk−1 in AU = UH when H is unreduced upper Hessenberg.
Comparing columns in AU = UH and AV = VG, it becomes apparent
that u1 = v1 is enough to guarantee that ui is parallel to vi for all i.

Y. Zhou Math-6316/CS-7366, SMU 204/209

 Other Eigen Algorithms 1: Jacobi iteration

 
cos θ sin θ
Jacobi rotator J = , looks very much like a
− sin θ cos θ
Givens rotator, but with an intrinsic difference:
The need to keep a similarity transformation.
E.g., Diagonalize a 2 × 2 real symmetric matrix using J
   
T a d X 0 2d
J J= =⇒ tan(2θ) =
d b 0 X b−a

Iteratively apply transformation to 2 rows and 2 columns of

A ∈ Rm×m
Loop over all pairs of rows/columns, quadratic convergence
O(m2 ) steps, O(m) operations per step =⇒ O(m3 ) operation
count

Y. Zhou Math-6316/CS-7366, SMU 205/209

 Other Eigen Algorithms 2: Divide-and-Conquer

Assume T is symmetric tridiagonal, split T into submatrices:

T1 T̂1
β β β
T= β
= + β β
T2 T̂2

The sum of a 2 × 2 block-diagonal matrix and a rank-one

correction
Split T and compute eigenvalues of T̂1 , T̂2 recursively
Assume diagonalizations T̂1 = Q1 D1 Q1T and T̂2 = Q2 D2 Q2T have
been computed, then
     T 
Q1 D1 Q1
T = + βzz T
Q2 D2 Q2T
with z T = (q1T , q2T ), where q1T is last row of Q1 and q2T is first row
of Q2
Y. Zhou Math-6316/CS-7366, SMU 206/209
 Secular equation of Divide-and-Conquer
 
D1
Eigenvalues of T are the eigenvalues of + βzz T
D2
Solve a (nonlinear) secular equation to get eigenvalues of T from
those of T̂1 , T̂2
In general, eigenvalues of D + ww T are the roots of the secular
equation
m
X wj2
f (λ) := 1 + = 0, where w T = [w1 , w2 , · · · , wm ]
dj − λ
j=1

d1 d2 d3 d4 λ

Y. Zhou Math-6316/CS-7366, SMU 207/209

 Cost of Divide-and-Conquer

Solve the secular equation f (λ) = 0 with a nonlinear solver, such

as Newton’s method on each interval (di , di+1 )
Very fast convergence, typically O(m) flops per root, O(m2 ) flops
for all roots
Total cost for divide-and-conquer algorithm (for computing
eigenvalues only):
 
m2 m2 m2 m2
O m2 + 2 2 + 4 2 + 8 2 + · · · + m 2 = O(m2 )
2 4 8 m

Most of the operations are spent in reducing A into the

tridiagonal T , and the constant in “Phase 2” is not important
However, for computing eigenvectors, divide-and-conquer
reduces Phase 2 to 4m3 /3 flops compared to 6m3 for the QR
algorithm 5
5 Stable algorithm for computing eigenvectors within DC developed one decade later

since the 1st DC algorithm was proposed

Y. Zhou Math-6316/CS-7366, SMU 208/209
 Computing SVD

Two phases for dense SVD

Phases 1: (direct finite steps)
Unitary bi-diagonalization
A → Ũ ∗ AṼ = B where B is bi-diagonal
Phases 2: (iterative) Iterate from bi-diagonal to diagonal.
Essentially performing QR algorithm on the tridiagonal Hermitian
B ∗ B, but without forming B ∗ B explicitly
Most of the important details of computing SVD can be found in
these matlab files:
Phase 1 bi-diagonalization by Householder reflectors (bidiag.m)
Phase 2 iteration to diagonal form (svdbiqr.m) ,
this code calls the implicit shifted QR using Given rotators (biqr.m)
Although mostly coded from scratch, (for small/medium sized
matrices) these codes compute SVD with comparable performance
to the Matlab built-in function svd which calls Lapack

Y. Zhou Math-6316/CS-7366, SMU 209/209