MA423 Matrix Computations
Lecture 10: Vector and matrix norms
Rafikul Alam
Department of Mathematics
IIT Guwahati
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Outline
• Vector norms
• Matrix norms
• Unitarily invariant norms
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Vector norms
Let V be a vector space over C. Then a function k · k : V → R is called a
norm on V if it satisfies the three fundamental properties:
(a) Positive definiteness: kv k ≥ 0 and kv k = 0 ⇐⇒ v = 0.
(b) Positively homogeneous: kαv k = |α| kv k for α ∈ C and v ∈ V.
(c) Triangle inequality: ku + v k ≤ kuk + kv k for u, v ∈ V.
Example: Consider Cn and the vector norms given by
1-norm: kxk1 := |x1 | + · · · + |xn |.
p
2-norm: kxk2 := |x1 |2 + · · · |xn |2 .
∞-norm: kxk∞ := max1≤j≤n |xj |.
More generally, for 1 ≤ p < ∞, the Höder p-norm is given by
kxkp := (|x1 |p + · · · + |xn |p )1/p .
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Matrix norms
Let A ∈ Cm×n . Then A : Cn −→ Cm , x 7−→ Ax, is a linear map. Suppose
Cn and Cm are equipped with norms. Then
kAxk
kAk := max = max kAxk
x6=0 kxk kxk=1
defines a norm on Cm×n and is called an induced matrix norm or a
subordinate matrix norm.
For the identity matrix kIxk = kxk and hence kI k = 1. Note that
kAxk ≤ kAk kxk
for all x ∈ Cn .
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Matrix norms
The matrix norm of A induced by the Hölder p-norm is denoted by kAkp .
Then kAk1 , kAk2 and kAk∞ are called 1-norm, 2-norm and ∞-norm of
A, respectively. Also kAk2 is called the spectral norm of A.
Theorem: Let A be an m × n matrix. Then
kAk1 = max kAej k1 = max kA(:, j)k1
1≤j≤n 1≤j≤n
p
kAk2 = λmax (A∗ A)
kAk∞ = max kei> Ak1 = max kA(i, :)k1 ,
1≤i≤m 1≤i≤m
where λmax (A∗ A) denotes that largest eigenvalue of A∗ A.
Proof: We have
Ax = x1 Ae1 + · · · + xn Aen ⇒ kAxk1 ≤ max1≤j≤n kAej k1 kxk1 . This yields
kAk1 ≤ max1≤j≤n kAej k1 . But kAej k1 ≤ kAk1 for all j = 1 : n. Hence we
have kAk1 = max1≤j≤n kAej k1 .
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Spectral norm
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Frobenius norm
The spectral norm of a matrix is difficult to compute. By contrast,
Frobenius norm is easy to compute and provides a nice geometric
framework for solving problems. Let A be an m-by-n matrix.
Then the Frobenius norm is given by
1/2
Xm X
n p p
kAkF := |A(i, j)|2 = Tr(A∗ A) = Tr(AA∗ ),
i=1 j=1
where Tr(A∗ A) is the trace of A∗ A.
Pm 1/2 1/2
P
> 2 n
Note that kAkF = i=1 kei Ak2 kAej k22
= . Also note
j=1
√
that for an n-by-n identity matrix I , we have kI kF = n which shows
that the Frobenius norm is not an induced matrix norm.
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Example
Let D := diag(λ1 , . . . , λn ) be a diagonal matrix. Then we q
have
Pn 2
kAk2 = kAk1 = kAk∞ = max(|λ1 |, . . . , |λn |) and kDkF = j=1 |λj | .
1000 999 −998 999
Consider A := . Then A−1 = .
999 998 999 −1000
Thus kAk∞ = kAk1 = kA−1 k∞ = kA−1 k1 = 1999.
On the other hand,
kAk2 = 1998.000500500375 and kAkF = 1998.000500500438.
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Submultiplicative matrix norm
A matrix norm is said to be submultiplicative if
kABk ≤ kAk kBk
holds for all A and B. An induced matrix norm is submultiplicative.
Indeed, kABxk ≤ kAk kBxk ≤ kAk kBk kxk =⇒ kABk ≤ kAk kBk.
The Frobebnius norm is also submultiplicative. Indeed, suppose A is
m-by-n and B is n-by-p matrices. Then (AB)ij = ei> ABej shows that
|(AB)ij | ≤ kei> Ak2 kBej k2 for i = 1 : m and j = 1 : n. Consequently,
X p
m X p
m X
X
kABk2F = |(AB)ij |2 ≤ kei> Ak22 kBej k22
i=1 j=1 i=1 j=1
m
X p
X
= kei> Ak22 kBej k22 = kAk2F kBk2F .
i=1 j=1
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Unitarily invariant matrix norm
Let Q ∈ Rn×n . Then Q is said to be orthogonal if Q > Q = QQ > = I . If
Q is orthogonal
p then p p
kQxk2 = hQx, Qxi = hx, Q > Qxi = hx, xi = kxk2 .
Let Q ∈ Cn×n . Then Qpis said to be unitary
p if Q ∗ Q =pQQ ∗ = I . If Q is
unitary then kQxk2 = hQx, Qxi = hx, Q ∗ Qxi = hx, xi = kxk2 .
It follows that kQk2 = 1 when Q is unitary and that error in x is NOT
magnified in the vector Qx.
A matrix norm is said to be unitariliy invariant if kQAPk = kAk holds for
any unitary matrices Q and P. The spectral norm and the Frobenius
norm are unitarily invariant norms. Indeed,
kQxk2 = kxk2 ⇒ kQAPk2 = kAk2 .
p p
kQAPkF = Tr((QAP)∗ QAP) = Tr(P ∗ A∗ AP)
p
= Tr(A∗ A) = kAkF .
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
