The Fundamental Theorem of Linear Algebra

Author(s): Gilbert Strang

Source: The American Mathematical Monthly , Nov., 1993, Vol. 100, No. 9 (Nov., 1993),
pp. 848-855
Published by: Taylor & Francis, Ltd. on behalf of the Mathematical Association of
America

Stable URL: https://www.jstor.org/stable/2324660

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

Taylor & Francis, Ltd. and Mathematical Association of America are collaborating with JSTOR
to digitize, preserve and extend access to The American Mathematical Monthly

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 The Fundamental Theorem of Linear
Algebra

Gilbert Strang

This paper is about a theorem and the pictures that go with it. The theorem
describes the action of an m by n matrix. The matrix A produces a linear
transformation from Rn to Rm-but this picture by itself is too large. The "truth"
about Ax = b is expressed in terms of four subspaces (two of Rn and two of Rm).
The pictures aim to illustrate the action of A on those subspaces, in a way that
students won't forget.
The first step is to see Ax as a combination of the columns of A. Until then the
multiplication Ax is just numbers. This step raises the viewpoint to subspaces. We
see Ax in the column space. Solving Ax = b means finding all combinations of the
columns that produce b in the column space:

[ l . 1 [ ] | j| = x1(column 1) + +xj(column n) = b.
Columns 'of A x J

The column space is the range R(A), a subspace of Rm. This abstraction, from
entries in A or x or b to the picture based on subspaces, is absolutely essential.
Note how subspaces enter for a purpose. We could invent vector spaces and
construct bases at random. That misses the purpose. Virtually all algorithms and
all applications of linear algebra are understood by moving to subspaces.
The key algorithm is elimination. Multiples of rows are subtracted from other
rows (and rows are exchanged). There is no change in the row space. This subspace
contains all combinations of the rows of A, which are the columns of AT. The row
space of A is the column space R(AT).
The other subspace of Rn is the nullspace N(A). It contains all solutions to
Ax = 0. Those solutions are not changed by elimination, whose purpose is to
compute them. A by-product of elimination is to display the dimensions of these
subspaces, which is the first part of the theorem.
The Fundamental Theorem of Linear Algebra has as many as four parts. Its
presentation often stops with Part 1, but the reader is urged to include Part 2.
(That is the only part we will prove-it is too valuable to miss. This is also as far as
we go in teaching.) The last two parts, at the end of this paper, sharpen the first
two. The complete picture shows the action of A on the four subspaces with the
right bases. Those bases come from the singular value decomposition.
The Fundamental Theorem begins with
Part 1. The dimensions of the subspaces.
Part 2. The orthogonality of the subspaces.

848 THE FUNDAMENTAL THEOREM OF LINEAR AGEBRA [November

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
The dimensions obey the most important laws of linear algebra:

dim R(A) = dim R(AT) and dim R(A) + dim N(A) = n.

When the row space has dimension r, the nullspace has dimension n - r.
Elimination identifies r pivot variables and n - r free variables. Those variables
correspond, in the echelon form, to columns with pivots and columns without
pivots. They give the dimension count r and n - r. Students see this for the
echelon matrix and believe it for A.
The orthogonality of those spaces is also essential, and very easy. Every x in the
nullspace is perpendicular to every row of the matrix, exactly because Ax = 0:

-row 1 0
Ax= -row 2- x= O
-row m 0

The first zero is the dot product of x with row 1. The last zero is the dot product
with row m. One at a time, the rows are perpendicular to any x in the nullspace.
So x is perpendicular to all combinations of the rows.

The nullspace N(A) is orthogonal to the row space R(AT).

What is the fourth subspace? If the matrix A leads to R(A) and N(A), then its
transpose must lead to R(AT) and N(AT). The fourth subspace is N(AT), the
nullspace of AT. We need it! The theory of linear algebra is bound up in the
connections between row spaces and column spaces. If R(AT) is orthogonal to
N(A), then-just by transposing-the column space R(A) is orthogonal to the
"left nullspace" N(AT). Look at ATy = 0:

column 1 of A 0
ATy = [ Y][ .]
column n of A 0

Since y is orthogonal to each column (producing each zero), y is orthogonal to the

whole column space. The point is that AT is just as good a matrix as A. Nothing is
new, except AT is n by m. Therefore the left nullspace has dimension m - r.
ATy = 0 means the same as yTA = OT. With the vector on the left, yTA is a
combination of the rows of A. Contrast that with Ax = combination of the
columns.

The First Picture: Linear Equations

Figure 1 shows how A takes x into the column space. The nullspace goes to the
zero vector. Nothing goes elsewhere in the left nullspace-which is waiting its
turn.
With b in the column space, Ax = b can be solved. There is a particular
solution xr in the row space. The homogeneous solutions xn form the nullsp
The general solution is xr + xn. The particularity of xr is that it is orthogo
every xn.
May I add a personal note about this figure? Many readers of Linear Algebra
and Its Applications [4] have seen it as fundamental. It captures so much about
Ax = b. Some letters suggested other ways to draw the orthogonal subspaces-
artistically this is the hardest part. The four subspaces (and very possibly the fig
itself) are of course not original. But as a key to the teaching of linear algebra, t
illustration is a gold mine.

1993] THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA 849

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 Figure 1. The action of A: Row space to column space, nulispace to zero.

Other writers made a further suggestion. They proposed a lower level textbook,
recognizing that the range of students who need linear algebra (and the variety of
preparation) is enormous. That new book contains Figures 1 and 2-also Figure 0,
to show the dimensions first. The explanation is much more gradual than in this
paper-but every course has to study subspaces! We should teach the important
ones.
The Second Figure: Least Squares Equations

If b is not in the column space, Ax = b cannot be solved. In practice we still

have to come up with a "solution." It is extremely common to have more equations
than unknowns-more output data than input controls, more measurements than
parameters to describe them. The data may lie close to a straight line b = C + Dt.
A parabola C + Dt + Et2 would come closer. Whether we use polynomials or
sines and cosines or exponentials, the problem is still linear in the coefficients
C, D, E:

C +Dt=bX CA + Dt+Et=bb
*'or
C+ Dtm = bm C+Dtm+Et X= bm

There are n = 2 or n = 3 unknowns, and m is larger. There is no x = (C, D) or

x = (C, D, E) that satisfies all m equations. Ax = b has a solution only when the
points lie exactly on a line or a parabola-then b is in the column space of the m
by 2 or m by 3 matrix A.
The solution is to make the error b - Ax as small as possible. Since Ax can
never leave the column space, choose the closest point to b in that subspace. This
point is the projection p. Then the error vector e = b - p has minimal length.
To repeat: The best combination p = AX is the projection of b onto the column
space. The error e is perpendicular to that subspace. Therefore e = b - AX is in
the left nullspace:

AT(bI Ac,) =O or AT4 b=ATb.

Calculus reaches the same linear equations by minimizing the quadratic
The chain rule just multiplies both sides of Ax = b by AT.

850 THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA [November

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 The "normal equations" are ATAx = ATb. They illustrate what is almost invari-
ably true-applications that start with a rectangular A end up computing with the
square symmetric matrix ATA. This matrix is invertible provided A has indepen-
dent columns. We make that assumption: The nullspace of A contains only x = 0.
(Then ATAx = 0 implies xTATAx = 0 which implies Ax = 0 which forces x = 0, so
ATA is invertible.) The picture for least squares shows the action over on the right
side-the splitting of b into p + e.

rw olumn

notpossible ~>b-p+

let e

Figure 2. Least squares: x minimiz

The Third Figure: Orthogonal Bases

Up to this point, nothing was said about bases for the four subspaces. Those
bases can be constructed from an echelon form-the output from elimination.
This construction is simple, but the bases are not perfect. A really good choice, in
fact a "canonical choice" that is close to unique, would achieve much more. To
complete the Fundamental Theorem, we make two requirements:

Part 3. The basis vectors are orthonormal.

Part 4. The matrix with respect to these bases is diagonal.

If v1, ..., vr is the basis for the row space and u1, . .. , ur is the basis for the
column space, then Avi = ojui. That gives a diagonal matrix E. We can further
ensure that i > 0.
Orthonormal bases are no problem-the Gram-Schmidt process is available.
But a diagonal form involves eigenvalues. In this case they are the eigenvalues of
ATA and AAT. Those matrices are symmetric and positive semidefinite, so they
have nonnegative eigenvalues and orthonormal eigenvectors (which are the bases!).
Starting from ATAvi = oi 2v , here are the key steps:

V[AAV~=
ViTATAVi ov[i soS that
= UJ2ViTVi IIAvill
htlAil-v = u

AATAvi = oj2Avj so that ui = Avi/ui is a unit eigenvector of AAT.

All these matrices have rank r. The r positive eigenvalues oi2 give the diagonal
entries vi of E.

1993] THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA 851

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 The whole construction is called the singular value decomposition (SVD). It
amounts to a factorization of the original matrix A into UIYVT, where

1. U is an m by m orthogonal matrix. Its columns u1, ... . Ur, -. , um are basis

vectors for the column space and left nullspace.
2. X is an m by n diagonal matrix. Its nonzero entries are a-1 > 0,..., or > 0.
3. V is an n by n orthogonal matrix. Its columns v1, ... ., v,.. ,vn are basis
vectors for the row space and nullspace.

The equations Avi = oui mean that AV = Ul. Then multiplication by VT

gives A = UYVT.
When A itself is symmetric, its eigenvectors ui make it diagonal: A = UAUT.
The singular value decomposition extends this spectral theorem to matrices that
are not symmetric and not square. The eigenvalues are in A, the singular values
are in E. The factorization A = UIVT joins A = LU (elimination) and A = QR
(orthogonalization) as a beautifully direct statement of a central theorem in linear
algebra.
The history of the SVD is cloudy, beginning with Beltrami and Jordan in the
1870's, but its importance is clear. For a very quick history and proof, and much
more about its uses, please see [1]. "The most recurring theme in the book is the
practical and theoretical value of this matrix decomposition." The SVD in linear
algebra corresponds to the Cartan decomposition in Lie theory [3]. This is one
more case, if further convincing is necessary, in which mathematics gets the
properties right-and the applications follow.

Example

A 1 * 2] = [3 l] 5& 0 [-2 l] =U>,

All four subspaces are 1-dimensional. The columns of A are multiples of [3] in U.
The rows are multiples of [1 2] in VT. Both ATA and AAT have eigenvalues 50
and 0. So the only singular value is a-, = 5.

Vl~~~~~~~? =Z a,. / N
A.+ I .il{)

Figure 3. Orthonormal bases that diagonalize A.

852 THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA [November

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 The SVD expresses A as a combination of r rank-one matrices:

A = UYVT = UiT+ +UOTVrT (here A- [[1 2])

The Fourth Figure: The Pseudoinverse

The SVD leads directly to the "pseudoinverse" of A. This is needed, just as the
least squares solution x was needed, to invert A and solve Ax = b when those
steps are strictly speaking impossible. The pseudoinverse A+ agrees, with A1
when A is invertible. The least squares solution of minimum length (having no
nullspace component) is x + =A +b. It coincides with X when A has full column
rank r = n-then ATA is invertible and Figure 4 becomes Figure 2.
A + takes the column space back to the row space [4]. On these spaces of equal
dimension r, the matrix A is invertible and A + inverts it. On the left nullspace,
A+ is zero. I hope you will feel, after looking at Figure 4, that this is the one
natural best definition of an inverse. Despite those good adjectives, the SVD and
A+ is too much for an introductory linear algebra course. It belongs in a second
course. Still the picture with the four subspaces is absolutely intuitive.

/rw \ /olumn \
/ space \A+ p - x + 5\space

X 7 +

nnullspace spa

Figure 4. The inverse of A (where possible) is the pseudoinverse A +.

The SVD gives an easy formula for A +, because it chooses the right bases. Since
Avi = ului, the inverse has to be A+ui = vi/oi. Thus the pseudoinverse of l
contains the reciprocals 1/1oi. The orthogonal matrices U and VT are inverted by
UT and V. All together, the pseudoinverse of A = UYIVT isA+= VI + UT.

Example (continued)

A+= -[2 l] [1/ ?] [o -3 l] = 1 3]

d5 O O 4/ 50 2 6.
Always A+A is the identity matrix on the row space, and zero on the nullspace:

A +A= [ 320
10 20 =projection
40T
onto the[2L
line through i.

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
Similarly AA + is the identity on the column space, and zero on the left nullspace:

AA+ = 1 = projection onto the line through [1].

A Summary of the Key Ideas

From its r-dimensional row space to its r-dimensional column space, A yields
an invertible linear transformation.

Proof: Suppose x and x' are in the row space, and Ax equals Ax' in the column
space. Then x - x' is in both the row space and nullspace. It is perpendicular to
itself. Therefore x = x' and the transformation is one-to-one.

The SVID chooses good bases for those subspaces. Compare with the Jordan form
for a real square matrix. There we are choosing the same basis for both domain
and range-our hands are tied. The best we can do is SAS-' = J or SA = JS. In
general J is not real. If real, then in general it is not diagonal. If diagonal, then in
general S is not orthogonal. By choosing two bases, not one, every matrix does as
well as a symmetric matrix. The bases are orthonormal and A is diagonalized.
Some applications permit two bases and others don't. For powers A' we need
S` to cancel S. Only a similarity is allowed (one basis). In a differential equation
u' = Au, we can make one change of variable u = Sv. Then v' = S'-ASv. But for
Ax = b, the domain and range are philosophically "not the same space." The row
and column spaces are isomorphic, but their bases can be different. And for least
squares the SVD is perfect.
This figure by Tom Hern and Cliff Long [2] shows the diagonalization of A.
Basis vectors go to -basis vectors (principal axes). A circle goes to an ellipse. The
matrix is factored into UIV'. Behind the scenes are two symmetric matrices ATA
and AAT. So we reach two orthogonal matrices U and V.

VT u'2e2 U

Vp U 2 e2 -IU

V1 4

We close by summarizing the action of A and AT and A':

Avi = oiui ATui = o,vi A +ui = vi/oi 1 < i < r.

The nullspaces go to zero. Linearity does the rest.

854 THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA [November

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms
 The support of the National Science Foundation (DMS 90-06220) is gratefully
acknowledged.

REFERENCES

1. Gene Golub and Charles Van Loan, Matrix Computations, 2nd ed., Johns Hopkins University Press
(1989).
2. Thomas Hern and Cliff Long, Viewing some concepts and applications in linear algebra, Visualiza-
tion in Teaching and Learning Mathematics, MAA Notes 19 (1991) 173-190.
3. Roger Howe, Very basic Lie theory, American Mathematical Monthly, 90 (1983) 600-623.
4. Gilbert Strang, Linear Algebra and Its Applications, 3rd ed., Harcourt Brace Jovanovich (1988).
5. Gilbert Strang, Introduction to Linear Algebra, Wellesley-Cambridge Press (1993).

Department of Mathematics
Massachusetts Institute of Technology
Cambridge, A 02139
[email protected]

An Identity of Daubechies
The generalization of an identity of
Daubechies using a probabilistic interpre-
tation by D. Zeilberger [100 (1993) 487],
has already appeared in SIAM Review
Problem 85-10 (June, 1985) in a slightly
more general context. In addition to a
similar probabilistic derivation there is
also a direct algebraic proof. Incidentally,
problem 10223 [99 (1992) 462] is the same
as the identity of Daubechies and a slight
generalization of this identity has ap-
peared previously as problem 183, Crux
Math. 3(1977) 69-70 and came from a list
of problems considered for the Canadian
Mathematical Olympiad. There was an
inductive solution of the latter by Mark
Kleinman, a high school student at the
time and one of the top students in the
U.S.A.M.O. and the I.M.O.
M. S. Klamkin
Department of Mathematics
University of Alberta
Edmonton, Alberta
CANADA T6G 2G1

1993] THE FUNDAMENTAL THEOREM OF LINEAR ALGEBRA 855

This content downloaded from

193.136.24.86 on Sun, 24 Mar 2024 13:58:54 +00:00
All use subject to https://about.jstor.org/terms