MA206L-Numerical Methods
Lecture 20: Numerical Linear Algebra - LU Decomposition
Panchatcharam Mariappan1
1 Associate Professor
Department of Mathematics and Statistics
IIT Tirupati, Tirupati
October 7, 2024
Linear System
• One of the important problem in many science and engineering.
• To solve an algebraic linear system
Ax = b
for the unknown vector x.
• The coefficient matrix A and the right-hand side vector b are known.
• What is the size of A, x and b?
• Assume: A is n × n, x and b are n−dimensional vector.
• The system may or may not have a solution.
• It may have an infinite solutions or unique solution.
1
Linear System
• Gaussian elimination method: one of the standard method for solving
the linear system
• Computer or calculator
• School days.
• However, in pure mathematics, the solution is given by
x = A−1 b
where A−1 denotes the inverse of the matrix A.
• But, in most of the real applications, it is not advised to find A−1
explicitly, indeed, it is recommended to solve for x.
2
Linear System
• But, in most of the real applications, it is not advised to find A−1
explicitly, indeed, it is recommended to solve for x.
• In applied mathematics, the largest and fastest computers can also face
difficulty to solve the system accurately when the number of unknowns
are in millions.
3
Linear System: Important 10 Questions
1. How do we store such a large linear system in the computer?
2. How do we know that the computed answers are correct?
3. How does the computer precision affects the results?
4. Can the algorithm fail?
5. How long will it take to compute the answers?
6. What is the asymptotic operation count of the algorithm?
7. Will the algorithm be stable for perturbation?
8. Can stability be controlled by pivoting?
9. Which strategy of pivoting should be used?
10. How do we know whether the matrix is ill-conditioned?
4
Diagonal Linear System
Definition 1 (Diagonal linear system)
A matrix A of order n is diagonal if all its nonzero entries are on its diagonal. A
diagonal linear system of order n is one whose coefficient matrix is diagonal.
That is (
dii ̸= 0 if i = j
Aij =
0 if i ̸= j
A diagonal linear system can be represented as
d11 0 0 ··· 0 x1 b1
0 d22 0 · · · 0 x2 b2
0
0 d33 · · · 0 x3 b3
=
.. .. .. . . .
. . .
. . . . . .. ..
0 0 0 · · · dnn xn bn
5
Diagonal Linear System
Finding the solution of the linear system is easy because each equation
determines the value of one unknown, provided that each diagonal entry is
nonzero. The solution is given by
bi
xi = , i = 1, 2, · · · , n
dii
Example 1
Consider the following linear system and find the solution.
9x1 = 3, 25x2 = 5, , 16x3 = 4, 36x4 = 6, 49x5 = 7
6
Diagonal Linear System
It is a diagonal linear system and it can be written as
9 0 0 0 0 x1 3
0 25 0 0 0 x2 5
0 0 16 0 0 x3 = 4
0 0 0 36 0 x4 6
0 0 0 0 49 x5 7
Without constructing this matrix also we can immediately conclude that
1
x1 3
x2 1
51
x3 =
41
x4
6
1
x5 7
7
Upper Triangular Linear System
Definition 2 (Upper triangular linear system)
A matrix A of order n is upper triangular if all its nonzero entries are on its
diagonal or strictly on the upper triangular entries. An upper triangular linear
system of order n is one whose coefficient matrix is upper triangular. That is
(
uij if i ≥ j
Aij =
0 if i < j
An upper triangular linear system can be represented as
u11 u12 u13 · · · u1n x1 y1
0 u22 u23 · · · u2n x2 y2
0
0 u33 · · · u3n x3 = y3
.. .. .. .. .
.. . .
.. ..
. . . .
0 0 0 · · · unn xn yn 8
Backward Substitution
In order to solve an upper triangular linear system, we need a backward
substitution which is given by the following algorithm
9
Backward Substitution
In order to solve an upper triangular linear system, we need a backward
substitution which is given by the following algorithm
n
X
yi − uij xj
j=i+1
xi = , i = n, n − 1, · · · , 1 (1)
uii
9
Backward Substitution
Example 2
Find the solution of the following linear system.
2 2 3 4 x1 20
0 5 6 7 x2 34
0 0 8 9 x3 = 25
0 0 0 10 x4 10
Backward Substitution
Example 2
Find the solution of the following linear system.
2 2 3 4 x1 20
0 5 6 7 x2 34
0 0 8 9 x3 = 25
0 0 0 10 x4 10
It is an upper triangular system. Using backward substitution, we obtain that
10 25 − 9x4 16
x4 = = 1 =⇒ x3 = =⇒ x3 = =2
10 8 8
34 − 6x3 − 7x4 15
=⇒ x2 = =⇒ x2 = =3
5 5
20 − 2x2 − 3x3 − 4x4 4
=⇒ x1 = =⇒ x1 = = 2
2 2
10
Forward Substitution
Definition 3 (Lower triangular linear system)
A matrix A of order n is upper triangular if all its nonzero entries are on its
diagonal or strictly on the lower triangular entries. A lower triangular linear
system of order n is one whose coefficient matrix is lower triangular. That is
(
lij if i ≤ j
Aij =
0 if i > j
A lower triangular linear system can be represented as
l11 0 0 ··· 0 y1 b1
l21 l22 0 · · · 0 y2 b2
l31 l32 u33 · · · 0 y3 b3
=
.. .. .. . . .. .. ..
. . . . . . .
ln1 ln2 ln3 · · · lnn yn bn 11
Forward Substitution
In order to solve a lower triangular linear system, one can apply forward
substitution. In the backward substitution, we obtain the value of xn first and
then using xn , we obtain the value of xn−1 Using xn , xn−1 , xn−2 is calculated.
Finally the value of x1 is calculated. On the other hand, in forward
substitution, we obtain the value of y1 first. Using y1 , we obtain the value of y2
and so on. The value of
12
Forward Substitution
In order to solve a lower triangular linear system, one can apply forward
substitution. In the backward substitution, we obtain the value of xn first and
then using xn , we obtain the value of xn−1 Using xn , xn−1 , xn−2 is calculated.
Finally the value of x1 is calculated. On the other hand, in forward
substitution, we obtain the value of y1 first. Using y1 , we obtain the value of y2
and so on. The value of
i−1
X
bi − lij xj
j=1
yi = , i = 1, 2, · · · , n (2)
lii
12
LU Decomposition
13
LU Decomposition
If we can decompose a square matrix A by in the form of
A = LU
then one can easily obtain the solution of the linear system
Ax = b
Ax = b =⇒ LU x = b =⇒ Ly = b where U x = y
Since Ly = b is a lower triangular system, by the forward substitution, we can
solve for y. Using y in U x = y, which is an upper triangular system, and the
backward substitution, we can obtain x.
14
LU Decomposition
The following steps are used to solve the linear system Ax = b.
1. Find L and U such that A = LU
2. Using forward substitution, Solve for y from Ly = b
3. Use y in U x = y
4. Using backward substitution, Solve for x from U x = y
As per the LU decomposition, we can write A as
a11 a12 a13 · · · a1n l11 0 0 ··· 0 u11 u12 u13 · · · u1n
a21 a22 a23 · · · a2n l21 l22 0 · · · 0 0 u22 u23 · · · u2n
a31 a32 a33 · · · a3n l31 l32 l33 · · · 0 0 0 u33 · · · u3n
=
.. .. .. .. .. .. .. .. . . .. .. .. .. .. ..
. . . . . . . . . . . . . . .
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 · · · unn
15
LU Decomposition
Example 3
Obtain the LU decomposition and then obtain the solution x.
6 −2 2 4 x1 16
12 −8 6 10 x2 26
=
3 −13 9 3 x3 −19
−6 4 1 −18 x4 −34
6 −2 2 4 l11 0 0 0 u11 u12 u13 u14
12 −8 6 10 l21 l22 0 0 0 u22 u23
u24
=
3 −13 9 3 l31 l32 l33 0 0 0 u33 u34
−6 4 1 −18 l41 l42 l43 l44 0 0 0 u44
As per LU Decomposition
16
LU Decomposition
Comparing the first entry of LU with A, we obtain that l11 u11 = 6. Now, we can
choose any real number l11 such that l11 u11 = 6. For the sake of convenience,
we always choose lii = aii , then we obtain that l11 = 6 and u11 = 1.
l11 u11 = 6, l11 = 6 =⇒ u11 = 1
−1
l11 u12 = −2 =⇒ u12 =
3
1
l11 u13 = 2 =⇒ u13 =
3
2
l11 u14 = 4 =⇒ u14 =
3
Since u11 is known, we can easily compute the first column of L by the
following formula.
ai1
li1 = , i>1
u11 17
LU Decomposition
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
12 −8 6 10 12 l22 0 0 0 u22 u23 u24
=
3 −13 9 3 3 l32 l33 0 0 0 u33 u34
−6 4 1 −18 −6 l42 l43 l44 0 0 0 u44
Therefore, the formula to compute u2j is given by
a2j − l21 u1j
u2j = , j>2
l22
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai2 − li1 u12
li2 = , i>2
u22 18
LU Decomposition
Similarly the formula to compute u2j is given by
a3j − l31 u1j − l32 u2j
u3j = ,j > 3
l33
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai3 − li1 u13 − li2 u23
li3 = , i>3
u33
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
12 −8 6 10 12 −8 0 0 0 1/2 −1/4 −1/4
=
3 −13 9 3 3 −24 9 0 0 0 2/9 −5/9
−6 4 1 −18 −6 4 18 −18 0 0 0 1/6
19
LU Decomposition
Now, let us consider the system Ly = b
6 0 0 0 y1 16
12 −8 0 0 y2 = 26
3 −24 9 0 y3 −19
−6 4 18 −18 y4 −34
Using forward substitution, we obtain that
16 8 8 3
y1 = = =⇒ 12 − 8y2 = 26 =⇒ y2 =
6 3 3 4
8 3
=⇒ 3 − 24 + 9y3 = −19 =⇒ y3 = −1
3 4
8 3 1
=⇒ −6 +4 + 18(1) − 18y4 = −34 =⇒ y4 =
3 4 6
20
LU Decomposition
Now, let us consider the system Ly = b
y1 8/3
y2 3/4
=
y3 −1
y4 1/6
Now, consider the system U x = y
1 −1/3 1/3 2/3 x1 8/3
0 1/2 −1/4 −1/4 x2 3/4
=
0 0 2/9 −5/9 x3 −1
0 0 0 1/6 x4 1/6
21
LU Decomposition
Using backward substitution, we obtain that
1/6 2 5
x4 = = 1 =⇒ x3 − = −1 =⇒ x3 = −2
1/6 9 9
1 −1 −1 3
=⇒ x2 + (−2) + (1) = =⇒ x2 = 1
2 4 4 4
1 1 2 8
=⇒ x1 − −2 + = =⇒ x1 = 3
3 3 3 3
22
LU Decomposition
For a general LU decomposition, we have first guess lkk ̸= 0 value or ukk ̸= 0
values and use the following formula for j > k and i > k
k−1
X
akj − lks usj
s=1
ukj = (3)
lkk
k−1
X
aik − lis usk
s=1
lik = (4)
ukk
From the above two equations, it is easy to note that lkk ̸= 0 and ukk ̸= 0 are
necessary.
23
LU Decomposition
Theorem 4
If all n leading principal minors of A are nonsingular, then A has an LU-
decomposition.
Definition 5 (Diagonally Dominant)
A square matrix A = (aij ) is said to be diagonally dominant if it satisfies the
following inequality:
n
X
|aii | > |aij | 1≤i≤n
j=1,j̸=i
Theorem 6
Every diagonally dominant matrix is non-singular and has an LU-
decomposition. 24
Crout, Doolittle, LDLT ,
Cholesky
25
Crout’s Decomposition
In LU-decomposition one of the condition specified is that lkk ̸= 0 and ukk ̸= 0
are necessary. If we choose, ukk = 1, then U is an upper triangular matrix with
its diagonal entries as 1. This decomposition is called Crout’s decomposition.
a11 a12 a13 · · · a1n l11 0 0 ··· 0 1 u12 u13 · · · u1n
a21 a22 a23 · · · a2n l21 u22 0 · · · 0 0 1 u23 · · · u2n
a31 a32 a33 · · · a3n l31 l32 u33 · · · 0 0 0 1 · · · u3n
=
.. .. .. . . .. .. .. .. . . .. .. .. .. . . ..
. . . . . . . . . . . . . . .
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 ··· 1
26
Doolittle’s Decomposition
If we choose, lkk = 1, then L is a lower triangular matrix with its diagonal
entries as 1. This decomposition is called Doolittle’s decomposition.
a11 a12 a13 · · · a1n 1 0 0 ··· 0 u11 u12 u13 · · · u1n
a21 a22 a23 · · · a2n l21 1 0 · · · 0 0 u22 u23
· · · u2n
a31 a32 a33 · · · a3n l31 l32 1 · · · 0 0 0 u33 · · · u3n
=
.. .. .. . .. .
.. ..
. .
.. .
.. . . .
. . .. ..
.. .. .. ..
. . . . . . .
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · 1 0 0 0 · · · unn
27
LDLT Decomposition
Observe that when Doolittle’s decomposition is applied, the first row of U and
A are same. Similarly, when Crout’s decomposition is applied, then the first
column of L and A are same. In fact we can prove uniqueness of LU
decomposition in case of Doolittle and Crout assumptions as follows:
Theorem 7
Every nonsingular matrix has a unique LU decomposition.
Suppose
A = L1 U1 = L2 U2 =⇒ L−1 −1
1 L2 = U1 U2
Since left side is lower triangular and right side is upper triangular, it should
be a diagonal matrix. Since diag(L−1 1 L2 ) = I (for Doolittle) or
−1
diag(U1 U2 ) = I (for Crout), it leads to L1 = L2 and U1 = U2 . Using
Gauss-Jordan method it can be obtained easily.
28
LDLT Decomposition
If A is a symmetric matrix and A has an LU decomposition, then it has an
LDLT decomposition. For,
A = LU =⇒ AT = U T LT
A = AT =⇒ LU = U T LT
Since L is unit lower triangular, it is invertible, we can write
U = L−1 U T LT =⇒ U (LT )−1 = L−1 U T
Transpose of an upper triangular matrix is a lower triangular and vice versa.
Since L is lower triangular, L−1 is also a lower triangular and hence L−1 U T is
a lower triangular matrix. On the other side, we have U (LT )−1 is an upper
triangular matrix. Therefore,
U (LT )−1 = L−1 U T = D =⇒ U = DLT =⇒ A = LDLT
29
Cholesky Decomposition
When A is symmetric and positive definite and U = LT , then we obtain that
A = LLT
Theorem 8
If A is real, symmetric and positive definite, then it has a unique factorization
A = LLT
in which L is lower triangular with positive diagonal.
Since A > 0, A = AT by LDLT decomposition, we can obtain
A = LDLT
A > 0 =⇒ D > 0 =⇒ L̃ ≡ LD1/2 =⇒ A = L̃L̃T
√
where the entries of D1/2 are dii . 30
Thanks
Doubts and Suggestions
[email protected] 31
MA206L-Numerical Methods
Lecture 20: Numerical Linear Algebra - LU Decomposition
Panchatcharam Mariappan1
1 Associate Professor
Department of Mathematics and Statistics
IIT Tirupati, Tirupati
October 7, 2024
31
