BITS Pilani
Hyderabad Campus
Lecture -9:
SYSTEM of LINEAR ALGEBRAIC EQUATIONS
Direct Methods: Gauss Elimination
1
Solving Systems of Equations
• A linear equation in n variables:
a1x1 + a2x2 + … + anxn = b
• For small (n ≤ 3), linear algebra provides several tools to solve
such systems of linear equations:
– Graphical method
– Cramer’s rule
– Method of elimination
• Nowadays, easy access to computers makes the solution of
very large sets of linear algebraic equations possible
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-2
Graphical Method
Graphical solution of set of two
simultaneous algebraic equations.
Interaction of lines represents the
solution
What about graphical representation
of system of three simultaneous
algebraic equations ?
Solution is the point of intersect of
three planes.
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-3
Graphical Method
Graphical Depiction of singular and ill-
conditioned systems
a) No solution
b) Infinite solutions
c) Ill-conditioned system (slopes are
very close )
a)
c) b)
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-4
Determinants and Cramer’s Rule
Ax b [A] : coefficient matrix
a11 a12 a13 a11 a12 a13 x1 b1
a a a x b
A a21 a22 a23 21 22 23 2 2
a31 a32 a33 a31 a32 a33 x3 b3
b1 a12 a13 a11 b1 a13 a11 a12 b1
b2 a22 a23 a21 b2 a23 a21 a22 b2
b3 a32 a33 a31 b3 a33 a31 a32 b3
x1 x2 x3
D D D
D : Determinant of A matrix
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-5
Computing the Determinant
a11 a12 a13
D a21 a22 a23
a31 a32 a33
Ax B D11
a22 a23
a32 a33
a22 a33 a32 a23
a21 a23
a11 a12 a13 D12 a21 a33 a31 a23
A a21 a22 a23 a31 a33
a21 a22
a31 a32 a33 D13 a21 a32 a31 a22
a31 a32
a22 a23 a21 a23 a21 a22
Determinant of A D a11 a12 a13
a32 a33 a31 a33 a31 a32
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-6
Gauss Elimination
a11 a12 a13 b1
• Solve Ax = b a b2
21 a22 a23
a31 a32 a33 b3
• Consists of two phases: Forward
–Forward elimination Elimination
–Back substitution a11 a12 a13 b1
0 a' '
a23 b2'
22
• Forward Elimination 0 0 ''
a33 b3''
reduces Ax = b to an upper
triangular system Tx = b’
b3'' b2' a23
'
x3
• Back substitution can then x3 '' x2 ' Back
a33 a22
solve Tx = b’ for x Substitution
b1 a13 x3 a12 x2
x1
a11
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-7
Forward Elimination
x1 - x2 + x3 = 6 x1 - x2 + x3 = 6
-(3/1) 3x1 + 4x2 + 2x3 = 9 0 +7x2 - x3 = -9
-(2/1) 2x1 + x2 + x3 = 7 -(3/7) 0 + 3x2 - x3 = -5
x1 - x2 + x3 = 6
0 7x2 - x3 = -9
0 0 -(4/7)x3=-(8/7)
Solve using BACK SUBSTITUTION: x3 = 2 x2=-1 x1 =3
0
0 0
0 0 0
0 0 0 0
0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-8
Back Substitution
1x0 +1x1 –1x2 +4x3 = 8
– 2x1 –3x2 +1x3 = 5
2x2 – 3x3 = 0
x3 = 2 2x3 = 4
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-9
Back Substitution
1x0 +1x1 –1x2 = 0
– 2x1 –3x2 = 3
x2 = 3 2x2 = 6
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-10
Back Substitution
1x0 +1x1 = 3
x1 = –6 – 2x1 = 12
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-11
Back Substitution
1x0 +1x1 = 3
x1 = –6 – 2x1 = 12
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-12
x0 = 9 1x0 = 9
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-13
Forward Elimination
a ji
a ji a ji aii 0
aii
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-14
Forward Elimination
M
U
L
T
I
P 4x0 +6x1 +2x2 – 2x3 = 8
L
I
E
R
S
-(2/4) 2x0 +5x2 – 2x3 = 4
-(-4/4) –4x0 – 3x1 – 5x2 +4x3 = 1
-(8/4) 8x0 +18x1 – 2x2 +3x3 = 40
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-15
Forward Elimination
M
U
L
T
I 4x0 +6x1 +2x2 – 2x3 = 8
P
L
I
E
R – 3x1 +4x2 – 1x3 = 0
S
-(3/-3) +3x1 – 3x2 +2x3 = 9
-(6/-3) +6x1 – 6x2 +7x3 = 24
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-16
Forward Elimination
4x0 +6x1 +2x2 – 2x3 = 8
M
U
L
T
– 3x1 +4x2 – 1x3 = 0
I
P
L
I 1x2 +1x3 = 9
E
R
(2/1) 2x2 +5x3 = 24
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-17
Forward Elimination
4x0 +6x1 +2x2 – 2x3 = 8
– 3x1 +4x2 – 1x3 = 0
1x2 +1x3 = 9
3x3 = 6
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-18
Operation count in
Forward Elimination
b11
2n 0 b22
2n 0 0 b33
2n 0 0 0 b44
2n 0 0 0 0 b55
2n 0 0 0 0 0 b66
2n 0 0 0 0 0 0 b77
2n 0 0 0 0 0 0 0 b66
2n 0 0 0 0 0 0 0 0 b66
TOTAL
1st column: 2n(n-1) 2n2 2(n-1)2 2(n-2)2 …….
TOTAL # of Operations for FORWARD ELIM INATION :
n
2n 2(n 1) ... 2 * (2) 2 * (1) 2 i 2
2 2 2 2
i 1
n(n 1)(2n 1)
2
6
O(n 3 )
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-19
Pitfalls of Elimination Methods
Division by zero
It is possible that during both elimination and back-substitution phases a
division by zero can occur.
For example:
2x2 + 3x3 = 8 0 2 3
4x1 + 6x2 + 7x3 = -3 A= 4 6 7
2x1 + x2 + 6x3 = 5 2 1 6
Solution: pivoting (to be discussed later)
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-20
Pitfalls (cont.)
Round-off errors
• Because computers carry only a limited number of significant figures,
round-off errors will occur and they will propagate from one iteration to the
next.
• This problem is especially important when large numbers of equations (100
or more) are to be solved.
• Always use double-precision numbers/arithmetic. It is slow but needed for
correctness!
• It is also a good idea to substitute your results back into the original
equations and check whether a substantial error has occurred.
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-21
Pitfalls (cont.)
ill-conditioned systems - small changes in coefficients result in large
changes in the solution. Alternatively, a wide range of answers can
approximately satisfy the equations.
(Well-conditioned systems – small changes in coefficients result in small
changes in the solution)
Problem: Since round off errors can induce small changes in the coefficients, these
changes can lead to large solution errors in ill-conditioned systems.
Example: b1 a12 10 2
x1 + 2x2 = 10
b2 a22 10.4 2 2(10) 2(10.4)
1.1x1 + 2x2 = 10.4 x1 4 x2 3
D 1(2) 2(1.1) 0.2
x1 + 2x2 = 10 b1 a12 10 2
1.05x1 + 2x2 = 10.4 b2 a22 10.4 2 2(10) 2(10.4)
x1 8 x2 1
D 1(2) 2(1.05) 0.1
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-22
ill-conditioned systems (cont.) –
• Surprisingly, substitution of the erroneous values, x1=8 and x2=1, into the original
equation will not reveal their incorrect nature clearly:
x1 + 2x2 = 10 8+2(1) = 10 (the same!)
1.1x1 + 2x2 = 10.4 1.1(8)+2(1)=10.8 (close!)
IMPORTANT OBSERVATION:
An ill-conditioned system is one with a determinant close to zero
• If determinant D=0 then there are infinitely many solutions singular system
• Scaling (multiplying the coefficients with the same value) does not change the
equations but changes the value of the determinant in a significant way.
However, it does not change the ill-conditioned state of the equations!
DANGER! It may hide the fact that the system is ill-conditioned!!
How can we find out whether a system is ill-conditioned or not?
Not easy! Luckily, most engineering systems yield well-conditioned results!
• One way to find out: change the coefficients slightly and recompute & compare
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-23
Techniques for Improving Solutions
• Use of more significant figures – double precision arithmetic
• Pivoting
If a pivot element is zero, normalization step leads to division by zero. The
same problem may arise, when the pivot element is close to zero. Problem
can be avoided:
– Pivoting
Switching the rows below so that the largest element is the pivot element.
– Partial pivoting
• Searching for the largest element in all rows and columns then switching.
• This is rarely used because switching columns changes the order of x’s
and adds significant complexity and overhead costly
• Scaling
– used to reduce the round-off errors and improve accuracy
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-24
Gauss-Jordan Elimination
a11 0 0 0 0
0 0 0 0 b11
0 x22 0 0 0 0 0 0 0 b22
x33 0 0 0 0 0 0 b33
0 0
0 0 0 x44 0 0 0 0 0 b44
x55 0 0 0 0
0 0 0 0 b55
0 0 0 0 0 x66 0 0 0 b66
0 0 0 0 0 0 x77 0 0 b77
0 0 0 0 0 0 0 x88 0
b66
0 0 0 0 0 0 0 0 x99 b66
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-25
Gauss-Jordan Elimination: Example
1 1 2 x1 8 1 1 2| 8
1 2 3 x 1 Augmented Matrix : 1 2 3 | 1
2
3 7 4 x3 10 3 7 4 |10
1 1 2 | 8 Scaling R2: 1 1 2 | 8
R2 R2 - (-1)R1 0 1 5 | 9 R2 R2/(-1) 0 1 5| 9
R3 R3 - ( 3)R1 0 4 2| 14 0 4 2| 14
R1 R1 - (1)R2 1 0 7 | 17 1 0 7 | 17
0 1 5 | 9 Scaling R3: 0 1 5 | 9
R3 R3-(4)R2 0 0 18 | 22 R3 R3/(18) 0 0 1 |11 / 9
R1 R1 - (7)R3 1 0 0 | 8.444 RESULT:
0 1 0 | 2.888
R2 R2-(-5)R3
0 0 1 | 1.222 x1=8.45, x2=-2.89, x3=1.23
Time Complexity?CHE F242 O(n3for
Numerical Methods ) Chemical Engineers, BITS, Pilani-Hyderabad Campus P-26
