1

System of Equations
CVG Lecture 11

Ata Babakhani

uOttawa.ca
 2

Agenda
• wooclap
• Error Analysis and System Condition
- Singular Systems
- Ill-conditioned Systems
• Iterative Methods
- Jacobi’s Method
- Gauss Seidel's Method
• Review

uOttawa.ca
 3

uOttawa.ca
 4

wooclap

uOttawa.ca
 5

Assignment, Quiz, Lab

• Assignment 3 will be posted on Brightspace, covering the System of Equations

• Quiz 2 is scheduled for Friday at 9:00 AM

• Attendance will be taken at 9:00 AM

• Quiz 2 questions cover nonlinear equations

• Assignment 1 solution has been posted on BS

• Check your Quiz 1 and Assignment 1 grades

• Kamesh: [email protected]

• Fatemeh: [email protected]

uOttawa.ca
 6

Quiz 2

• There are 7 questions, all of which are related to nonlinear equations.

• Students will have 45 minutes to complete the quiz.

• Students may consult their books or other reference material.

• Students should use the provided space to answer the questions.

• Students must upload a file (such as Microsoft Excel) containing their

calculations in detail for the computational questions.

• Conduct all calculations with at least four decimal places.

uOttawa.ca
 7

Error Analysis and System Condition

Cases that cause problems in a system of linear equations:
1. Singular system (no solution or infinite solution)
2. Ill-conditioned system
Example 1. (Singular system with no solution)
𝑥1 + 2𝑥2 = 10 𝑥1
2 𝑥1 + 4𝑥2 = 5 𝑥2 =

Example 2. (Singular system with infinite solution)

𝑥1 + 2𝑥2 = 10
?
𝑥1
2 𝑥1 + 4𝑥2 = 20 𝑥2 =
𝐴 𝑥 = 𝑏 The determinant of the coefficients matrix is 0
uOttawa.ca
 8

Error Analysis and System Condition

Cases that cause problems in a system of linear equations:
1. Singular system (no solution or infinite solution)
2. Ill-conditioned system
An ill-conditioned system is one where small changes in the
numerical values of coefficient matric [A] or right-side vector [b]
cause large changes in the solution vector {x}.
Example 3. (Ill-conditioned system)
𝑥1 + 2𝑥2 = 10 𝑥1 8
𝑥 = Round-off error induces small
1.05 𝑥1 + 2𝑥2 = 10.4 2 1
changes in the coefficients, but
𝑥1 + 2𝑥2 = 10 𝑥1 4 leads to large solution error
1.1 𝑥1 + 2𝑥2 = 10.4 𝑥2 = 3
uOttawa.ca
 9

Review

The determinant of the This problem may occur depending

on the method used (Direct)
coefficients matrix is 0
This problem may occur
uOttawa.ca
regardless of the methodused
used (Direct/Iterative)
 10

Review Matrices
How to use the inverse matrix method to solve a system of linear equations:

a11 a12a12 ... ...a1na1n x1 x1  b1b1 

a11
aa a a ... ...a a  x  x  b b 
 21 22 22
 21      = ==2 2 
2n 2n 2 2

×
 ... ... ... ... ... ... ... ...   ...   ...... 
 ...
aa a a ... ...a a  x  x  b b 
 n1 n1 n 2 n 2 nn nn  n  n   n n 

𝐴𝑥Ԧ = 𝑏 𝑥Ԧ = 𝐴−1 𝑏

[A] must be Non-Singular and its inverse is accessible

uOttawa.ca
 11

Disadvantages of Direct Methods

• Sometimes, we do not really need to solve the system exactly.
Direct methods cannot accomplish this because the process
must be completed to obtain a solution.
• Sometimes, we have a pretty good idea of an approximate
guess for the solution. Direct methods cannot make good use
of such information.
• Because of the large number of calculations, direct approaches
are time-consuming.
• If you have an ill-conditioned system, using the direct methods
may give you an answer with a large error.

Motivation for Iterative Methods

uOttawa.ca
 12

System of equations

Direct Methods Iterative Methods

LU Decomposition
Gauss-Seidel’s
Gauss Elimination (Cholesky Jacobi’s Method
Method
Thomas)

System of Linear Equations System of Linear and some Nonlinear Equations

uOttawa.ca
 13

Iterative Methods

a11 x1 + a12 x2 +...+ a1n xn = b1

a21 x1 + a22 x2 +...+ a2 n xn = b2
...
an1 x1 + an 2 x2 +...+ ann xn = bn

Have an initial with : x1(0) , x2(0) ,..., xn(0)

Start guess: → x1(1) , x2(1) ,..., xn(1) → ...x1( p) , x2( p) ,..., xn( p )

uOttawa.ca
 14

Iterative Methods Steps

We need to ensure that the method converges the answer
a11 x1 + a12 x2 +...+ a1n xn = b1
a21 x1 + a22 x2 +...+ a2 n xn = b2
...
an1 x1 + an 2 x2 +...+ ann xn = bn

In each row, the absolute value of the entry in the main diagonal must
be greater than the sum (in absolute value) of other entries in that row.
This criterion is sufficient but not necessary for convergence
condition for convergence:
n
aii   aij for i = 1, 2, 3,..., n (*)
j =1
j i

 n

max  1
or  aij   1
uOttawa.ca i  aii j =1

 i j 
 15

Iterative Methods Steps

Rearrange the equations and isolate xi in equation i

b1 a12 a a
x1 = − x2 −...− 1k x k −...− 1n xn
a11 x1 + a12 x2 +...+ a1n xn = b1 a11 a11 a11 a11

a21 x1 + a22 x2 +...+ a2 n xn = b2 bk a k 1 a a a
xk = − x1 −...− kk −1 x k −1 − kk +1 x k +1 ...− kn xn
ak 2 ak 2 a kk a kk a kk
...

an1 x1 + an 2 x2 +...+ ann xn = bn xn =
bn an1
−
a a
x1 − n 2 x2 −...− nn −1 xn −1
ann ann ann ann

A simple way to obtain initial guesses is to assume that they are all zero

uOttawa.ca
 16

System of equations

Direct Methods Iterative Methods

LU Decomposition
Gauss-Seidel’s
Gauss Elimination (Cholesky Jacobi’s Method
Method
Thomas)

System of Linear Equations System of Linear and some Nonlinear Equations

uOttawa.ca
 17

Jacobi’s Method-Example

8 x1 + x2 − x3 = 8 8 1 −1
2 x1 + x2 + 9 x3 = 12 A= 2 1 9
x1 − 7 x2 + 2 x3 = −4 1 −7 2

𝑖 = 1 → ( 1 + −1 ) = 2 < 8 (𝑂𝐾)
𝑖 = 2 → 2 + 9 = 11 > 1 (NOT 𝑂𝐾)
𝑖 = 3 → 1 + −7 = 8 > 2 (NOT 𝑂𝐾)

Therefore, the iterative method may not converge in this form.

uOttawa.ca
 18

Jacobi’s Method-Example (cont’d)

8 1 −1 8 1 −1
A= 2 1 9 A= 2 1 9
1 −7 2 1 −7 2

8 1 −1
A= 1 −7 2
2 1 9

𝑖 = 1 → ( 1 + −1 ) = 2 < 8 (𝑂𝐾)
𝑖=2→ 1 + 2 =3<7 (𝑂𝐾)
𝑖=3→ 2 + 1 =3<9 (𝑂𝐾)

uOttawa.ca
 19

Jacobi’s Method-Example (cont’d)

8 x1 + x2 − x3 = 8 8 x1 + x2 − x3 = 8 2/8 <1
2 x1 + x2 + 9 x3 = 12 x1 − 7 x2 + 2 x3 = −4 3/ 7 < 1
x1 − 7 x2 + 2 x3 = −4 2 x1 + x2 + 9 x3 = 12 3/ 9 < 1

x2 x3
x1 = 1 − +
8 8
4 x1 2
x2 = + + x3
7 7 7
12 2 x1 x2
x3 = − −
9 9 9
uOttawa.ca
 20

Jacobi’s Method-Example (cont’d)

x2 x3
x1 = 1 − +
8 8
4 x1 2
x2 = + + x3
7 7 7
12 2 x1 x2
x3 = − −
9 9 9
Iteration 0 1 2 3 4 5
X1 0 1 1.095238 0.994048 0.99263 1.001027
X2 0 0.571429 1.095238 1.027211 0.990079 0.998461
X3 0 1.333333 1.047619 0.968254 0.998299 1.00274

uOttawa.ca
 21

System of equations

Direct Methods Iterative Methods

Gauss LU Gauss-Seidel’s
Jacobi’s Method
Elimination Decomposition Method

System of Linear Equations System of Linear and some Nonlinear Equations

uOttawa.ca
 22

Gauss-Seidel’s Method-Example
We can introduce a simple improvement to Jacobi’s method by making use of
the latest estimates of 𝑥𝑖 in estimating the other values

8 x1 + x2 − x3 = 8
2 x1 + x2 + 9 x3 = 12
x1 − 7 x2 + 2 x3 = −4

x2 x3
x1 = 1 − +
8 x1 + x2 − x3 = 8 2 / 8 <81 8
4 x1 2
x1 − 7 x2 + 2 x3 = −4 x2 = 7+< 1 + x3
3 /
7 7 7
2 x1 + x2 + 9 x3 = 12 3/ 9 < 1
12 2 x1 x2
uOttawa.ca x3 = − −
9 9 9
 23

Gauss-Seidel’s Method-Example (cont’d)

x2 x3
x1 = 1 − +
8 8
4 x1 2
x2 = + + x3
7 7 7
12 2 x1 x2
x3 = − −
9 9 9

Iteration 0 1 2 3 4 5
X1 0 1 1.039683 0.996851 1.000565 0.99993
X2 0 0.714286 1.014739 0.996563 1.00039 0.999942
X3 0 1.031746 0.989544 1.001082 0.999831 1.000022

uOttawa.ca
 24

8 x1 + x2 − x3 = 8 2/8 <1
x1 − 7 x2 + 2 x3 = −4 Gauss-Seidel
3/ 7 < 1 vs Jacobi
2 x1 + x2 + 9 x3 = 12 3/ 9 < 1
Iteration 0 1 2 3 4 5
X1 0 1 1.039683 0.996851 1.000565 0.99993
X2 0 0.714286 1.014739 0.996563 1.00039 0.999942
X3 0 1.031746 0.989544 1.001082 0.999831 1.000022

Iteration 0 1 2 3 4 5
X1 0 1 1.095238 0.994048 0.99263 1.001027
X2 0 0.571429 1.095238 1.027211 0.990079 0.998461
X3 0 1.333333 1.047619 0.968254 0.998299 1.00274
Relative True Error (%) True Answer
Jacobi Gauss-Seidel x
1
>
0.1027 0.007
0.1539 0.0058
1
0.274 0.0022
1
Faster convergence
uOttawa.ca
 25

Conclusion

Mathematical Solution of a System of Equations

Iterative Methods Direct Methods

Truncation Error Round-off Error Round-off Error

Accuracy Precision Precision

uOttawa.ca
 26

Review

a11 x1 + a12 x2 +...+ a1n xn = b1

a21 x1 + a22 x2 +...+ a2 n xn = b2
...
an1 x1 + an 2 x2 +...+ ann xn = bn

Ax = b

uOttawa.ca
 27

Review

𝐴 𝑥Ԧ = 𝑆 𝑆 𝑇 𝑥Ԧ = 𝑏

uOttawa.ca
 28

Review

condition for convergence:

n
aii   aij for i = 1, 2, 3,..., n (*)
j =1
j i

 n

max  1  1
or
i  aii
a
j =1
ij

 i j 

uOttawa.ca