Linear Algebra

Ch 2. Solving Linear Systems

Chia-Yen Lee Ph.D. (李家岩 博士)

Department of Information Management (資訊管理學系)

National Taiwan University (國立台灣大學)
1
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Credit for the materials goes to 盧信銘 教授 and 王之彥 教授

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 2
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 3
Preliminaries

 The goal of this lecture:

⚫ Develop systematic procedure for solving linear systems that
works directly with the matrix form
⚫ Establish conditions for a solution to a linear system to exist
⚫ Establish conditions for a unique solution to a linear system
⚫ Re-examine solutions to homogeneous systems
⚫ Develop a systematic procedure for computing A–1
⚫ Discover equivalent conditions to nonsingularity

 Will also examine a special technique for solving linear systems:

LU decomposition

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 4
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)
Key Terms:
Row echelon form 階梯矩陣
Reduced row echelon form 簡化階梯矩陣

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 5
Echelon Form of a Matrix
 Solve a system using elimination of variables
x1 + x2 + 2x3 = −1 x1 + x2 + 2x3 = −1
x1 − 2x2 + x3 = −5  0 − 3x2 − x3 = −4 
3x1 + x2 + x3 = 3 0 − 2x2 − 5x3 = 6

x1 + x2 + 2x 3 = −1 x1 + x2 + 2x 3 = −1
0 + 3x2 + x3 = 40 + 6x2 + 2x 3 = 8
0 + 2x2 + 5x 3 = −6 0 + 6x2 + 15x 3 = −18

x1 + x2 + 2x3 = −1 x1 + x2 + 2x3 = −1
0 + 6x2 + 2x3 = 80 + 6x2 + 2x3 = 8
0 + 0 + 13x3 = −26 0 + 0 + x3 = −2

 Continuing the process gives x1 = 1, x2 = 2 and x3 = -2

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 6
Echelon Form of a Matrix

 Consider the linear system in matrix form

1 1 2  x1   −1
    
1 −2 1  x2  =  −5
3
 1 1  x3   3
 To find the solution, transformed it into:
1 1 2   x1   −1  1 0 0  x1   1 
    
0 1 1 3  x2  = 4 3  0 1 0  x2  =  2
0
 0 1   x3   −2  0 0 1   x   −2
  3  
Row echelon form Reduced row echelon form
Row echelon form 階梯矩陣 Reduced row echelon form 簡化階梯矩陣

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen

7 Lee, Ph.D.
Echelon Form of a Matrix

 We will use the following terminologies.

Pivot Columns
 1 1 2 −1  1 1 2 −1  1 1 2 −1
 1 −2 1 −5  0 −3 −1 −4   0 −3 −1 −4 
     
3 1 1 3 0 −2 −5 3 0 0 −13 3 26 3

Pivots (主元/軸/樞)

1 1 2 −1  1 0 0 1 
  0 1 1 3 4 3   0 1 0 2 
   
0 0 1 −2  0 0 1 −2 

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 8
Echelon Form of a Matrix
 Definition: - An m x n matrix A is said to be in reduced row echelon form
if it has the following properties
a) If any rows consist of entirely zeros, they are at the bottom of the
matrix (全0的row在最下面)
b) If a row does not consist entirely of zeros, the first nonzero entry in
that row is 1 (不是全0的row，第一個非零元素是1)
c) If rows i and i + 1 are two successive rows which do not consist
entirely of zeros, then the first nonzero entry of row i + 1 is to the right
of the first nonzero entry of row i (上下兩個相鄰的row, 下面那個
row的第一個非零元素在上面那個row的第一個非零元素的右邊)
d) If a column contains the first nonzero entry of some row, then all the
other entries in that column are zero (如果一個column有某個row的
第一個非零元素，則這個column的其他元素為零)

 Definition: - If an m x n matrix A satisfies properties (a), (b) and (c), it is

said to be in row echelon form

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 9
Echelon Form of a Matrix

Examples - Row Echelon Form

1 5 0 2 −2 4 
  1 0 0 0
0 1 0 3 4 8  
0 0 0 1 7 −2 0 1 0 0
  0 0 1 0
0 0 0 0 0 0  
0 0 0 0 0 0  0 0 0 1

0 0 1 3 5 7 9
 
0 0 0 0 1 −2 3
0 0 0 0 0 1 2
 
0 0 0 0 0 0 1
0 0 0 0 0 0 0 


Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 10
Echelon Form of a Matrix

Examples - Reduced Row Echelon Form

1 0 0 0 1 0 0 0 −2 4 


0 1 0 
0 0 1 0 0 4 8 
0 0 1 0 0 0 0 1 7 −2
   
0 0 0 1 0 0 0 0 0 0
0 0 0 0 0 0 


1 2 0 0 1
 
0 0 1 2 3
0 0 0 0 0


Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 11
Your Turn
 True or false: the following matrices are in reduced row
echelon form
 (1) (2)
1 2 0 4 1 0 3 4
   
0 0 0 0  0 2 − 2 5 
0 0 1 −3 0 0 1 2 
   
 (3) (4) 
0 0 0 1 0 0 1 2 3 4
 
0 1 −2 5
0 0 0 0 1 0
0 0 0 0 0 1 0 0 1 2
0 0 0 0 0 0  
0 0 0 0

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 12
Echelon Form of a Matrix

Comments - Row Echelon Form

 Can define column echelon form and reduced column
echelon form in a similar manner. Basically, they are just
the transpose of the corresponding row forms

 Close connection between echelon forms and the solution

of linear equations

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 13
Elementary Row Operation (基本列運算)

 Definition: An elementary row operation on a matrix A is any one

of the following operations
a) Interchange rows i and j of matrix A
b) Multiply row i of A by c ≠ 0
c) Add c times row i of A to row j of A, i ≠ j
 The three row operations are connect to
a) Interchange two equations
b) Multiply a given equation by a nonzero constant
c) Add a constant times one equation to another equation
 We know that the solution to a system of linear equations is
unchanged if we do any of the following

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 14
Elementary Row Operation

 Comment
⚫ The first elementary row operation, exchange two rows, is needed to deal with zeros in
the pivot positions. For example, consider

2 x2 = 1 0 2  x1  1  0 2 1 
   =    
3x1 + 4 x2 = 2 3 4  x2   2 3 4 2

⚫ No multiple of the first row will remove the 3 in the second row. The solution with
equations is just to exchange the equations. The solution with matrices is just to
exchange the rows.

3x1 + 4 x2 = 2 3 4  x1   2 3 4 2
   =    
2 x2 = 1 0 2  x2  1  0 2 1 
⚫ This problem can also occur as an intermediate step in solving a larger system and has
the same fix

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 15
Row Equivalence

 Definition: An m x n matrix P is row equivalent to an m x n

matrix Q if P can be obtained by applying a finite sequence
of elementary row operations to Q.

 The basic method for solving the system AX = B is to apply

elementary row operations to the augmented matrix [ A:B ]
to get a new augmented matrix [ C:D ] such that the CX = D
is easy to solve
⚫ If [ C:D ] is in row echelon form, the method is called
Gaussian Elimination
⚫ If [ C:D ] is in reduced row echelon form, the method is
called Gauss-Jordan Elimination

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 16
What do you think?

 Given a nonzero matrix A, can we perform elementary row operations to

transform A to
⚫ Row echelon form? (true or false)
⚫ Reduced row echelon form? (true or false)
⚫ If your answer is false, can you provide an example?

𝑎11 𝑎12 ⋯ 𝑎1𝑛 0 1 0 0 ⋯ 0

𝑎21 𝑎22 ⋯ 𝑎2𝑛 0 0 1 0 ⋯ 0
𝑎31 𝑎32 ⋯ 𝑎3𝑛 ➔ 0 0 0 1 ⋯ 0
⋮ ⋮ ⋮
𝑎𝑛1 𝑎𝑛2 ⋯ 𝑎𝑚𝑛 0 0 0 0 ⋯ 0

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 17
What do you think? (Discussion)

Solution

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 18
Using elementary row operations to solve linear systems
 Theorem 2.3: Let A and C be m x n matrices. If the system AX = b
has an augmented matrix [ A:b ] which is row equivalent to an
augmented matrix [ C:d ], then the systems AX = b and CX = d
have the same solution.

 Proof: The elementary row operations on the augmented matrices

correspond to the following operations on the equations, which
leave the solution unchanged.
a) Interchange two equations
b) Multiply a given equation by a nonzero constant
c) Add a constant times one equation to another equation
So, the two systems have the same solution

QED
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 19
Using elementary row operations to solve linear systems

 Theorem 2.3
 Since performing elementary row operations on the augmented
matrices correspond to the following operations on the equations
a) Interchange two equations
b) Multiply a given equation by a nonzero constant
c) Add a constant times one equation to another equation
❑ ➔Two row equivalent augmented matrices corresponds to two
equivalent linear systems (same solutions)
 If the system AX = b has an augmented matrix [ A:b ] which is
row equivalent to an augmented matrix [ C:d ], then the systems
AX = b and CX = d have the same solution
⚫ A and C are m x n matrices.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 20
Your Turn

 True or false: If A and C are row equivalent m x n matrices,

then the homogeneous systems AX = 0 and CX = 0 are
equivalent.

Discussion

QED

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 21
 22
Back substitution

 If Gauss-Jordan reduction is used, the solution is just read off. If

Gaussian elimination is used, the solution must be found by back
substitution
1 2 3 4 5 6 Note 𝑥5 is the free variable
 
C : D = 
0 1 2 3 −1 7
  0 0 1 2 3 7 Set x5 = r and backsolve
 
0 0 0 1 2 9  x1 = −1 −10r
x2 = 2 + 5r
x1 + 2 x2 + 3x3 + 4 x4 + 5 x5 = 6
x3 = −11 + r
x2 + 2 x3 + 3x4 − x5 = 7 x4 = 9 − 2r
x3 + 2 x4 + 3x5 = 7
Infinite number of solutions
x4 + 2 x5 = 9
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen
22 Lee, Ph.D.
Identify inconsistent system in row echelon form

Example
1 2 3 4 5
 
C : D  =  0 1 2 3 6
 
0 0 0 0 1


The last equation is 0 x1+0 x2+0 x3+0 x4 = 1, which cannot be

satisfied

 Clearly, several things can happen and a careful and organized

analysis of the problem is required

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 23
Echelon Form of a Matrix

Homogeneous Systems
 Homogeneous system, AX = 0, of m equations in n unknowns
occurs often applications
 Example (1) How many free variables in the solution
of this system?
1 0 0 0 2 0

0 0 1 0 3 0
0 0 0 1 4 0 (2) What does it mean to have free variables?
 
0 0 0 0 0 0

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 24
The Case of m<n (self-learning)

Homogeneous Systems
 Theorem 2.4: A homogeneous system of m linear equations
in n unknowns always has a nontrivial solution if m < n, i.e. if
the number of unknowns exceeds the number of equations.

 Proof - Let B be a matrix in reduced row echelon form that is

row equivalent to A. The homogeneous systems AX = 0 and
BX = 0 are equivalent. (proof continue on next slide)

A B
𝑎11 𝑎12 ⋯ 𝑎1𝑛 0 1 𝑏12 0 ⋯ 0 0
𝑎2𝑛 𝑎22 ⋯ 𝑎2𝑛 0 0 0 1 ⋯ 0 0
➔
⋱ ⋮ ⋱ ⋮
𝑎𝑚1 𝑎𝑚2 ⋯ 𝑎𝑚𝑛 0 0 0 0 ⋯ 1 0

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 25
Echelon Form of a Matrix (self-learning)

Homogeneous Systems
 Proof (continue): Let z be the number of nonzero rows of B.
Then z ≤ m. Since m < n, then z < n. So, we are solving z
equations in n unknowns and can solve for z of the unknowns
in terms of the remaining n - z unknowns, which can take any
values. So, BX = 0 and thus AX = 0 have nontrivial solutions.

A B
𝑎11 𝑎12 ⋯ 𝑎1𝑛 0 1 𝑏12 0 ⋯ 0 0
𝑎2𝑛 𝑎22 ⋯ 𝑎2𝑛 0 0 0 1 ⋯ 0 0
➔
⋱ ⋮ ⋱ ⋮
𝑎𝑚1 𝑎𝑚2 ⋯ 𝑎𝑚𝑛 0 0 0 0 ⋯ 1 0

At most m non-zero rows QED

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 26
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 27
Elementary Matrices
 Definition: - An n x n elementary matrix of Type I, Type II or
Type III is a matrix obtained from the identity matrix In by
performing a single elementary row operation of Type I,
Type II or Type III, respectively
⚫ Type I - exchange two rows
⚫ Type II - multiply a row by a nonzero constant
⚫ Type III - add a multiple of one row to another

 Each elementary row operation can be performed on a

matrix A by multiplying A by an appropriate elementary
matrix E

 This is useful for proofs, but not for practical computations

since too many operations are required.
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 28
Your Turn

 Which of these are elementary matrices and nonelementary matrices?

1 0 0  1 0 0 
(a ) 0 3 0 (b) 1 0 0 (c) 0 1 0
0 1 0
0 0 1 0 0 0

1 0 0  1 0 0 
(d ) 0 0 1 (e) 1 0 ( f ) 0 2 0 
0 1 0 2 1 0 0 −1

Multiply 2nd row by 3

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 29
Elementary Matrices (Type I)
Elementary Row Operations
 Interchange rows i and j of matrix A
0 1 0  a11 a12 a13 a14  a21 a22 a23 a24 
    
1 0 0  a21 a22 a23 a24  =  a11 a12 a13 a14 
0
 0 1  a31 a32 a33 a34   a31 a32 a33 a34 
In general, rows i and j of A can be interchanged by
multiplying A by the m x m matrix E defined as
1 if k  i and k  j
e pq = 0 except for ekk = 
0 if k = i or k = j
and eij = 1, e ji = 1
E is a special case of a permutation matrix

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 30
Elementary Matrices (Type II)

Elementary Row Operations

 Multiply row i of A by c ≠ 0

1 0 0  a11 a12 a13 a14   a11 a12 a13 a14 

    
0 c 0  a21 a22 a23 a24  = ca21 ca22 ca23 ca24 
0
 0 1  a31 a32 a33 a34   a31 a32 a33 a34 
In general, row i of A can be multiplied by c by multiplying A by the
m x m matrix E defined as
0 if p  q

e pq = 1 if p = q  i
c if p = q = i


Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 31
Elementary Matrices (Type III)

Elementary Row Operations

 Add c times row i of A to row j of A, i ≠ j
1 0 0  a11 a12 a13 a14 
  
0 1 0  a21 a22 a23 a24 
0
 c 1  a31 a32 a33 a34 
 a11 a12 a13 a14 
 
=  a21 a22 a23 a24 
ca + a ca22 + a32 ca23 + a33 ca24 + a34 
 21 31

In general, can add c times row i to c p = j and q = i


row j of A by multiplying A by the pq 1 p = q
e =
0 otherwise
m x m matrix E defined as 

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 32
Additional Comments on Elementary Matrices

Comments
 The appropriate elementary matrix E may be obtained by
performing the desired operation on the rows of the m x m
identity matrix Im

 Can define elementary matrices F to perform elementary

column operations by doing the corresponding operations
on the columns of the identity matrix. The matrices F are
applied by post multiplying, i.e. AF

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 33
Elementary Matrices
 Theorem 2.5: Let A be an m x n matrix and let an elementary row
operation of Type I, Type II or Type III be performed on A to yield
matrix B. Let E be the elementary matrix obtained from Im by
performing on it the same elementary row operation that was
performed on A. Then B = EA.

 Proof: (Theorem 2.5) Omitted.

⚫ Try it yourself! Do a general argument based on the examples in the previous
slides.
 This theorem means that the following two things are essentially the
same:
Type I row operation 𝐵 = 𝐸1 𝐴
A→ Type II row operation →B 𝐵 = 𝐸2 𝐴
Type III row operation 𝐵 = 𝐸3 𝐴

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 34
Elementary Matrices and Row Equivalence
 Theorem 2.6 - If A and B are m x n matrices, then A is row
𝑘
equivalent to B if and only if 𝑩 = 𝑬𝒌 𝑬𝒌−𝟏 … 𝑬𝟐 𝑬𝟏 𝑨, where 𝑬𝑖 𝑖=1
are m x m elementary matrices.

 Proof -  Suppose A is row equivalent to B. By definition, B can be

obtained by applying a finite number of elementary row operations to A.
Each elementary row operation can be accomplished by multiplying by
an appropriate elementary matrix. So the transformation from A to B
can be accomplished by multiplying by a sequence of elementary
matrices  i i=1 . So B = Ek Ek−1 E2E1A
k
E

  Now suppose that we have B = Ek Ek−1 E2E1A where each Ei is

an elementary matrix. Since B is obtained by applying a sequence of
elementary row operations to A, then A is row equivalent to B. QED
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 35
Your Turn

 Using elementary matrices

 Find a sequence of elementary matrices that can be used to write the matrix
A in row-echelon form.
0 1 3 5
A =  1 − 3 0 2
 
2 − 6 2 0

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 36
Your Turn

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 37
Inverse of Elementary Matrices

 Elementary Matrix  Inverse Matrix

0 1 0  0 1 0 
E1 = 1 0 0 = R12 ( R12 ) = E = 1 0 0 = R12
−1
1
−1
(Elementary Matrix)
0 0 1 0 0 1
Type III

 1 0 0 1 0 0 
E2 =  0 1 0 = R13( −2 ) (R ( −2 ) −1
13
−1
) = E = 0 1 0 = R13( 2 ) (Elementary Matrix)
2
− 2 0 1 2 0 1

 1 0 0 1
( )
1
( ) 1 0 0
E3 = 0 1 0 = R3 2 ) = E = 0 1 0 = R3 (Elementary Matrix)
2 −1 −1 ( 2)
(R 3 3
0 0 12  0 0 2
Type II

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 38
 Inverse of Elementary Matrices

 Theorem 2.7: An elementary matrix is nonsingular and its inverse

is an elementary matrix of the same type.

 Proof
⚫ Type I - Let E switch rows i and j. Then the effects of E may be
undone by applying E again, i.e. EE = I. So, E is invertible and E-1 = E
⚫ Type II - Let E multiply the ith row by c ≠ 0. Let F multiply the ith row
by 1 / c. Then EF = FE = I. So, E is invertible and E-1 = F
⚫ Type III - Let E add c times the ith row to the jth row. Let F add (-c)
times the ith row to the jth row. Then EF = FE = I. So, E is invertible
and E-1 = F

Nonsingular matrix: 非奇異矩陣

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 39
 Inverse of Elementary Matrices

 Lemma 2.1: Let A be a an n x n matrix and let the homogeneous system AX = 0 have
only the trivial solution X = 0 . Then A is row equivalent to In .

 Theorem 2.8: A is nonsingular if and only if A is a product of elementary matrices.

 Corollary 2.2: A is nonsingular if and only if A is row equivalent to In

 Theorem 2.9: The homogeneous system of n linear equations in n unknowns, AX =

0, has a nontrivial solution if and only if A is singular

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 40
Your Turn
1 0 0 0 0
0 1 0 0 0
 What is the inverse of 0 0 1 0 0?
0 71 0 1 0
0 0 0 0 1

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 41
Elementary Matrices

Equivalent statements for n x n matrix A

1. A is nonsingular
2. AX = 0 has the trivial solution only. (implied by 1.)
3. A is row (column) equivalent to In. (implied by 2.)
4. The system AX = B has a unique solution for every n x 1 matrix B (implied by 1.)
5. A is a product of elementary matrices. (implied by 3.)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 42
Animated Linear Algebra
 Linear transformations and matrices | Chapter 3, Essence of linear algebra (https://www.youtube.com/watch?v=kYB8IZa5AuE)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 43
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 44
Finding A-1
 Eg: From A to I, and finding 𝐴−1
Find a sequence of elementary matrices whose product is
 − 1 − 2
A= 
Sol:
 3 8
− 1 − 2 r1( −1) 1 2 r12( −3 ) 1 2
A=  ⎯⎯
⎯→   ⎯⎯ ⎯→  
3 8 3 8  0 2 
1 2 r21( −2 ) 1 0
1
( ) 1
⎯⎯→ 
r2 2
⎯⎯⎯→  = I Therefore R21
( )
( −2 )
  R2 2 R12( −3) R1( −1) A = I
0 1  0 1
1
( )
Thus A = ( R1( −1) ) −1 ( R12( −3) ) −1 ( R ) ( R21
2
2 −1
( −2 ) −1
)

( −1) − 1 0 1 0 1 0 1 2
=R 1
( 3)
R R R
12
( 2)
2
( 2)
21 =  3 1 0 2 0 1
 0 1    
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 45
Finding A-1

 Have shown that if A is nonsingular, then it is row equivalent

to In . Also, have shown that it can be expressed as the
product of elementary matrices.
I n = Ek Ek −1 E2E1A  A = E1−1E2−1 E−k −11E−k 1

−1 −1 −1
 Then 𝑨−1 = 𝑬1−1 𝑬−1
2 ⋯ 𝑬 𝑬
𝑘−1 𝑘 = 𝑬𝑘 𝑬𝑘−1 ⋯ 𝑬2 𝑬1 So,
A-1 can be represented as the product of the elementary
matrices that reduce A to In
 Create the partitioned matrix 𝑨 𝑰𝑛 and apply the
operations that reduce A to In
Ek Ek −1 E2E1  A I n  = I n A −1 

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 46
Finding A-1

Example  1 1 3
 Compute inverse of A = 1 2 3
0 1 1

1 1 3 1 0 0  1 1 3 1 0 0
1 2 3 0 1 0   0 1 0 −1 1 0 
   
0 1 1 0 0 1  0 1 1 0 0 1 
1 1 3 1 0 0  1 1 0 −2 3 −3
 0 1 0 −1 1 0   0 1 0 −1 1 0
   
0 0 1 1 −1 1  0 0 1 1 −1 1 
1 0 0 −1 2 −3  −1 2 −3
 0 1 0 −1 1 0   A −1 =  −1 1 0
   
0 0 1 1 −1 1   1 −1 1 

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 47
Finding A-1
Equivalent statements for n x n matrix A
(proof omitted)
1. A is nonsingular
2. AX = 0 has the trivial solution only. (implied by 1.)
3. A is row (column) equivalent to In. (implied by 2.)
4. The system AX = B has a unique solution for every n x 1
matrix B (implied by 1.)
5. A is a product of elementary matrices. (implied by 3.)
6. det(A)≠ 0. (will introduced in Chapter 3)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 48
Finding A-1

Equivalent statements for n x n matrix A

⚫ A is singular
⚫ AX = 0 has non-trivial solutions. (proof omitted,
see text book)
⚫ A is row equivalent to B with a row of zeros (will
discuss about this later).

 Theorem 2.10: An n x n matrix A is singular if and only if A

is row equivalent to a matrix B that has a row of zeros.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 49
Finding A-1

 Consider the following example.

 Given a square matrix A, we can drive it toward reduced
row echelon form by elementary row operations.
 What will we get in the end?
𝑟𝑜𝑤 𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑟𝑜𝑤 𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑟𝑜𝑤 𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛
𝐴 𝐵1 𝐵2 … 𝑍
 Will we get an identity matrix? What are the possibilities?

 Ans: Two cases; Z can be an identify matrix, or not.

If Z is not an identity matrix, then it must has a row of zeros.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 50
Finding A-1

 Theorem 2.10: An n x n matrix A is singular if and only if A

is row equivalent to a matrix B that has a row of zeros.

 Proof
⚫  Let A be singular. Apply Gauss-Jordan reduction to A to get a
matrix B in reduced row echelon form which is row equivalent to A.
B cannot be In since if it were, A would be nonsingular. Since B is
in reduced row echelon form, B must have at least one row of
zeros at the bottom.
(… continue next slide)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 51
Finding A-1
 Proof
⚫  Let a A be row equivalent to a matrix B that has a row of all zeros.
⚫ [First show that B is singular]
⚫ The matrix B is singular since there does not exist a matrix C such that BC
= CB = In .
⚫ To see this, let the ith row of B consist of all zeros. The ith row of BC is
generated by taking the ith row of B and multiplying each column of C by it.
So, the ith row of BC is all zeros, but BC = In . So, B must be singular.
𝑏11 𝑐11 1
1
0 0 ⋯ 0 ⋯ = 0 0 ⋯ 0
i-th row i-th row
𝑏𝑛𝑛 𝑐𝑛1 𝑐𝑛𝑛 0 1

⚫ [Show that B is singular → A is singular]

⚫ A is row equivalent to B, so 𝑩 = 𝑬𝒌 𝑬𝒌−𝟏 ⋯ 𝑬𝟐 𝑬𝟏 𝑨
⚫ If A is nonsingular then so is B since it is the product of nonsingular
matrices. Since we know B is singular, then A must be singular. QED
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 52
Finding A-1

 The preceding theorem gives a way to determine if A is

singular. Also, the determination of singularity/nonsingularity
can be made in the process of computing A-1

 Form the augmented matrix [𝑨|𝑰𝑛 ], then put it into reduced

row echelon form to get 𝑪 𝑫 .
⚫ If C = In then A is nonsingular and D = A-1
⚫ If C ≠ In then C has a row of zeros and thus is singular.
So, A is singular and A-1 does not exist

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 53
Finding A-1
 Theorem 2.11: If A and B are n x n matrices such that AB = In,
then BA = In . Thus A is nonsingular and B = A-1

 Proof (by contradiction)

⚫ First show that if AB = In then A is nonsingular.

⚫ Suppose that A is singular. Then A is row equivalent to a matrix C that
𝑘
has a row of zeros, i.e. 𝐶 = 𝐸𝑘 𝐸𝑘−1 … . 𝐸2 𝐸1 𝐴, where 𝐸𝑖 𝑖=1 is a set of
elementary matrices.
⚫ Since 𝐶𝐵 = 𝐸𝑘 𝐸𝑘−1 … . 𝐸2 𝐸1 𝐴𝐵, so CB is row equivalent to AB.
⚫ Since C has a row of zeros, so does CB.
⚫ AB is singular since it is row equivalent to a matrix with a row of zeros.
⚫ However AB = In , which is nonsingular.
─ ➔Contradiction.
⚫ So, A is nonsingular and A-1 exists. A-1 ( AB ) = A-1 In so B = A-1
QED
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen
54 Lee, Ph.D.
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 55
Equivalent Matrices
 Defn - If A and B are two m x n matrices, then A is equivalent to B if
B is obtained from A by a finite sequence of elementary row or
elementary column operations

 Theorem 2.12: If A is any nonzero m x n matrix, then A is equivalent

to a partitioned matrix of the form
 Ir r 0 n −r 
 
0 0
 m−r r m−r n −r 

 Theorem 2.13: Two m x n matrices A and B are equivalent if and

only if B = PAQ for some nonsingular matrices P and Q.

 Theorem 2.14: An n x n matrix A is nonsingular if and only if A is

equivalent to In
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 56
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares (OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 57
LU-Factorization

 Current method of solving linear equations, Gauss-Jordan reduction, consists of

forming the augmented matrix and applying Gauss-Jordan
A B  reduction to it to
obtain a matrix where C is the solution. I n C
 

 Gauss-Jordan reduction works fine for small systems. For large systems, there are
more efficient methods that do not require getting the reduced row echelon form, or
even the row echelon method.
 If the coefficient matrix has either of the triangular forms, the system is easy to solve.
 Will develop a method for solving a general system AX = B by a method based on
these observations.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 58
LU-Factorization

Observations
 Upper triangular system AX = B

 a11 a12 a13 a1n b1 

0 a a23 a2 n b2 
 22

0 0 a33 a3n b3  aii  0 for 1  i  n
 
 
 0 0 0 0 ann bn  System may be solved by
back substitution
ann xn = bn  xn = bn ann
(
an−1,n−1xn−1 + an−1,n xn = bn−1  xn−1 = bn−1 − an−1,n xn an−1,n−1 )
 n 
x j =  bj −
 
= +
a jk xk 

a jj j = n −1, n − 2, ,1
 k j 1 
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 59
LU-Factorization

Observations
 Lower triangular system AX = B
a 0 0 0 b1 
 11 
 a21 a22 0 0 b2 
a a32 a33 0 b3  aii  0 for 1  i  n
 31 
 
a an 2 an3 ann bn  System may be solved by
 n1 forward substitution
a11x1 = b1  x1 = b1 a11
a21x1 + a22 x2 = b2  x2 = (b2 − a21x1 ) a22
 j −1 
 
x j =  b j − a jk xk 
= 
a jj j = 2,3, , n
 k 1 
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 60
LU-Factorization

 Defn - An n x n matrix A is said to be in triangular form when it satisfies the

following properties
a) A is either an upper or lower triangular matrix
b) All of the diagonal entries of A are nonzero
 Defn - Let A be a nonsingular matrix and suppose that there is a lower triangular
matrix L and an upper triangular matrix U such that A = LU. Then A is said to have
an LU factorization or an LU decomposition
 Method - To solve AX = B where A = LU, express the system as LUX = B. Let UX
= Z, then LZ = B. Solve this system by forward substitution to get Z, then solve UX
= Z by back substitution to get X.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 61
LU-Factorization

 Example - Consider AX = B

 6 −2 −4 4  x1   2
   
 3 −3 −6 1  x2   −4
=
 −12 8 21 −8  x3   8
    
 −6 0 −10 7   x4   −43

Can show A has an LU factorization

 1 0 0 0 6 −2 −4 4

 12 1 0 0 
0 −2 −4 −1
L= U= 
 −2 −2 1 0 0 0 5 −2
   
 −1 1 −2 1 0 0 0 8

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 62
LU-Factorization
 Example (continued)
A = LU, let Z = UX and solve LZ = B
z1 = 2,
 1 0 0 0  z1   2 1 z + z = −4  z = − 5
  
1 2 1 0 0  z2   −4 2 1 2 2
=
 −2 −2 1  
0 z3  8  −2 z1 − 2 z2 + z3 = 8  z3 = 2
    
 −1 1 −2 1  z4   −43 − z1 + z2 − 2 z3 + z4 = −43  z4 = −32

Solve UX = Z by back substitution

8 x4 = −32  x4 = −4
6 −2 −4 4  x1   2 
   5 x3 − 2 x4 = 2  x3 = −1.2
0 −2 −4 −1  x2   −5
= −2 x2 − 4 x3 − x4 = −5  x2 = 6.9
0 0 5 −2 3  2
  x 
    
0 0 0 8  4   −32
x 6 x1 − 2 x2 − 4 x3 + 4 x4 = 2  x1 = 4.5
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 63
LU-Factorization

Questions
 How do we find L and U?

 Are L and U unique? (omitted)

 Are there other computational issues? (omitted)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 64
LU-Factorization

 If the nn matrix A can be written as the product of a lower triangular matrix L
and an upper triangular matrix U, then A=LU is an LU-factorization of A

A = LU L is a lower triangular matrix

U is an upper triangular matrix

 If a square matrix A can be row reduced to an upper triangular matrix U using

only the row operation of adding a multiple of one row to another row below
it, then it is easy to find an LU-factorization of A.

Ek  E2 E1 A = U
A = E1−1 E2−1  Ek−1U
A = LU
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 65
LU-Factorization

Eg: LU-factorization
 1 − 3 0
(a) A = 1 2 (b) A = 0 1 3
1 0 2 − 10 2
Sol: (a)

A=  1 2  ⎯⎯→ 1 2  = U
r12(-1)
1 0 0 − 2

 R12( −1) A = U
 A = ( R12( −1) ) −1U = LU

( −1) −1 1 0
 L = (R 12 ) =R (1)
12 = 
1 1 

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 66
LU-Factorization
Eg: (LU-factorization)
 1 − 3 0
(a) A = 1 2 (b) A = 0 1 3
1 0 2 − 10 2

(b)  1 − 3 0  1 − 3 0 1 − 3 0 

A= 0 
1 3 ⎯⎯r13( −2 )
⎯→ 0  
1 3 ⎯⎯→ 0 1 3  = U
(4)
r23
     
2 − 10 2 0 − 4 2 0 0 14
( 4 ) ( −2 )
 R23 R13 A = U

 A = ( R13( −2) ) −1 ( R23

( 4 ) −1
) U = LU

 L = ( R13( −2) ) −1 ( R23

( 4 ) −1 ( −4 )
) = R13( 2) R23
1 0 0 1 0 0 1 0 0
=  0 1 0  0 1 0  =  0 1 0 
2 0 1 0 − 4 1 2 − 4 1
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 67
LU-Factorization

 Solving Ax=b with an LU-factorization of A

Ax = b If A = LU , then LUx = b
Let y = Ux, then Ly = b

Two steps:

(1) Write y = Ux and solve Ly = b for y

(2) Solve Ux = y for x

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 68
LU-Factorization

 Eg: Solving a linear system using LU-factorization

x1 − 3 x2 = −5
x2 + 3 x3 = − 1
2 x1 − 10 x2 + 2 x3 = − 20
Sol:

 1 − 3 0  1 0 0  1 − 3 0
A = 0 1 3 = 0 1 0  0 1 3 = LU
    
2 − 10 2 2 − 4 1 0 0 14
(1) Let y = Ux, and solve Ly = b

 1 0 0  y1   − 5  y1 = −5
 0 1 0   y 2  =  − 1   y = −1
2 − 4 1  y3  − 20 2
y3 = −20 − 2 y1 + 4 y2
= −20 − 2( −5) + 4( −1) = −14
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 69
LU-Factorization

(2) Solve the following system Ux = y

 1 − 3 0  x1   − 5 
0 1 3  x2  =  − 1 
0 0 14  x3  − 14
So x3 = −1
x2 = −1 − 3 x3 = −1 − (3)(−1) = 2
x1 = −5 + 3 x2 = −5 + 3(2) = 1
Thus, the solution is

1
x=2
 
− 1

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 70
LU-Factorization
0 1 2
 Example: Find the LU factorization of 𝐴 = 1 1 1.
2 3 5
 Sol:
(1)
We cannot do 𝑟21 because LU factorization required to perform only the
row operation of adding a multiple of one row to another row below it.

 In this case, we cannot perform LU factorization for A unless switching

two rows first.
0 1 0
⚫ Use 𝑃 = 1 0 0 , a permutation matrix, to switch the first two
0 0 1
1 1 1
rows: 𝑃𝐴 = 0 1 2 .
2 3 5
⚫ Now we can perform LU factorization on 𝑃𝐴

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 71
LU-factorization

 Example Sol. (continued):

 𝑃𝐴 =
1 1 1 1 1 1 1 1 1
(−2) −1
0 1 2 →𝑟 13 → 0 1 2 →𝑟23 → 0 1 2 = 𝑈;
2 3 5 0 1 3 0 0 1
−1 −2
 𝑅23 𝑅13 𝑃𝐴 = 𝑈 →
1 0 0 1 0 0 1 0 0 1 1 1
2 1
 𝑃𝐴 = 𝑅13 𝑅23
𝑈= 0 1 0 0 1 0 𝑈= 0 1 0 0 1 2
2 0 1 0 1 1 2 1 1 0 0 1
0 1 0 1 0 0 1 1 1
−1
 𝐴 = 𝑃 𝐿𝑈 = 1 0 0 0 1 0 0 1 2
0 0 1 2 1 1 0 0 1

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 72
Your Turn
1 1 1 1
1 2 3 4
 Find an LU factorization of the matrix 𝐴 = .
1 3 6 10
1 4 10 20

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 73
Keywords

 elementary row operation: 基本列運算

 row equivalent: 列等價
 row-echelon form: 列梯形形式
 reduced row-echelon form: 簡化列梯形形式
 Gaussian elimination: 高斯消去法
 Gauss-Jordan elimination: 高斯-喬登消去法
 homogeneous system: 齊次系統
 trivial solution: 顯然解
 nontrivial solution: 非顯然解

Productivity
1.74 Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D.
Topics

 Preliminaries
 Echelon Form of a Matrix
 Elementary Matrices
 Finding A-1
 Equivalent Matrices
 LU-Factorization
 Applications: Polynomial Curve Fitting
 Applications: The Global Positioning System (GPS)
 Applications: Linear Regression by Ordinary Least Squares
(OLS)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 75
Applications: Polynomial Curve Fitting
 Polynomial Curve Fitting (Larson, 2016, p.26)
⚫ Given n points (x1, y1), (x2, y2),…, (xn, yn) on the xy-plane, find a
polynomial function of degree n–1 passing through these n points

Productivity
1.76 Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D.
The Global Positioning System (GPS)

 GPS is used in a variety of situations for determining

geographical locations.
 Very useful for civilian and military use.
 How does it work?
 It solves a linear system using the information received
from four satellites.

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 77
GPS
Z: through North Pole
 To simplify the discussion, set the radius of
the earth to 1.
 The surface of earth is:
𝑥2 + 𝑦2 + 𝑧2 = 1
 Time unit: 1/100 seconds
x
 Normalized Light speed = 0.47
y
⚫ Light speed: 299792458 m/s
⚫ Earth radius: 6371000 m
⚫ Normalized light speed: (L) / (E) = 47.05579 unit/s
 Idea: three-dimensional triangulation.
 Complication: We do not know the distance
to the satellites!

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 78
GPS (Cont’d.)

 Need very precise clock to be able to compute the distance from the receiver
to the satellites!
 This type of clock is very expensive.
 Need a cheap solution
 → Solve for distance using one additional satellite. Time for sending
 signal from satellite
Satellite Position Time (from
midnight) 1/100 s
1 (1.11, 2.55, 2.14) 1.29
2 (2.87, 0.00, 1.43) 1.31
3 (0.00, 1.08, 2.29) 2.75
4 (1.54, 1.01, 1.23) 4.06

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 79
GPS (Cont’d.)

 Let (𝑥, 𝑦, 𝑧) be your position, and let 𝑡 be the signals arrive.

 What is the distance from you to the first satellite?
⚫ 𝑑 = 0.47 (𝑡 − 1.29)
⚫𝑑= 𝑥 − 1.1 2 + 𝑦 − 2.55 2 + 𝑧 − 2.14 2

 Putting the two equations together and rearrange,

⚫ 𝑥 − 1.11 2 + 𝑦 − 2.55 2 + 𝑧 − 2.14 2 = 0.472 𝑡 − 1.29 2
⚫ ➔ 2.22𝑥 + 5.10𝑦 + 4.28𝑧 − 0.57𝑡 = 𝑥 2 + 𝑦 2 + 𝑧 2 − 0.22𝑡 2 + 11.95

Satellite Position Time (from

midnight)
1 (1.11, 2.55, 2.14) 1.29
Equation (1)
2 (2.87, 0.00, 1.43) 1.31
3 (0.00, 1.08, 2.29) 2.75
4 (1.54, 1.01, 1.23) 4.06
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 80
GPS (Cont’d.)

 Repeat the process for all satellites

⚫ 2.22𝑥 + 5.10𝑦 + 4.28𝑧 − 0.57𝑡 = 𝑥 2 + 𝑦 2 + 𝑧 2 − 0.22𝑡 2 + 11.95
⚫ 5.74𝑥 + 2.86𝑧 − 0.58𝑡 = 𝑥 2 + 𝑦 2 + 𝑧 2 − 0.22𝑡 2 + 9.90
⚫ 2.16𝑦 + 4.58𝑧 − 1.21𝑡 = 𝑥 2 + 𝑦 2 + 𝑧 2 − 0.22𝑡 2 + 4.74
⚫ 3.08𝑥 + 2.02𝑦 + 2.46𝑧 − 1.79𝑡 = 𝑥 2 + 𝑦 2 + 𝑧 2 − 0.22𝑡 2 + 1.26

 Subtract the first equation from each of the other equations:

⚫ 3.52𝑥 − 5.10𝑦 − 1.42𝑧 − 0.01𝑡 = −2.05
⚫ −2.22𝑥 − 2.94𝑦 + 0.30𝑧 − 0.64𝑡 = −7.21
⚫ 0.86𝑥 − 3.08𝑦 − 1.82𝑧 − 1.22𝑡 = −10.69

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 81
GPS (Cont’d.)

 The augmented matrix:

3.52 −5.10 −1.42 −0.01 −2.05
 −2.22 −2.94 0.30 −0.64 −7.21 −→
0.86 −3.08 −1.82 −1.22 −10.69
1 0 0 0.36 2.97
 0 1 0 0.03 0.81 −→
0 0 1 0.79 5.91
 𝑥 = 2.97 − 0.36𝑡
 𝑦 = 0.81 − 0.03𝑡
 𝑧 = 5.91 − 0.79𝑡

 Substitute to Equation (1)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 82
GPS (Cont’d.)

 Substitute to Equation (1)

 2.97 − 0.36𝑡 − 1.11 2 + 0.81 − 0.03𝑡 − 2.55 2
+ 5.91 − 0.79𝑡 − 2.14 2
=
0.472 𝑡 − 1.29 2
⚫ ➔ 0.54𝑡 2 − 6.65𝑡 + 20.32 = 0
⚫ ➔ 𝑡 = 6.74 and 𝑡 = 5.56
 The first solution gives 𝑥, 𝑦, 𝑧 = (0.55, 0.61, 0.56)
 The second solution gives 𝑥, 𝑦, 𝑧 = 0.96, 0.65, 1.46 . (not on surface,
rejected)

 ➔ 𝑥, 𝑦, 𝑧 = (0.55, 0.61, 0.56)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 83
Ordinary Least Squares (OLS)
 Ordinary Least Squares (OLS)
𝑝
⚫ Linear equation 𝑦𝑖 = 𝛽0 + σ𝑗=1 𝛽𝑗 𝑥𝑗𝑖 + 𝜀𝑖
─ 𝑦𝑖 : response variable
─ 𝑥𝑗𝑖 : independent variable
─ 𝛽0 : intercept
─ 𝛽𝑗 : coefficient of the independent variable (i.e. slope)
─ 𝜀: noise (or call residual)
⚫ The aim of this approach is to minimize the residual for all residuals
⚫ We square the residual before minimizing
⚫ We can derive the intercept and slope parameter using basic calculus
𝑝
y y 𝑦ො𝑖 = 𝛽መ0 + σ𝑗=1 𝛽መ𝑗 𝑥𝑗𝑖
𝜀

Induction
(歸納法)
x x
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen
84 Lee, Ph.D.
Ordinary Least Squares (OLS)
 Minimizing Residual Sum of Squares (RSS)
𝑝
⚫ Minimize 𝑅𝑆𝑆 = σ𝑛𝑖=1 𝜀𝑖2 = σ𝑛𝑖=1(𝑦𝑖 − 𝑦ො𝑖 )2 = σ𝑛𝑖=1(𝑦𝑖 − (𝛽0 + σ𝑗=1 𝛽𝑗 𝑥𝑗𝑖 ))2
⚫ Minimize 𝑅𝑆𝑆 = (𝒀 − 𝑿𝜷)𝑇 (𝒀 − 𝑿𝜷)
⚫ This is a quadratic function with the p+1 variables. Differentiating with
respect to 𝛽, we obtain
𝜕𝑅𝑆𝑆
─ = −2𝑿𝑇 𝒀 − 𝑿𝜷
𝜕𝜷
𝜕2 𝑅𝑆𝑆
─ = 2𝑿𝑇 𝑿
𝜕𝜷𝜕𝜷𝑇
⚫ Assume that 𝑿 has full column rank, and hence 𝑿𝑇 𝑿 is positive definite,
we se the first derivative to zero.
─ 𝑿𝑇 𝒀 − 𝑿𝜷 = 𝟎
⚫ Then, we obtain the unique solution For 𝑦𝑖 = 𝛽0 + 𝛽1 𝑥1𝑖 + 𝛽2 𝑥2𝑖

─𝜷෡ = (𝑿𝑇 𝑿)−1 𝑿𝑇 𝒀 𝛽

1

𝛽2

𝛽0 𝛽1 𝛽2
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 85
Animated Linear Algebra
 Matrix multiplication as composition | Chapter 4, Essence of linear algebra (https://www.youtube.com/watch?v=XkY2DOUCWMU)

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 86
Self-Practice Ch. 2

 Assignment discussion in recitation class…

 2.1: 5, 8
 2.2: 13, 15, 28
 2.3: 11, 22, 26, 28
 2.4: 2, 9
 2.5: 8, 9

Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 87
Productivity Optimization Lab@NTU LA02_ Solving Linear Systems Chia-Yen Lee, Ph.D. 88