Lecture 2
Solving Linear Equations
CL249
Error Analysis: Highlights
• Numerical techniques yield solutions that are approximations to exact
solutions.
• Use significant digits- number of digits that can be used with
confidence.
Usually Significant. digits include all the digits that are certain plus one
uncertain digit
Accuracy and Precision
• Accuracy- how closely a computed value agrees with true value.
• Precision: How closely individually measured/computed values agree
with each other. This is related to the number of significant figures
used while reporting the answer.
Significant Digits
• All non-zero digits are significant
• Zeros appearing anywhere between two non-zero digits are
significant.
• Trailing zeros in a number containing a decimal point
• Exact numbers, and irrationally defined numbers like Euler’s number
(e) and pi (π), have an infinite number of significant figures.
• For any value written in scientific notation as A ×10x, the number of
significant figures is determined by applying the above rules only to
the value of A; the x is considered an exact number and thus has an
infinite number of significant figures.
Not Significant
• Leading zeros
Unclear: The significance of trailing zeros in a number not containing
a decimal point can be ambiguous.
Use Scientific Notation to eliminate ambiguity
Errors
Inherent Errors: Errors in input data
Two types:
Data Error: Experimental data may have limited accuracy and precision.
We must choose the significant digits accordingly.
Conversion Error: Limitation of the computer representation
Numerical Errors: Procedural Errors:
Roundoff Errors
Truncation Errors
Other Errors
• Modeling Errors: Wrong parameters, wrong model,
• Blunders: human errors
Types of Errors
• Truncation Errors and Round off errors
• True value=approx. value + error
So, 𝐸𝑡 = 𝑡𝑟𝑢𝑒 𝑣𝑎𝑙𝑢𝑒 − 𝑎𝑝𝑝𝑟𝑜𝑥. 𝑣𝑎𝑙𝑢𝑒
More useful to look at fractional error
𝑇𝑟𝑢𝑒 𝑒𝑟𝑟𝑜𝑟 𝑇𝑟𝑢𝑒 𝑒𝑟𝑟𝑜𝑟
• 𝑇𝑟𝑢𝑒 𝑓𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝑎𝑙 𝑒𝑟𝑟𝑜𝑟 = or *100%
𝑇𝑟𝑢𝑒 𝑣𝑎𝑙𝑢𝑒 𝑇𝑟𝑢𝑒 𝑣𝑎𝑙𝑢𝑒
• In reality, we may not know the exact solution
𝐴𝑝𝑝𝑟𝑜𝑥 𝐸𝑟𝑟𝑜𝑟
• Approx. fractional Error= ∗ 100%
𝐴𝑝𝑝𝑟𝑜𝑥 𝑣𝑎𝑙𝑢𝑒
Iterative Methods: Truncation
𝑥2 𝑥3 𝑥𝑛
• 𝑒𝑥 =1+𝑥+ + + ⋯ +…
2 3! 𝑛!
• Take 𝑥 = 0.5
Also assume answer is 1.648721
1.648721−1.5
• 𝑒 0.5 =1+𝑥 = 1.5 Error= ∗ 100% = 9.02%
1.648721
𝑥2 0.25
• 𝑒 0.5 = 1 + 𝑥 + = 1 + 0.5 + = 1.625, Error= 1.44%
2 2
• After 6 terms
𝑒 0.5 = 1.648697917 and 𝐸𝑟𝑟𝑜𝑟 = 0.00142%
Computer Algorithm
• Many calculations involve loops- iterations to improve answer.
• But how do we stop ?
• Assign a tolerance (or error) and a max number of iterations. You will
exit the loop either if the error becomes smaller than the tolerance.
Or if you exceed the maximum iterations (for safety).
• Question- what to do if you don’t have the exact solution for
comparison ?
Computer Representation of numbers
Integer Representation: 16 bit
1 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1
Sign: 0 =positive, 1=negative
Remaining (magnitude)=: 1 × 20 + 1 × 22 + 1 × 23 + 1 × 25 + 1 × 27 = 1 +
4 + 8 + 32 + 128 = 173.
Final Answer: -173
Range= With 16 bits, we use only 15 bits for the integer.
Largest number is 111111111111111 in binary = 215 − 1 = 32767.
Because of sign, we can go from -32767 to +32767
Floating Point Numbers
Exponent
Mantissa
Sign
Number =sign ⋅ 𝑚𝑏 𝑒
𝑚 = 𝑚𝑎𝑛𝑡𝑖𝑠𝑠𝑎
𝑏 = 𝑏𝑎𝑠𝑒 𝑜𝑓 𝑛𝑢𝑚𝑏𝑒𝑟 𝑠𝑦𝑠𝑡𝑒𝑚 , has to be specified
𝑒 = 𝑒𝑥𝑝𝑜𝑛𝑒𝑛𝑡 (one bit is often kept for sign of exponent)
Eg. 196.72 = 0.19672 × 103 in base 10 system.
Aim- allow both fractions as well as very large numbers to be represented.
Examples
IEEE-754 Single Precision
1 bit sign
8 bit exponent
23 bit fractional part
IEEE-754 Double Precision
1 bit sign
11 bit exponent
52 bit fractional part f
Remember
• There is a limited range of quantities that can be represented.
• If you try to employ numbers outside the range we get overflow and
underflow error
• Even within the range, there are only finite number of quantities that
can be represented.
Linear Equations
Solve 𝐴𝑥 = 𝑏
Direct or Indirect Methods
Direct:
Method Initial form Final Form
Gauss Elimination 𝐴𝑥 = 𝑏 𝑈𝑥 = 𝑐
LU Decomposition 𝐴𝑥 = 𝑏 𝐿𝑈𝑥 = 𝑏
Gauss Jordan method 𝐴𝑥 = 𝑏 𝐼𝑥 = 𝑐
Linear Equations
Allowed Operations:
1. Exchanging two equations (can change the sign of |𝐴|)
2. Multiplying an equation by a nonzero constant (multiplies 𝐴 by
the same constant).
3. Multiplying an equation by a nonzero constant and then subtracting
it from another equation in the set. (Leaves 𝐴 unchanged).
Gauss Elimination-
Elimination (Rows are numbered 1 to n)
For k in range(1, 𝑛 − 1):
# Call the Pivot module
For 𝑖 in range 𝑘 + 1, 𝑛 :
If 𝑎 𝑘, 𝑘 ≠ 0:
𝑙𝑎𝑚 = 𝑎 𝑖, 𝑘 /𝑎[𝑘, 𝑘]
𝑎 𝑖, 𝑘 + 1: 𝑛 = 𝑎 𝑖, 𝑘 + 1: 𝑛 − 𝑙𝑎𝑚 ∗ 𝑎[𝑘, 𝑘 + 1: 𝑛]
𝑏 𝑖 = 𝑏 𝑖 − 𝑙𝑎𝑚 ∗ 𝑏[𝑘]
Remarks
In the previous algorithm,
• We did not calculate 𝑎[𝑖, 𝑘]. This is supposed to become zero due to
operation.
• So, index for column, 𝑗, starts with 𝑘 + 1 and not 𝑘. So in the updated
matrix, the lower triangular part is not explicitly set to 0. If you print
out the matrix after completion of elimination part, it will show all the
lower triangular part coefficients. But we do not use its contents.
Back Substitution- Pseudo code (Not python)
𝑥[𝑛] = 𝑏[𝑛]/𝑎[𝑛, 𝑛]
For 𝑘 in range(𝑛 − 1,1, −1):
𝑥 𝑘 = (𝑏 𝑘 − 𝑑𝑜𝑡(𝑎 𝑘, 𝑘 + 1: 𝑛 , 𝑥[𝑘 + 1: 𝑛]))/𝑎[𝑘, 𝑘]
Operation Count:
Elimination:𝑛3 /3 operations
Substituion:𝑛2 /2
gaussElimin
## module gaussElimin
𝑥 = 𝑔𝑎𝑢𝑠𝑠𝐸𝑙𝑖𝑚𝑖𝑛(𝑎, 𝑏)
## Module
𝑑𝑒𝑓 𝑔𝑎𝑢𝑠𝑠𝐸𝑙𝑖𝑚𝑖𝑛 𝑎, 𝑏 :
#Forward Elimination
# Back Substitution
return x
Partial Pivoting- Pseudocode (not python)
Please check array index: in loop for ii- it has to go till last row
# Select Pivot Row #𝑆𝑤𝑎𝑝 𝑘 𝑤𝑖𝑡ℎ 𝑝
If (𝑝 ≠ 𝑘):
p=k For 𝑗𝑗 = 𝑘, 𝑛
big= 𝑎 𝑘, 𝑘 𝑑𝑢𝑚𝑚𝑦 = 𝑎 𝑝, 𝑗𝑗
For 𝑖𝑖 𝑖𝑛 𝑟𝑎𝑛𝑔𝑒 𝑘 + 1, 𝑛 : 𝑎 𝑝, 𝑗𝑗 = 𝑎 𝑘, 𝑗𝑗
𝑎 𝑘, 𝑗𝑗 = 𝑑𝑢𝑚𝑚𝑦
𝑑𝑢𝑚𝑚𝑦 = |𝑎 𝑖𝑖, 𝑘 | 𝑑𝑢𝑚𝑚𝑦 = 𝑏 𝑝
𝑖𝑓 𝑑ummy > big : 𝑏 𝑝 =𝑏 𝑘
# choose ii as pivot row 𝑏 𝑘 = 𝑑𝑢𝑚𝑚𝑦
𝑝 = 𝑖𝑖
Combining partial pivoting with GE
Stage 1: Forward Elimination
For k= 1, 𝑛 − 1:
# 𝑘 is the pivot row, go from first row to last-1.
# Call pivot module
# Call Elimination module
Stage 2:
Back Substitution
Comments
• Pivoting helps reduce rounding errors; you are less likely to
add/subtract with very small number (or very large) numbers.
• Before any division, ensure that the divisor is not zero
• Finding the determinant of a matrix:
Use forward elimination method to get upper triangular matrix.
Product of all the diagonal elements is the determinant.
For each Row swap, there is a sign flip.
Always ensure that you are not dividing by 0.
Gauss Jordan Method
• Similar to Gauss Elimination. Except that when we eliminate a
coefficient, do it for all rows (except the pivot row).
• After the elimination, we normalize (scale) the row by the diagonal
element.
• So matrix 𝐴 is converted to identity matrix.
• So, 𝐴𝑥 = 𝑏 becomes 𝐼𝑥 = 𝑐, the Solution is 𝑥 = 𝑐
Gauss Jordan- Elimination
Elimination
For k in range(1, 𝑛 − 1):
For 𝑖 in range 1, 𝑛 :
If k ≠ 𝑖:
# Check that a[k,k] is not zero
𝑙𝑎𝑚 = 𝑎 𝑖, 𝑘 /𝑎[𝑘, 𝑘]
𝑎 𝑖, 𝑘: 𝑛 = 𝑎 𝑖, 𝑘: 𝑛 − 𝑙𝑎𝑚 ∗ 𝑎[𝑘, 𝑘: 𝑛]
𝑏 𝑖 = 𝑏 𝑖 − 𝑙𝑎𝑚 ∗ 𝑏[𝑘]
Matrix Inverse
To obtain the inverse of a n × n matrix A :
1.Create the partitioned matrix (𝐴|𝐼)
•, where I is the identity matrix.
•Perform Gauss-Jordan Elimination on the partitioned matrix with
the objective of converting the first part of the matrix to reduced-
row echelon form.
•If done correctly, the resulting partitioned matrix will take the
form (𝐼|𝐴−1 )
•Double-check your work by making sure that 𝐴 ⋅ 𝐴−1 = 𝐼
