Introduction to Numerical Methods

 What are NUMERICAL METHODS?

 Why do we need them?

1
Numerical Methods
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.

2
What do we need?
Basic Needs in the Numerical Methods:
 Practical:
Can be computed in a reasonable amount of time.
 Accurate:
 Good approximate to the true value,
 Information about the approximation error
(Bounds, error order,… ).

3
Outlines of the Course
 Taylor Theorem  Solution of linear
 Number Equations
Representation  Least Squares curve
 Solution of nonlinear fitting
Equations  Solution of ordinary
 Interpolation differential equations
 Numerical  Solution of Partial
Differentiation differential equations
 Numerical Integration

4
Solution of Nonlinear Equations
 Some simple equations can be solved analytically:
x2  4x  3  0
4 4 2  4(1)(3)
Analytic solution roots 
2(1)
x  1 and x  3

 Many other equations have no analytical solution:

x 9  2 x 2  5  0

x  No analytic solution
xe 

5
Methods for Solving Nonlinear Equations

o Bisection Method

o Newton-Raphson Method

o Secant Method

6
Solution of Systems of Linear Equations

x1  x2  3
x1  2 x2  5
We can solve it as :
x1  3  x2 , 3  x 2  2 x2  5
 x2  2, x1  3  2  1
What to do if we have
1000 equations in 1000 unknowns.
7
Cramer’s Rule is Not Practical
Cramer' s Rule can be used to solve the system :
3 1 1 3
5 2 1 5
x1   1, x2  2
1 1 1 1
1 2 1 2

But Cramer' s Rule is not practical for large problems.

To solve N equations with N unknowns, we need (N  1)(N  1)N!
multiplications.
To solve a 30 by 30 system, 2.3 1035 multiplications are needed.
A super computer needs more than 10 20 years to compute this.
8
Methods for Solving Systems of Linear
Equations
o Naive Gaussian Elimination

o Gaussian Elimination with Scaled

Partial Pivoting

o Algorithm for Tri-diagonal

Equations

9
Curve Fitting
 Given a set of data:

x 0 1 2
y 0.5 10.3 21.3

 Select a curve that best fits the data. One

choice is to find the curve so that the sum
of the square of the error is minimized.

10
Interpolation
 Given a set of data:

xi 0 1 2
yi 0.5 10.3 15.3

 Find a polynomial P(x) whose graph

passes through all tabulated points.

yi  P( xi ) if xi is in the table
11
Methods for Curve Fitting
o Least Squares
o Linear Regression
o Nonlinear Least Squares Problems

o Interpolation
o Newton Polynomial Interpolation
o Lagrange Interpolation

12
Integration
 Some functions can be integrated
analytically:

3 3
1 2 9 1
1 xdx  2 x 1  2  2  4
But many functions have no analytical solutions :
a

e
 x2
dx  ?
0

13
Methods for Numerical Integration
o Upper and Lower Sums

o Trapezoid Method

o Romberg Method

o Gauss Quadrature

14
Solution of Ordinary Differential Equations
A solution t o the differenti al equation :
x(t )  3x (t )  3x(t )  0
x (0)  1; x(0)  0
is a function x(t) that satisfies the equations.

* Analytical solutions are availablefor

special cases only.
15
Solution of Partial Differential Equations
Partial Differential Equations are more
difficult to solve than ordinary differential
equations:

 u
2
 u
2
 20
x 2
t 2

u (0, t )  u (1, t )  0, u ( x,0)  sin(x)

16
Summary
Topics Covered in the Course
 Numerical Methods:
Algorithms that are
 Solution of Nonlinear Equations
used to obtain  Solution of Linear Equations
numerical solution of a
 Curve Fitting
mathematical problem.
 We need them when  Least Squares
No analytical solution  Interpolation
exists or it is difficult  Numerical Integration
to obtain it.
 Numerical Differentiation
 Solution of Ordinary Differential
Equations
 Solution of Partial Differential
Equations

17
Number Representation and Accuracy
 Number Representation
 Normalized Floating Point Representation
 Significant Digits
 Accuracy and Precision
 Rounding and Chopping

18
Representing Real Numbers
 You are familiar with the decimal system:

312.45  3 102  1101  2 100  4 101  5 102

 Decimal System: Base = 10 , Digits (0,1,…,9)

 Standard Representations:

 3 1 2 . 4 5
sign integral fraction
part part
19
Normalized Floating Point Representation
 Normalized Floating Point Representation:

 d . f1 f 2 f 3 f 4  10 n
sign mantissa exponent

d  0,  n : signed exponent
 Scientific Notation: Exactly one non-zero digit appears
before decimal point.
 Advantage: Efficient in representing very small or very
large numbers.

20
Binary System

 Binary System: Base = 2, Digits {0,1}

 1. f1 f 2 f 3 f 4  2  n
sign mantissa signed exponent

(1.101)2  (1  1  21  0  22  1  23 )10  (1.625)10

21
Fact
 Numbers that have a finite expansion in one numbering
system may have an infinite expansion in another
numbering system:

(1.1)10  (1.000110011001100...) 2
 You can never represent 1.1 exactly in binary system.

22
IEEE 754 Floating-Point Standard
 Single Precision (32-bit representation)
 1-bit Sign + 8-bit Exponent + 23-bit Fraction

S Exponent8 Fraction23

 Double Precision (64-bit representation)

 1-bit Sign + 11-bit Exponent + 52-bit Fraction

S Exponent11 Fraction52
(continued)

23
Significant Digits

 Significant digits are those digits that can be

used with confidence.

 Single-Precision: 7 Significant Digits

1.175494… × 10-38 to 3.402823… × 1038

 Double-Precision: 15 Significant Digits

2.2250738… × 10-308 to 1.7976931… × 10308

24
Remarks

 Numbers that can be exactly represented are called

machine numbers.

 Difference between machine numbers is not uniform

 Sum of machine numbers is not necessarily a machine

number

25
Calculator Example
 Suppose you want to compute:
3.578 * 2.139
using a calculator with two-digit fractions

3.57 * 2.13 = 7.60

True answer: 7.653342

26
Significant Digits - Example
48.9

27
Accuracy and Precision

 Accuracy is related to the closeness to the true

value.

 Precision is related to the closeness to other

estimated values.

28
29
Rounding and Chopping

 Rounding: Replace the number by the nearest

machine number.

 Chopping: Throw all extra digits.

30
Rounding and Chopping

31
Error Definitions – True Error
Can be computed if the true value is known:

Absolute True Error

Et  true value  approximation
Absolute Percent Relative Error
true value  approximation
t  *100
true value

32
Error Definitions – Estimated Error
When the true value is not known:

Estimated Absolute Error

Ea  current estimate  previous estimate
Estimated Absolute Percent Relative Error
current estimate  previous estimate
a  *100
current estimate

33
Notation
We say that the estimate is correct to n
decimal digits if:
n
Error  10

We say that the estimate is correct to n

decimal digits rounded if:
1 n
Error   10
2

34
Summary
 Number Representation
Numbers that have a finite expansion in one numbering system
may have an infinite expansion in another numbering system.

 Normalized Floating Point Representation

 Efficient in representing very small or very large numbers,
 Difference between machine numbers is not uniform,
 Representation error depends on the number of bits used in
the mantissa.

35
 Taylor Theorem

 Motivation
 Taylor Theorem
 Examples

36
Motivation
 We can easily compute expressions like:
3 10 2
2( x  4)
But, How do you compute 4.1, sin(0.6) ?

Can we use the definition

b
a
to compute sin(0.6)?
0.6
Is this a practical way?

37
Remark

 In this course, all angles are assumed to

be in radian unless you are told otherwise.

38
Taylor Series
The Taylor series expansion of f ( x ) about a :
f ( 2) (a ) f ( 3) ( a )
f (a )  f (a ) ( x  a ) 
'
( x  a) 
2
( x  a ) 3  ...
2! 3!
or

1 (k )
Taylor Series   k!
f (a ) ( x  a )k
k 0
If the series converge, we can write :
∞
1 (k )
f ( x)  ∑ k!
f (a ) ( x  a )k
k 0

39
Maclaurin Series
 Maclaurin series is a special case of Taylor
series with the center of expansion a = 0.
The Maclauri n series expansion of f ( x ) :
( 2) ( 3)
f ( 0 ) f ( 0) 3
f ( 0)  f ( 0) x 
'
x 
2
x  ...
2! 3!
If the series converge, we can write :
∞
1 (k )
f ( x)  ∑ k!
f ( 0) x k
k 0
40
Maclaurin Series – Example 1
Obtain Maclauri n series expansion of f ( x )  e x

f ( x)  e x f ( 0)  1
f ' ( x)  e x f ' ( 0)  1
f ( 2) ( x )  e x f ( 2 ) ( 0)  1
f (k ) ( x)  e x f ( k ) (0)  1 for k  1
∞ ∞
1 (k ) xk x2 x3
ex  ∑ k!
f ( 0) x  ∑
k
k!
 1 x 
2!

3!
 ...
k 0 k 0
The series converges for x  ∞.
41
Taylor Series
3

Example 1
2.5
exp(x)
1+x+0.5x 2
2

1+x

1.5

1
1

0.5

0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

42
Maclaurin Series – Example 2
Obtain Maclauri n series expansion of f ( x )  sin( x ) :
f ( x )  sin( x ) f ( 0)  0
f ' ( x )  cos( x ) f ' ( 0)  1
f ( 2 ) ( x )   sin( x ) f ( 2 ) ( 0)  0
f ( 3) ( x )   cos( x ) f ( 3) (0)  1
∞
f ( k ) ( 0) k x3 x5 x7
sin( x )  ∑ x  x     ....
k 0
k! 3! 5! 7!
The series converges for x  ∞.
43
4

3
x

1 x-x 3/3!+x 5/5!

0 sin(x)

-1
x-x 3/3!

-2

-3

-4
-4 -3 -2 -1 0 1 2 3 4

44
Maclaurin Series – Example 3
Obtain Maclaurin series expansion of : f ( x)  cos( x)
f ( x )  cos( x ) f ( 0)  1
f ' ( x )   sin( x ) f ' ( 0)  0
f ( 2 ) ( x )   cos( x ) f ( 2 ) ( 0 )  1
f ( 3) ( x )  sin( x ) f ( 3) (0)  0
∞
f ( k ) ( 0) x 2
x 4
x 6
cos( x )  ∑ ( x ) k  1     ....
k 0
k! 2! 4! 6!
The series converges for x  ∞.
45
Maclaurin Series – Example 4
1
Obtain Maclauri n series expansion of f(x) 
1 x
1
f ( x)  f ( 0)  1
1 x
1
f ' ( x)  f ' ( 0)  1
1  x 2
2
f ( x) 
( 2)
f ( 2 ) ( 0)  2
1  x 3
6
f ( 3) ( x )  f ( 3) (0)  6
1  x 4
1
Maclaurin Series Expansion of :  1  x  x 2  x 3  ...
1 x
Series converges for | x |  1
46
Example 4 - Remarks

 Can we apply the series for x≥1??

 How many terms are needed to get a good

approximation???

These questions will be answered using

Taylor’s Theorem.

47
Taylor Series – Example 5
1
Obtain Taylor series expansion of f(x)  at a  1
x
1
f ( x)  f (1)  1
x
1
f ' ( x)  2 f ' (1)  1
x
2
f ( 2) ( x )  3 f ( 2 ) (1)  2
x
6
f ( 3) ( x )  4 f ( 3) (1)  6
x
Taylor Series Expansion ( a  1) : 1  ( x  1)  ( x  1) 2  ( x  1) 3  ...
48
Taylor Series – Example 6
Obtain Taylor series expansion of f(x)  ln( x ) at ( a  1)

1 1 2
f ( x )  ln( x ) , f ' ( x )  , f ( x )  2 , f ( x )  3
( 2) ( 3)
x x x
f (1)  0, f ' (1)  1, f ( 2 ) (1)  1 f ( 3) (1)  2

1 2 1
Taylor Series Expansion : ( x  1)  ( x  1)  ( x  1) 3  ...
2 3

49
Convergence of Taylor Series

 The Taylor series converges fast (few terms

are needed) when x is near the point of
expansion. If |x-a| is large then more terms
are needed to get a good approximation.

50
Taylor’s Theorem
If a function f ( x ) possesses derivative s of orders 1, 2, ..., ( n  1)
on an interval containing a and x then the value of f ( x ) is given by :

(n+1) terms Truncated

Taylor Series
n
f ( k ) (a )
f ( x)  ∑ k!
( x  a)k  Rn
k 0
Remainder
where :
f ( n 1) ( )
Rn  ( x  a ) n 1 and  is between a and x.
( n  1)!
51
Taylor’s Theorem
We can apply Taylor' s theorem for :
1
f(x)  with the point of expansion a  0 if | x |  1.
1 x

If x  1, then the function and its

derivative s are not defined.
 Taylor Theorem is not applicable .

52
Error Term

To get an idea about the approximat ion error,

we can derive an upper bound on :
( n 1)
f ( )
Rn  ( x  a ) n 1
( n  1)!
for all values of  between a and x.

53
Error Term - Example
How large is the error if we replaced f ( x )  e by x

the first 4 terms ( n  3) of its Taylor series expansion

at a  0 when x  0.2 ?
f (n) ( x)  e x f ( n ) ( ) ≤ e 0.2 for n ≥ 1
f ( n 1) ( )
Rn  ( x  a ) n 1
( n  1)!
e 0.2 n 1
Rn  0 .2   R3  8.14268E  05
( n  1)!
54
Alternative form of Taylor’s Theorem
Let f ( x ) have derivative s of orders 1, 2, ..., ( n  1)
on an interval containing x and x  h then :
n (k )
f ( x) k
f ( x  h)   k!
h  Rn ( h  step size)
k 0

f ( n 1) ( ) n 1
Rn  h where  is between x and x  h
( n  1)!

55
Taylor’s Theorem – Alternative forms
( n 1)
n
f ( k ) (a ) f ( )
f ( x)   ( x  a) 
k
( x  a ) n 1
k 0 k! ( n  1)!
where  is between a and x.

a  x, x  x  h

n
f ( k ) ( x ) k f ( n 1) ( ) n 1
f ( x  h)   h  h
k 0 k! ( n  1)!
where  is between x and x  h.
56
Mean Value Theorem
If f ( x ) is a continuous function on a closed interval [a , b]
and its derivative is defined on the open interval ( a , b)
then there exists ξ  ( a , b)
f(b)  f(a)
f ' (ξ ) 
ba
Proof : Use Taylor' s Theorem for n  0, x  a , x  h  b
f(b)  f(a)  f ' (ξ ) (b  a )

57
Alternating Series Theorem
Consider the alternating series :
S  a1  a2  a3  a4  
 a  a  a  a  The series converges
 1 2 3 4

If  and then  and
 lim a  0  S  S n  an 1
 n n 

S n : Partial sum (sum of the first n terms)

an 1 : First omitted term
58
Alternating Series – Example
1 1 1
sin(1) can be computed using : sin(1)  1     
3! 5! 7!
This is a convergent alternating series since :
a1  a2  a3  a4   and lim an  0
n 

Then :
 1 1
sin(1)  1   
 3!  5!
 1 1 1
sin(1)  1    
 3! 5!  7!
59
Example 7

Obtain the Taylor series expansion

of f ( x )  e 2 x 1 at a  0.5 (the center of expansion)
How large can the error be when ( n  1) terms are used
to approximat e e 2 x 1 with x  1 ?

60
Example 7 – Taylor Series
Obtain Taylor series expansion of f ( x )  e 2 x 1 , a  0.5
f ( x)  e 2 x 1 f (0.5)  e 2
f ' ( x)  2e 2 x 1 f ' (0.5)  2e 2
f ( 2) ( x)  4e 2 x 1 f ( 2) (0.5)  4e 2
f ( k ) ( x)  2 k e 2 x 1 f ( k ) (0.5)  2 k e 2
∞
f ( k ) (0.5)
e 2 x 1
 ∑ k!
( x  0.5) k
k 0

( x  0.5) 2 k 2 ( x  0.5)
k
 e  2e ( x  0.5)  4e
2 2 2
 ...  2 e  ...
2! k!
61
Example 7 – Error Term
f ( k ) ( x)  2 k e 2 x 1
f ( n 1) ( )
Error  ( x  0.5) n 1
(n  1)!
(1  0.5) n 1
Error  2 n 1 e 2 1
(n  1)!
n 1
( 0. 5)
Error  2 n 1 max e 2 1
(n  1)!  [ 0.5,1]
e3
Error 
(n  1)!
CISE301_Topic1 62