Introduction for Numerical Analysis

Chang YANG

School of Mathematics, Harbin Institute of Technology, Harbin, China

Autumn semester

1 / 29
References

R ICHARD L. B URDEN AND J. D OUGLAS FAIRES, Numerical Analysis,

Cengage Learning, 2015.
T IMOTHY S AUER, Numerical Analysis, China Machine Press, 2011.

2 / 29
I. Definition
Numerical analysis is the study of algorithms that use numerical
approximation for the problems of mathematical analysis.

II. Motivations
(i) Experimental study is very expensive. Alternatively, mathematical
models are developed.
(ii) To find the solution of the models: the explicit solution may not exist; or
they exist but they are not appropriate for practical use...
(iii) Propose an approximated solution, calculated by a computer.

III. Develop efficient algorithms

(i) Convergence and stability of numerical method.
(ii) Computational cost.

3 / 29
Outline

1 Error analysis

2 Representation of real number in computer

3 Conditioning, Stability and Complexity

4 Some tips in numerical analysis

4 / 29
Table of contents

1 Error analysis

2 Representation of real number in computer

3 Conditioning, Stability and Complexity

4 Some tips in numerical analysis

5 / 29
Sources of error

(i) Errors due to the model, that can be controlled by a proper choice of the
mathematical model.
(ii) Errors in the data, that can be reduced by enhancing the accuracy in the
measurement of the data themselves.
(iii) Truncations errors, arising from having replaced in the numerical model
limits by operations that involve a finite number of steps.
(iv) Round-off errors, arising from limitation of computer.

6 / 29
Measurement of error

Absolute error and Absolute error bound

Let x be the exact value, x ∗ be an approximation.
The absolute error is E(x ∗ ) = |x − x ∗ |.
Because x is usually unknown, so E(x ∗ ) can not be find exactly.
The absolute error bound is δ(x ∗ ), such that E(x ∗ ) = |x − x ∗ | ≤ δ(x ∗ ).

Example: Let x = π = 3.1415926 . . . , x ∗ = 3.14159. Then we have

E(x ∗ ) = |x − x ∗ | ≤ 0.5 × 10−5 .

Therefore, the absolute error bound δ(x ∗ ) = 0.5 × 10−5 .

7 / 29
Measurement of error

Relative error and Relative error bound

Let x be the exact value, x ∗ be an approximation.
|x−x ∗ | |x−x ∗ |
The relative error is Er (x ∗ ) = |x| or Er (x ∗ ) = |x ∗ | .
Because x is usually unknown, so Er (x ∗ ) can not be find exactly.
|x−x ∗ |
The relative error bound is δr (x ∗ ), such that Er (x ∗ ) = |x ∗ | ≤ δr (x ∗ ).

Example: Let x = π = 3.1415926 . . . , x ∗ = 3.14159. Then we have

|x − x ∗ |
Er (x ∗ ) = ≤ 1.59 × 10−6 .
|x ∗ |

Therefore, the relative error bound δr (x ∗ ) = 1.59 × 10−6 .

8 / 29
Measurement of error

Significant Figure
Let x be the exact value, x ∗ be an approximation.
Assume x ∗ = ±10k × 0.a1 a2 . . . an . . . , with ai ∈ {0, . . . , 9} and a1 6= 0.
1
If |x − x ∗ | ≤ 2 × 10k −n , then x ∗ has n significant figures.

Example: Let x = π = 3.1415926 . . . , x ∗ = 3.14159. Then we have

E(x ∗ ) ≈ 0.000003 ≤ 0.5 × 10−5 .

Therefore, k = 1, n = 6, and x ∗ has 6 significant figures.

Theorem
1
(i) If x ∗ has n significant figures, then Er (x ∗ ) ≤ 2a1 × 101−n .
1
(ii) If Er (x ∗ ) ≤ 2(a1 +1) × 101−n , then x ∗ has at least n significant figures.

9 / 29
Table of contents

1 Error analysis

2 Representation of real number in computer

3 Conditioning, Stability and Complexity

4 Some tips in numerical analysis

10 / 29
Represent a real number with basis b

All the real number x can be represented under the form

x = ±m × be , with b ≥ 2.

where
Fraction m:

m = m1 b−1 + m2 b−2 + . . . , with mi ∈ {0, 1, . . . , b − 1}.

Exponent e:

e = e0 b0 + e1 b1 + · · · + es−1 bs−1 , with s ∈ N.

Computer has finite memory!

The fraction is truncated by the first r numbers.
Maximal value for s.

11 / 29
Standard IEEE (754-1985)

Float double precision: 8 octets (64 bits), b = 2

Fraction: 52 bits

m = 2−1 + m2 2−2 + · · · + m53 2−53 , with mi ∈ {0, 1}.

Exponent: 11 bits, e ∈ [−1021, 1024]

e = e0 20 + e1 21 + · · · + e10 210 − 1022, with ei ∈ {0, 1}.

The values e = −1022 and e = 1025 are reserved to represent 0 and

infinity.
We denote the set of number of Standard IEEE (754-1985) by F.

12 / 29
Exercises

Exercise 1
Compute
the largest number of F: Xmax .
the smallest positive number of F: Xposmin .

Exercise 2
Propose two algorithms to determine a numerical approximation of Xmax and
Xposmin respectively.

13 / 29
Round-off error

Only the elements in F are authorised.

A real number x = ±m × 2e , m = 0.1m2 m3 . . . is rounded off by following
way
rd(x) = sign(x)(0.1m2 . . . m53 )2 × 2e .
If Xposmin < |x| < Xmax , then
The absolute round-off error is
1 −53 e
|x − rd(x)| ≤ 2 2 .
2
The relative round-off error is
x − rd(x) 1 2−53 2e
≤ ≤ 2−53 ' 10−16 .
x 2 |m|2e

Exercise
Compute the smallest number such that rd(x + 0.5) > 0.5.
Refind this number by a numerical algorithm.

14 / 29
Table of contents

1 Error analysis

2 Representation of real number in computer

3 Conditioning, Stability and Complexity

4 Some tips in numerical analysis

15 / 29
Conditioning of a numerical problem

Definition
The conditioning represents the sensibility of result of a numerical problem
with respect of small variations of data.
Well-conditioned: a small variation of data leads a small variation of
result.
Ill-conditioned: a small variation of data leads a large variation of result.

Example: Let f : R → R
Taylor expansion

f (x + δx) = f (x) + f 0 (x)δx + o(δx),

δf := f (x + δx) − f (x).

Conditioning
δf x xf 0 (x)
k= ' .
f δx f (x)

16 / 29
Conditioning of arithmetic operations

Let f ∈ C 2 (Rn ). What is the conditioning of f ?

By Taylor expansion
n
X ∂f
δf = f (x + δx) − f (x) = δxi + O(|δx|2 ).
∂xi
i=1

Conditioning is determined by

∂f xi
ki = .
∂xi f (x)

Exercise
Compute the conditioning of addition and multiplication operations.

17 / 29
Stability of an algorithm

Definition
The stability of an algorithm refers to propagation of errors during
computational steps, to the precision of the obtained results, to the capacity
of algorithm of not amplify much possible gap.

Example: The algorithm to compute the roots of polynomial

p(x) = ax 2 + bx + c based the following formula
√ √
−b + b2 − 4ac −b − b2 − 4ac
x1 = , x2 = .
2a 2a
This algorithm turns out to be unstable in practical. For example,
a = 1, b = −(10−16 + 1), c = 10−16 .

Exercise
Propose a more stable algorithm.

18 / 29
Complexity of an algorithm

Definition
Number of elementary operations carried out by the algorithm (+, ×, sqrt,
power, . . . )
Pn
Example: Evaluation of polynomial p(x) = i=0 ai x i :
Pn
Number of multiplications: i=0 i = n(n+1)
2 .
Number of additions: n.
algorithmic cost: O(n2 ).

Definition
Polynomial cost: O(nk ).
Exponential cost: O(an ).
Factorial cost: O(n!).
Pn
Example: Evaluation of polynomial p(x) = i=0 ai x i :
Horner’s method: p(x) = a0 + x(a1 + x(a2 + x(. . . an ))).
algorithmic cost?
19 / 29
Table of contents

1 Error analysis

2 Representation of real number in computer

3 Conditioning, Stability and Complexity

4 Some tips in numerical analysis

20 / 29
Avoid subtraction of two similar numbers

Example: Let x = 1000, compute

√ √
y= x +1− x.

y = 0.015807437428958.
√ Now, we √
take only four significant figures in
computation, then x + 1 ≈ 31.64, x ≈ 31.62, thus y ≈ y ∗ = 0.02. It rests
only one significant figure.

|y − y ∗ | |0.015807437428958 − 0.02|
Er (y ∗ ) = = = 20.96%.
|y ∗ | 0.02

The relative error is more than 20%!!!

An improvement:
√ √ 1
y= x +1− x=√ √ .
x +1+ x
1
This time, y ∗∗ = x1∗ +x2∗ ≈ 0.01581, which has four significant figures.

21 / 29
Avoid divisor with very small absolute value

Example:
x
z=
y
Let x = 2.7182, y = 0.001, the exact z = 2718.2. In the case the divisor has
a turbulence y ∗ = 0.0011, then

x∗ 2.7182
z∗ = ∗
= ≈ 2471.1.
y 0.0011

the absolute error is E(z ∗ ) = 247.1, the relative error is Er (z ∗ ) = 10%.

22 / 29
Avoid large number ’eat’ small number

Solve the equation

x 2 − (1 + 10−16 )x + 10−16 = 0.

It is easy to find the two exact solutions are x1 = 1, x2 = 10−16 . By the roots
formulae p
1 + 10−16 ± (1 + 10−16 )2 − 4 × 10−16
x1,2 = .
2
Because
p of round-off error, we have rd(1 + 10−16 ) = 1,
rd( (1 + 10−16 )2 − 4 × 10−16 ) = 1, thus x1 = 1, x2 = 0. Obviously, x2 is
wrong. To correct this fault, we recast x2 as follows

10−16
x2 = = 10−16 . (1)
x1

23 / 29
Simplify complexity of an algorithm

Example: To compute x 255 :

(i) compute directly, we need 254 multiplications.
(ii) We recast it as follows

x 255 = x · x 2 · x 4 · x 8 · x 16 · x 32 · x 64 · x 128 .

This time, we only need 14 multiplications.

24 / 29
Simplify complexity of an algorithm

Example: To compute x 255 :

(i) compute directly, we need 254 multiplications.
(ii) We recast it as follows

x 255 = x · x 2 · x 4 · x 8 · x 16 · x 32 · x 64 · x 128 .

This time, we only need 14 multiplications.

25 / 29
Use more stable algorithm
R1 xn
Example: Let In = 0 x+5
dx, then compute I20 .
We notice that
1 1
x n + 5x n−1
Z Z
1 n1 1
In + 5In−1 = dx = x n−1 dx = x | = .
0 x +5 0 n 0 n
1
we thus find a recurrence formula In = n − 5In−1 . Moreover,
R1 1
I0 = 0 x+5 dx = ln(x + 5)|10 = ln 65 .

Therefore, we have the following

algorithm

In = n1 − 5In−1 ,

(A)
I0 = ln 56 .

Unfortunately, algorithm (A) is not

stable.
The exact I20 = 0.007999520844283.

26 / 29
Use more stable algorithm

We can prove the following properties

1 1
< In−1 < , (n > 1).
6n 5n
∗
We thus can use I40 = 12 ( 6×41
1
+ 1
5×41 ) to approximate I40 .

Therefore, we have the following

algorithm
1
− 15 In ,

In−1 = 5n
(B)
I40 .

Algorithm (B) is stable.

27 / 29
Complementary exercises

Exercise
Calculate Z 1
In = e−1 x n ex dx.
0

(i) Prove the recursive formula (forward algorithm)

In = 1 − nIn−1 .

(ii) Take I0 = 1 − e−1 as an initial data, to compute I13 , is the forward

algorithm stable?
(iii) Prove
1 1
< In−1 < .
n+1 n
(iv) Use I26 = 12 ( 27
1 1
+ 28 ) as an initial to compute I13 . Is the backward
1
algorithm In−1 = n (1 − In ) stable?

28 / 29
Complementary exercises

Exercise
How to avoid loss significant figures :
(i) 1 − cos(x), x is near 0.
√ √
(ii) ( 1 + x 2 − 1 − x 2 )−1 , x is near 0.
(iii) x − sin(x), x is near 0.

Exercise
Assume x1∗ = 1.216, x2∗ = 3.654 have 3 significant figures, then find the
relative error bound of x1∗ x2∗ .

29 / 29