Optimization Techniques The Fibonacci sequence is found in many different disciplines

Numerical Optimization Techniques and in nature. For example, it has been used to describe plant
life growth, estimate population increases over a specified
timeframe, model virus breakouts, and predict the behavior
▪ Fibonacci Method of financial markets.
• Fibonacci method can be used to find the minimum of a function
even if the function is no differentiable and not continuous.
• Elimination Methods 𝑓𝑥
𝑓𝑥
• This method makes use of the sequence of Fibonacci numbers, *𝐹𝑛+, 𝑓𝑥 𝑓𝑥
for placing the experiments.
• These numbers are defined as: 𝐹0 = 𝐹1 = 1, 𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2
which yield the sequence 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, . . . .
• The method is based on evaluating the objective function at
𝑎
different points in the interval. b
Optimization Techniques
Numerical Optimization Techniques

▪ Fibonacci Method
• Fibonacci method can be used to find the minimum of a function
even if the function is no differentiable and not continuous.

• Elimination Methods
• This method makes use of the sequence of Fibonacci numbers, *𝐹𝑛+,
for placing the experiments.

• These numbers are defined as: 𝐹0 = 𝐹1 = 1, 𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2

which yield the sequence 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, . . . .
• The method is based on evaluating the objective function at
𝑎
different points in the interval. b
• Narrow the range until the minimizer is “boxed in” with sufficient
accuracy.
Optimization Techniques
Numerical Optimization Techniques

▪ Fibonacci method has the following limitations

• The initial interval of uncertainty, by 𝑎 ≤ 𝑥 ≤ 𝑏 , has to be known.
• The function being optimized has to be unimodal.
• The total number of experiments 𝑛 has to be determined.

𝑎 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 1: Define 𝐿0 = 𝑏 − 𝑎 as the initial interval of uncertainty.
𝐹
• Step 2: Define: 𝐿∗2 = 𝑛−2 𝐿0 and place the first two experiments at points 𝑥 1 and 𝑥2 .
𝐹𝑛
This gives: 𝑥1 = 𝑎 + 𝐿∗ , 𝑥2 = 𝑏 − 𝐿∗
2 2

• Step 3: Find 𝑓 𝑥1 and 𝑓(𝑥2)

𝑓 𝑥1 𝑓 𝑥2
𝐿∗2 𝐿∗2

𝑎 𝑥1 𝑥2 b

𝐿0
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 < 𝑓 𝑥2 𝑓 𝑥2

𝑓 𝑥1

𝑎 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 < 𝑓 𝑥2 𝑓 𝑥2

❖ Delete 𝑥2, 𝑏
𝑓 𝑥1
❖ 𝑥𝑗 = 𝑎 + (𝑥2 − 𝑥1)

𝑎 𝑥1 𝑥2
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 >𝑓 𝑥2 𝑓 𝑥1

𝑓 𝑥2

𝑎 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 >𝑓 𝑥2 𝑓 𝑥1

❖ Delete 𝑎, 𝑥1
𝑓 𝑥2
❖ 𝑥𝑗 = 𝑥1 + (𝑏 − 𝑥2)

𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 =𝑓 𝑥2
𝐹𝑛−𝑗 𝑓 𝑥1 𝑓 𝑥2
❖ 𝑥𝑗 = 𝑎𝑗 + (𝑏 − 𝑎) 𝑗
𝐹𝑛

a 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 5: After n experiments
✓ Find the final interval 𝑥, 𝑥 + 𝐿𝑛 that minimizes is located

𝑎 𝑥 𝑥 + 𝐿𝑛 b

𝐿0
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 5: After n experiments
✓ Find the final interval 𝑥, 𝑥 + 𝐿𝑛 that minimizes is located

𝐿𝑛
✓ Calculate the ratio of the final to the initial interval of uncertainty:
𝐿0

𝑎 𝑥 𝑥 + 𝐿𝑛 b

𝐿0
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐹𝑛−2 2
𝑛 = 4, 𝑎 = −1, 𝑏 = 1, 𝐿 0 = 𝑏 − 𝑎 = 1 − −1 = 2, 𝐿∗2 = 𝐿0 = × 2 = 0.8
𝐹𝑛 5

𝑥1 = 𝑎 + 𝐿∗2 = −1 + 0.8 = −0.2 → 𝑓 𝑥1 = 1.0174

𝑥2 = 𝑏 − 𝐿∗ = 1 − 0.8 = 0.2 → 𝑓 𝑥 = 1.0227 𝑓 𝑥2 = 1.0227
2 2
𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎 = −1 𝑥1 = −0.2 𝑥2 = 0.2 b=1

𝐿0 = 2
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐹𝑛−2 2
𝑛 = 4, 𝑎 = −1, 𝑏 = 1, 𝐿 0 = 𝑏 − 𝑎 = 1 − −1 = 2, 𝐿∗2 = 𝐿0 = × 2 = 0.8
𝐹𝑛 5

𝑥1 = 𝑎 + 𝐿∗2 = −1 + 0.8 = −0.2 → 𝑓 𝑥1 = 1.0174

𝑥2 = 𝑏 − 𝐿∗ = 1 − 0.8 = 0.2 → 𝑓 𝑥 = 1.0227 𝑓 𝑥2 = 1.0227
2 2
𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎 = −1 𝑥1 = −0.2 𝑥2 = 0.2
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
𝑥1 = −0.2 → 𝑓 𝑥1 = 1.0174

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = −1 + 0.2 − −0.2 = −0.6

𝑓 𝑥3 = 1.1135

𝑓 𝑥3 = 1.1135 𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥3 → Delete the interval ,𝑎, 𝑥3-

𝑎 = −1 𝑥3 = −0.6 𝑥1 = −0.2 𝑥2 = 0.2

 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
𝑥1 = −0.2 → 𝑓 𝑥1 = 1.0174

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = −1 + 0.2 − −0.2 = −0.6

𝑓 𝑥3 = 1.1135

𝑓 𝑥3 = 1.1135 𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥3 → Delete the interval ,𝑎, 𝑥3-

𝑥3 = −0.6 𝑥1 = −0.2 𝑥2 = 0.2

 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
𝑥1 = −0.2 → 𝑓 𝑥1 = 1.0174
𝐿3 = 0.8
𝑥4 = 𝑥3 + 𝑥2 − 𝑥1 = −0.6 + 0.2 − −0.2 = −0.2 𝑆𝑖𝑛𝑐𝑒 𝑥4 = 𝑥1

𝐹𝑛−𝑗 𝐹4−4 1
𝑥4 = 𝑎 𝑗 + (𝑏 − 𝑎) 𝑗 = 𝑥3 + 𝑥2 − 𝑥3 = −0.6 + 0.2 − −0.6 = −0.6 + 0.16 = −0.44
𝐹𝑛 𝐹4 5

𝑥3 = −0.6 𝑥1 = −0.2 𝑥2 = 0.2

𝐿3 = 0.8
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
𝑥1 = −0.2 → 𝑓 𝑥1 = 1.0174
𝐿3 = 0.8
𝑥4 = −0.44 → 𝑓 𝑥4 = 1.070
𝑓 𝑥4 = 1.070

𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥4 → Delete the interval ,𝑥 3 , 𝑥4-

𝑥3 = −0.6 𝑥1 = −0.2 𝑥2 = 0.2

𝑥4 = −0.44
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
𝑥1 = −0.2 → 𝑓 𝑥1 = 1.0174
𝐿3 = 0.8
𝑥4 = −0.44 → 𝑓 𝑥4 = 1.070
𝑓 𝑥4 = 1.070

𝑓 𝑥1 = 1.0174

𝑓 𝑥1 < 𝑓 𝑥4 → Delete the interval ,𝑥 3 , 𝑥4-

𝑥4 = −0.44 𝑥1 = −0.2 𝑥2 = 0.2

 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿0 = 2
𝐿2 = 1.2
The value of that minimizes is located in the interval [-0.44, 0.2] 𝐿3 = 0.8
𝐿4 = 0.64

𝑥4 = −0.44 𝑥2 = 0.2

𝐿4 = 0.64
Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Fibonacci method
using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution.
𝐿4 0.64
The ratio of the final to the initial interval of uncertainty: = = 0.32 𝐿0 = 2
𝐿0 2
𝐿4 = 0.64
1.9

1.8 𝐿0 = 2
1.7

1.6

1.5

1.4 𝐿4 = 0.64
1.3

1.2

1.1

1
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
𝑎 = −1 𝑥4 = −0.44 𝑥2 = 0.2 b=1
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐹𝑛−2 3
𝑛 = 5, 𝑎 = 0, 𝑏 = 3, 𝐿 0 = 𝑏 − 𝑎 = 3 − 0 = 3, 𝐿∗2 = 𝐿0 = × 3 = 1.125
𝐹𝑛 8

𝑥1 = 𝑎 + 𝐿∗2 = 0 + 1.125 = 1.125 → 𝑓 𝑥1 = −0.2124 𝑓 𝑥2 = −0.1132

𝑥2 = 𝑏 − 𝐿∗2 = 3 − 1.125 = 1.875 → 𝑓 𝑥2 = −0.1132
𝑓 𝑥1 = −0.2124

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎=0 𝑥1 = 1.125 𝑥2 = 1.875 b=3

𝐿0 = 3
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐹𝑛−2 3
𝑛 = 5, 𝑎 = 0, 𝑏 = 3, 𝐿 0 = 𝑏 − 𝑎 = 3 − 0 = 3, 𝐿∗2 = 𝐿0 = × 3 = 1.125
𝐹𝑛 8

𝑥1 = 𝑎 + 𝐿∗2 = 0 + 1.125 = 1.125 → 𝑓 𝑥1 = −0.2124 𝑓 𝑥2 = −0.1132

𝑥2 = 𝑏 − 𝐿∗2 = 3 − 1.125 = 1.875 → 𝑓 𝑥2 = −0.1132
𝑓 𝑥1 = −0.2124

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎=0 𝑥1 = 1.125 𝑥2 = 1.875

 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥1 = 1.125 → 𝑓 𝑥1 = −0.2124

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = 0 + 1.875 − 1.125 = 0.75 𝑓 𝑥1 = −0.2124

𝑓 𝑥3 = −0.2821 𝑓 𝑥3 = −0.2821

𝑓 𝑥1 > 𝑓 𝑥3 → Delete the interval ,𝑥 1 , 𝑥2-

𝑎=0 𝑥3 = 0.75 𝑥1 = 1.125 𝑥2 = 1.875
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥1 = 1.125 → 𝑓 𝑥1 = −0.2124

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = 0 + 1.875 − 1.125 = 0.75 𝑓 𝑥1 = −0.2124

𝑓 𝑥3 = −0.2821 𝑓 𝑥3 = −0.2821

𝑓 𝑥1 > 𝑓 𝑥3 → Delete the interval ,𝑥 1 , 𝑥2-

𝑎=0 𝑥3 = 0.75 𝑥1 = 1.125
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥3 = 0.75 → 𝑓 𝑥3 = −0.2821
𝐿3 = 1.125
𝑥4 = 𝑎 + 𝑥1 − 𝑥3 = 0 + 1.125 − 0.75 = 0.375 𝑓 𝑥3 = −0.2821

𝑓 𝑥4 = −0.303
𝑓 𝑥4 = −0.303

𝑓 𝑥4 < 𝑓 𝑥3 → Delete the interval ,𝑥 3 , 𝑥1-

𝑎=0 𝑥4 = 0.375 𝑥3 = 0.75 𝑥1 = 1.125
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥3 = 0.75 → 𝑓 𝑥3 = −0.2821
𝐿3 = 1.125
𝑥4 = 𝑎 + 𝑥1 − 𝑥3 = 0 + 1.125 − 0.75 = 0.375 𝑓 𝑥3 = −0.2821

𝑓 𝑥4 = −0.303
𝑓 𝑥4 = −0.303

𝑓 𝑥4 < 𝑓 𝑥3 → Delete the interval ,𝑥 3 , 𝑥1-

𝑎=0 𝑥4 = 0.375 𝑥3 = 0.75
Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥4 = 0.375 → 𝑓 𝑥4 = −0.303 𝐿3 = 1.125
𝐿4 = 0.75
𝑥5 = 𝑎 + 𝑥3 − 𝑥4 = 0 + 0.75 − 0.375 = 0.375 𝑆𝑖𝑛𝑐𝑒 𝑥5 = 𝑥4

𝑎=0 𝑥4 = 0.375 𝑥3 = 0.75

𝐿4 = 0.75
Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥4 = 0.375 → 𝑓 𝑥4 = −0.303 𝐿3 = 1.125
𝐿4 = 0.75
𝐹𝑛−𝑗 𝐹5−5 1
𝑥5 = 𝑎 + (𝑏 − 𝑎) 𝑗 = 𝑎 + 𝑥3 − 𝑎 = 0 + × 0.75 − 0 = 0.094
𝐹𝑛 𝐹5 8

𝑎=0 𝑥4 = 0.375 𝑥3 = 0.75

𝐿4 = 0.75
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥4 = 0.375 → 𝑓 𝑥4 = −0.303 𝑓 𝑥5 = −0.1835
𝐿3 = 1.125
𝐿4 = 0.75
𝑥5 = 0.094 → 𝑓 𝑥5 = −0.1835 𝑓 𝑥4 = −0.303

𝑓 𝑥5 > 𝑓 𝑥4 → Delete the interval ,𝑎, 𝑥5-

𝑎=0 𝑥5 = 0.094 𝑥4 = 0.375 𝑥3 = 0.75
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
𝑥4 = 0.375 → 𝑓 𝑥4 = −0.303 𝑓 𝑥5 = −0.1835
𝐿3 = 1.125
𝐿4 = 0.75
𝑥5 = 0.094 → 𝑓 𝑥5 = −0.1835 𝑓 𝑥4 = −0.303

𝑓 𝑥5 > 𝑓 𝑥4 → Delete the interval ,𝑎, 𝑥5-

𝑥5 = 0.094 𝑥4 = 0.375 𝑥3 = 0.75
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿2 = 1.875
The value of that minimizes is located in the interval [0.094, 0.75] 𝐿3 = 1.125
𝐿4 = 0.75
𝐿5 = 0.656

𝑥5 = 0.094 𝑥4 = 0.375 𝑥3 = 0.75

𝐿5 = 0.656
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [0,3] by using Fibonacci method
using n = 5?
0.75 −1
1
𝑓 𝑥 = 0.65 − − 0.65𝑥 𝑡𝑎𝑛
1 + 𝑥2 𝑥
▪ Solution.
𝐿0 = 3
𝐿5 0.656
The ratio of the final to the initial interval of uncertainty: = = 0.21867 𝐿5 = 0.656
𝐿0 3
0

-0.05 𝐿0 = 3
-0.1

-0.15

-0.2
𝐿5 = 0.656
-0.25

-0.3

-0.35
𝑎=0 𝑥5 = 0.094 𝑥3 = 0.75 b=3
0 0.5 1 1.5 2 2.5 3
Optimization Techniques
Numerical Optimization Techniques

▪ Golden Section Method:

• Golden section method can be used to find the minimum of a
function even if the function is no differentiable and not
continuous. 𝑓𝑥
𝑓𝑥
• Elimination Methods 𝑓𝑥 𝑓𝑥
• This method makes use of the golden ratio

• Golden ratio is define as: 𝑎+𝑏 = 𝑎 = 𝛾 (1.618)

𝑎 𝑏
• The method is based on evaluating the objective function at
different points in the interval. 𝑎
b
Optimization Techniques
Numerical Optimization Techniques

▪ Golden Section Method:

• Golden section method can be used to find the minimum of a
function even if the function is no differentiable and not
continuous.

• Elimination Methods
• This method makes use of the golden ratio

• Golden ratio is define as: 𝑎+𝑏 = 𝑎 = 𝛾 (1.618)

𝑎 𝑏
• The method is based on evaluating the objective function at
different points in the interval. 𝑎
b
• Narrow the range progressively until the minimizer is “boxed
in” with sufficient accuracy.
Optimization Techniques
Numerical Optimization Techniques

▪ Golden Section Method has the following limitations

• The initial interval of uncertainty, by 𝑎 ≤ 𝑥 ≤ 𝑏 , has
to be known.

• The function being optimized has to be unimodal.

𝑎 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 1: define 𝐿0 = 𝑏 − 𝑎 as the initial interval of uncertainty.
• Step 2: find the location of 𝑥1 𝑎𝑛𝑑 𝑥2 by:
𝐿0 𝐿0
𝐿∗2 = = = 0.382𝐿 0
𝛾2 1.6182

This gives: 𝑥1 = 𝑎 + 𝐿∗ = 𝑎 + 0.382𝐿0 , 𝑥2 = 𝑏 − 𝐿∗ = 𝑏 − 0.382𝐿0

2 2

• Step 3: Find 𝑓 𝑥1 and 𝑓(𝑥2),

𝑓 𝑥1 𝑓 𝑥2
𝐿∗2 𝐿∗2

𝑎 𝑥1 𝑥2 b

𝐿0
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 < 𝑓 𝑥2 𝑓 𝑥2

𝑓 𝑥1

𝑎 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 < 𝑓 𝑥2 𝑓 𝑥2

❖ Delete 𝑥2, 𝑏
𝑓 𝑥1
❖ 𝑥𝑗 = 𝑎 + (𝑥2 − 𝑥1)

𝑎 𝑥1 𝑥2
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 >𝑓 𝑥2 𝑓 𝑥1

𝑓 𝑥2

𝑎 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 >𝑓 𝑥2 𝑓 𝑥1

❖ Delete 𝑎, 𝑥1
𝑓 𝑥2
❖ 𝑥𝑗 = 𝑥1 + (𝑏 − 𝑥2)

𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 4: Compare 𝑓 𝑥1 and 𝑓(𝑥2)
✓ If 𝑓 𝑥1 =𝑓 𝑥2
(𝑏−𝑎) 𝑗 𝑓 𝑥1 𝑓 𝑥2
❖ 𝑥𝑗 = 𝑎𝑗 + 𝛾𝑗

a 𝑥1 𝑥2 b
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 5: After n experiments
✓ Find the final interval 𝑥, 𝑥 + 𝐿𝑛 that minimizes is located

𝑎 𝑥 𝑥 + 𝐿𝑛 b

𝐿0
Optimization Techniques
Numerical Optimization Techniques

▪ Procedure
• Step 5: After n experiments
✓ Find the final interval 𝑥, 𝑥 + 𝐿𝑛 that minimizes is located

𝐿𝑛
✓ Calculate the ratio of the final to the initial interval of uncertainty:
𝐿0

𝑎 𝑥 𝑥 + 𝐿𝑛 b

𝐿0
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2
𝑛 = 4, 𝑎 = −1, 𝑏 = 1, 𝐿0 = 𝑏 − 𝑎 = 1 − −1 = 2, 𝐿∗2 = 0.382 × 𝐿0 = 0.382 × 2 = 0.764

𝑥1 = 𝑎 + 𝐿∗2 = −1 + 0.764 = −0.236 → 𝑓 𝑥1 = 1.0236

𝑥2 = 𝑏 − 𝐿∗2 = 1 − 0.764 = 0.236 → 𝑓 𝑥2 = 1.0324

𝑎 = −1 𝑥1 = −0.236 𝑥2 = 0.236 b=1

𝐿0 = 2
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2
𝑛 = 4, 𝑎 = −1, 𝑏 = 1, 𝐿0 = 𝑏 − 𝑎 = 1 − −1 = 2, 𝐿∗2 = 0.382 × 𝐿0 = 0.382 × 2 = 0.764

𝑥1 = 𝑎 + 𝐿∗2 = −1 + 0.764 = −0.236 → 𝑓 𝑥1 = 1.0236

𝑥2 = 𝑏 − 𝐿∗ = 1 − 0.764 = 0.236 → 𝑓 𝑥 = 1.0324 𝑓 𝑥2 = 1.0324
2 2
𝑓 𝑥1 = 1.0236

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎 = −1 𝑥1 = −0.236 𝑥2 = 0.236 b=1

𝐿0 = 2
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2
𝑛 = 4, 𝑎 = −1, 𝑏 = 1, 𝐿0 = 𝑏 − 𝑎 = 1 − −1 = 2, 𝐿∗2 = 0.382 × 𝐿0 = 0.382 × 2 = 0.764

𝑥1 = 𝑎 + 𝐿∗2 = −1 + 0.764 = −0.236 → 𝑓 𝑥1 = 1.0236

𝑥2 = 𝑏 − 𝐿∗ = 1 − 0.764 = 0.236 → 𝑓 𝑥 = 1.0324 𝑓 𝑥2 = 1.0324
2 2
𝑓 𝑥1 = 1.0236

𝑓 𝑥1 < 𝑓 𝑥2 → Delete the interval ,𝑥 2 , 𝑏-

𝑎 = −1 𝑥1 = −0.236 𝑥2 = 0.236
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236

𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236
𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = −1 + 0.236 − −0.236 = −0.528

𝑎 = −1 𝑥3 = −0.528 𝑥2 = 0.236
𝑥1 = −0.236
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236

𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236 𝑓 𝑥3 = 1.0936

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = −1 + 0.236 − −0.236 = −0.528
𝑓 𝑥1 = 1.0236
𝑓 𝑥3 = 1.0936

𝑓 𝑥3 > 𝑓 𝑥1 → Delete the interval ,𝑎, 𝑥3-

𝑎 = −1 𝑥3 = −0.528 𝑥2 = 0.236
𝑥1 = −0.236
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236

𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236 𝑓 𝑥3 = 1.0936

𝑥3 = 𝑎 + 𝑥2 − 𝑥1 = −1 + 0.236 − −0.236 = −0.528
𝑓 𝑥1 = 1.0236
𝑓 𝑥3 = 1.0936

𝑓 𝑥3 > 𝑓 𝑥1 → Delete the interval ,𝑎, 𝑥3-

𝑥3 = −0.528 𝑥1 = −0.236 𝑥2 = 0.236
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236
𝐿3 = 0.764
𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236
𝑥4 = 𝑥3 + 𝑥2 − 𝑥1 = −0.528 + 0.236 − −0.236 = −0.056

𝑥3 = −0.528 𝑥2 = 0.236
𝑥1 = −0.236 𝑥4 = −0.056
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236
𝐿3 = 0.764
𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236 𝑓 𝑥1 = 1.0236

𝑥4 = 𝑥3 + 𝑥2 − 𝑥1 = −0.528 + 0.236 − −0.236 = −0.056

𝑓 𝑥4 = 1.0015
𝑓 𝑥4 = 1.0015

𝑓 𝑥4 < 𝑓 𝑥1 → Delete the interval ,𝑥 3 , 𝑥1-

𝑥3 = −0.528 𝑥2 = 0.236
𝑥1 = −0.236 𝑥4 = −0.056
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236
𝐿3 = 0.764
𝑥1 = −0.236 → 𝑓 𝑥1 = 1.0236 𝑓 𝑥1 = 1.0236

𝑥4 = 𝑥3 + 𝑥2 − 𝑥1 = −0.528 + 0.236 − −0.236 = −0.056

𝑓 𝑥4 = 1.0015
𝑓 𝑥4 = 1.0015

𝑓 𝑥4 < 𝑓 𝑥1 → Delete the interval ,𝑥 3 , 𝑥1-

𝑥1 = −0.236 𝑥4 = −0.056
𝑥2 = 0.236
 Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2

𝐿2 = 1.236
The value of that minimizes is located in the interval [-0.236, 0.236]
𝐿3 = 0.764
𝐿4 = 0. 472

𝑥1 = −0.236 𝑥2 = 0.236
𝐿4 = 0.472
Optimization Techniques
Numerical Optimization Techniques

▪ Example. Minimize the following function in the interval [-1,1] by using Golden Section
method using n = 4?
𝑓 𝑥 = 𝑒 𝑥 − sin 𝑥

▪ Solution. 𝐿0 = 2
𝐿4 0.472
The ratio of the final to the initial interval of uncertainty: = = 0.236 𝐿4 = 0. 472
𝐿0 2
1.9

1.8
𝐿0 = 2
1.7

1.6

1.5

1.4 𝐿4 = 0.472
1.3

1.2

1.1

1
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
𝑎 = −1 𝑥1 = −0.236 𝑥2 = 0.236 b=1