From 75f32b0c87a5564315a5efd8148a5e20778504d3 Mon Sep 17 00:00:00 2001 From: DrDrunkenstein42 Date: Wed, 27 Apr 2022 17:13:52 +0530 Subject: [PATCH 1/7] Divide and conquer - minor addition to preconditions --- src/dynamic_programming/divide-and-conquer-dp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/dynamic_programming/divide-and-conquer-dp.md b/src/dynamic_programming/divide-and-conquer-dp.md index 03bd8807b..15afd61af 100644 --- a/src/dynamic_programming/divide-and-conquer-dp.md +++ b/src/dynamic_programming/divide-and-conquer-dp.md @@ -4,8 +4,8 @@ Divide and Conquer is a dynamic programming optimization. ### Preconditions Some dynamic programming problems have a recurrence of this form: $$dp(i, j) = -\min_{0 \leq k \leq j} \\{ dp(i - 1, k - 1) + C(k, j) \\}$$ Where $C(k, j)$ is a cost -function and $dp(i, j) = 0$ when $j \lt 0$. +\min_{0 \leq k \leq j} \\{ dp(i - 1, k - 1) + C(k, j) \\}$$, where $C(k, j)$ is a cost +function and $dp(i, j) = 0$ when $j \lt 0$. Further, $C(i, j)$ must satisfy the quadrangle inequality, i.e., $C(b, c) \leq C(a, d)$ for all $a \leq b \leq c \leq d$. Say $0 \leq i \lt m$ and $0 \leq j \lt n$, and evaluating $C$ takes $O(1)$ time. Then the straightforward evaluation of the above recurrence is $O(m n^2)$. There From 72a34c3220c278a9e5dac32693feee5a3b4f0f0d Mon Sep 17 00:00:00 2001 From: omkarbajaj073 <73171175+omkarbajaj073@users.noreply.github.com> Date: Thu, 28 Apr 2022 10:41:59 +0530 Subject: [PATCH 2/7] Update divide-and-conquer-dp.md Changed one sentence, to make the concept a little clearer. --- src/dynamic_programming/divide-and-conquer-dp.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/dynamic_programming/divide-and-conquer-dp.md b/src/dynamic_programming/divide-and-conquer-dp.md index 15afd61af..eed84d1df 100644 --- a/src/dynamic_programming/divide-and-conquer-dp.md +++ b/src/dynamic_programming/divide-and-conquer-dp.md @@ -11,9 +11,10 @@ Say $0 \leq i \lt m$ and $0 \leq j \lt n$, and evaluating $C$ takes $O(1)$ time. Then the straightforward evaluation of the above recurrence is $O(m n^2)$. There are $m \times n$ states, and $n$ transitions for each state. -Let $opt(i, j)$ be the value of $k$ that minimizes the above expression. If -$opt(i, j) \leq opt(i, j + 1)$ for all $i, j$, then we can apply -divide-and-conquer DP. This is known as the _monotonicity condition_. The optimal +Let $opt(i, j)$ be the value of $k$ that minimizes the above expression. Assuming that the +cost function satisfies the qudrangle inequality, we can show that +$opt(i, j) \leq opt(i, j + 1)$ for all $i, j$. This is known as the _monotonicity condition_. +Then, we can apply divide and conquer DP. The optimal "splitting point" for a fixed $i$ increases as $j$ increases. This lets us solve for all states more efficiently. Say we compute $opt(i, j)$ From b472d03d7951663f82dde3f3f3361cde94700a70 Mon Sep 17 00:00:00 2001 From: DrDrunkenstein42 Date: Fri, 29 Apr 2022 15:27:25 +0530 Subject: [PATCH 3/7] Update - divide-and-conquer-dp.md --- src/dynamic_programming/divide-and-conquer-dp.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/dynamic_programming/divide-and-conquer-dp.md b/src/dynamic_programming/divide-and-conquer-dp.md index 03bd8807b..1e8996741 100644 --- a/src/dynamic_programming/divide-and-conquer-dp.md +++ b/src/dynamic_programming/divide-and-conquer-dp.md @@ -80,9 +80,9 @@ int solve() { ### Things to look out for The greatest difficulty with Divide and Conquer DP problems is proving the -monotonicity of $opt$. Many Divide and Conquer DP problems can also be solved -with the Convex Hull trick or vice-versa. It is useful to know and understand -both! +monotonicity of $opt$. One special case where this is true is when the cost function satisfies the quadrangle inequality, i.e., $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ for all $a \leq b \leq c \leq d$. +Many Divide and Conquer DP problems can also be solved with the Convex Hull trick or vice-versa. It is useful to know and understand +both! ## Practice Problems - [AtCoder - Yakiniku Restaurants](https://atcoder.jp/contests/arc067/tasks/arc067_d) From bd849da13a62e020910434649effdd1f67033bed Mon Sep 17 00:00:00 2001 From: DrDrunkenstein42 Date: Fri, 29 Apr 2022 15:27:35 +0530 Subject: [PATCH 4/7] Article added - Knuth's optimization --- src/dynamic_programming/knuth-optimization.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 src/dynamic_programming/knuth-optimization.md diff --git a/src/dynamic_programming/knuth-optimization.md b/src/dynamic_programming/knuth-optimization.md new file mode 100644 index 000000000..958a346e2 --- /dev/null +++ b/src/dynamic_programming/knuth-optimization.md @@ -0,0 +1,71 @@ +# Knuth's optimization + +Knuth's optimization is a special case range DP, that can optimize the time complexity of solutions by a linear factor. + +### Preconditions + +This DP Speedup is applied for dp transitions of the form - $dp(i, j) = +\min_{i \leq k < j} [ dp(i, k) + dp(k+1, j) + C(i, j) ]$. Further, +the cost function $C$ must satisfy the following conditions (for $a \leq b \leq c \leq d$) - +1. $C(b, c) \leq C(a, d)$ +2. $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ + +Now, just as in divide and conquer DP, let $opt(i, j)$ be the value of $k$ that minimizes the above expression in the dp transition. Then, we can show the following monotonicity condition - $opt[i][j-1] \leq opt[i][j] \leq opt[i+1][j]$. The proof for this is linked in the resources for this article, which contains a sample problem and itss editorial as well. + +This simplifies the problem a lot. If we process the dp states in such a way that we calculate $dp(i, j-1)$ and $dp(i+1, j)$ before $dp(i, j)$, and in doing so calculate $opt(i, j-1)$ and $opt(i+1, j)$, then when we process $dp(i, j)$, instead of testing values of $k$ from $i$ to $j-1$, we only need to loop from $opt(i, j-1)$ to $opt(i+1, j)$. + +## Generic implementation + +Even though implementation varies based on problem, here's a fairly generic +template. Notice that the structure of the code is almost identical to that of Range DP. +It can be shown (link in resources) that the 2 inner loops combined take $\mathcal{O}(n)$ time to run. Thus, the overall time complexity of the algorithm is $\mathcal{O}(n^2)$. + +```{.cpp file=knuth_optimization} +int M, N; +int dp[N][N]; + +// Implement cost function C. +int C(int i, int j); + +int solve() { + + for (int i = 0; i < N; i++) { + opt[i][i] = i; + // Initialize dp[i][i] according to the problem + } + + for (int i = N-2; i >= 0; i--) { + for (int j = i+1; j < N; j++) { + int mn = INT_MAX; + int cost = C(i, j); + for (int k = opt[i][j-1]; k <= min(j-1, opt[i+1][j]); k++) { + if (mn >= dp[i][k] + dp[k+1][j] + cost) { + opt[i][j] = k; + mn = dp[i][k] + dp[k+1][j] + cost; + } + } + dp[i][j] = mn; + } + } + + cout << dp[0][N-1] << endl; +} +``` + +### Things to look out for + +The most common application of Knuth's optimization is in Range DP, with the given transition. The only difficulty is in proving that the cost function satisfies the given conditions. The simplest case is when the cost function C(i, j) is simply the sum of the elements of the subarray S[i, i+1, ... j] for some array (depending on the question). However, they can be more complicated at times. + +Further, note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems (found very rarely), such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $opt[i][j-1] \leq opt[i][j] \leq opt[i+1][j]$, and thus, use this optimization. + +## Practice Problems +- [UVA - Cutting Sticks](https://onlinejudge.org/external/100/10003.pdf) +- [UVA - Prefix Codes](https://onlinejudge.org/external/120/12057.pdf) +- [SPOJ - Breaking String](https://www.spoj.com/problems/BRKSTRNG/) +- [UVA - Optimal Binary Search Tree](https://onlinejudge.org/external/103/10304.pdf) + + +## References +- [Geeksforgeeks Article (with proof of the optimization)](https://www.geeksforgeeks.org/knuths-optimization-in-dynamic-programming/) +- [Tutorial on USACO.guide]() +- [Doc on DP Speedups](https://home.cse.ust.hk/~golin/COMP572/Notes/DP_speedup.pdf) From cb54b72af7f94bc3fc5d0d3447c20d3a837e4183 Mon Sep 17 00:00:00 2001 From: DrDrunkenstein42 Date: Fri, 29 Apr 2022 15:30:45 +0530 Subject: [PATCH 5/7] Minor bug fix --- src/dynamic_programming/divide-and-conquer-dp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/dynamic_programming/divide-and-conquer-dp.md b/src/dynamic_programming/divide-and-conquer-dp.md index 9d55f3e42..d8894565e 100644 --- a/src/dynamic_programming/divide-and-conquer-dp.md +++ b/src/dynamic_programming/divide-and-conquer-dp.md @@ -5,7 +5,7 @@ Divide and Conquer is a dynamic programming optimization. ### Preconditions Some dynamic programming problems have a recurrence of this form: $$dp(i, j) = \min_{0 \leq k \leq j} \\{ dp(i - 1, k - 1) + C(k, j) \\}$$, where $C(k, j)$ is a cost -function and $dp(i, j) = 0$ when $j \lt 0$. Further, $C(i, j)$ must satisfy the quadrangle inequality, i.e., $C(b, c) \leq C(a, d)$ for all $a \leq b \leq c \leq d$. +function and $dp(i, j) = 0$ when $j \lt 0$. Say $0 \leq i \lt m$ and $0 \leq j \lt n$, and evaluating $C$ takes $O(1)$ time. Then the straightforward evaluation of the above recurrence is $O(m n^2)$. There From 8d127e0a98f01998cdd942aa4de00fc40a2ce83b Mon Sep 17 00:00:00 2001 From: DrDrunkenstein42 Date: Fri, 6 May 2022 15:52:24 +0530 Subject: [PATCH 6/7] Update - Knuth Optimization Proof of Correctness --- src/dynamic_programming/knuth-optimization.md | 82 +++++++++++++++---- src/index.md | 1 + 2 files changed, 69 insertions(+), 14 deletions(-) diff --git a/src/dynamic_programming/knuth-optimization.md b/src/dynamic_programming/knuth-optimization.md index 958a346e2..c69785f44 100644 --- a/src/dynamic_programming/knuth-optimization.md +++ b/src/dynamic_programming/knuth-optimization.md @@ -1,19 +1,73 @@ -# Knuth's optimization +# Knuth's Optimization -Knuth's optimization is a special case range DP, that can optimize the time complexity of solutions by a linear factor. +Knuth's optimization, also known as the Knuth-Yao Speedup, is a special case range DP, that can optimize the time complexity of solutions by a linear factor. -### Preconditions +### Conditions This DP Speedup is applied for dp transitions of the form - $dp(i, j) = \min_{i \leq k < j} [ dp(i, k) + dp(k+1, j) + C(i, j) ]$. Further, the cost function $C$ must satisfy the following conditions (for $a \leq b \leq c \leq d$) - + 1. $C(b, c) \leq C(a, d)$ -2. $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ -Now, just as in divide and conquer DP, let $opt(i, j)$ be the value of $k$ that minimizes the above expression in the dp transition. Then, we can show the following monotonicity condition - $opt[i][j-1] \leq opt[i][j] \leq opt[i+1][j]$. The proof for this is linked in the resources for this article, which contains a sample problem and itss editorial as well. +2. $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ (the quadrangle inequality [QI]) + +### Algorithm + +Now, just as in divide and conquer DP, let $opt(i, j)$ be the value of $k$ that minimizes the above expression in the dp transition (this is referred to as the "optimal splitting point" further in this article). Then, we can show the following monotonicity condition - $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$. The proof for this is linked in the resources for this article, which contains a sample problem and itss editorial as well. This simplifies the problem a lot. If we process the dp states in such a way that we calculate $dp(i, j-1)$ and $dp(i+1, j)$ before $dp(i, j)$, and in doing so calculate $opt(i, j-1)$ and $opt(i+1, j)$, then when we process $dp(i, j)$, instead of testing values of $k$ from $i$ to $j-1$, we only need to loop from $opt(i, j-1)$ to $opt(i+1, j)$. +### Proof of correctness + +To prove the correctness of this algorithm, it suffices to prove that $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$ assuming the given conditions are satisfied. + +Lemma: $dp(i, j)$ also satisfies the quadrangle inequality, given the conditions of the problem are satisfied. +The proof for this lemma uses strong induction. It has been taken from the paper Efficient Dynamic Programming Using Quadrangle Inequalities, authored by F. Frances Yao, which introduced the Knuth-Yao Speedup (this particular statement is Lemma 2.1 in the paper). The idea is to induct on the length $l = d - a$. The case where $l = 1$ is trivial. For $l > 1$ consider 2 cases: + +1. $b = c$ +The inequality reduces to $dp(a, b) + dp(b, d) \leq dp(a, d)$ (This assumes that $dp(i, i) = 0$ for all $i$, which is the case for all problems using this optimization). Let $opt[a][d] = z$. + + - If $z < j$, + Note that $dp(a, b) \leq dp_{z}(a, b) = dp(a, z) + dp(z+1, b) + C(a, b)$. + Therefore, + $dp(a, b) + dp(b, d) \leq dp(a, z) + dp(z+1, b) + dp(b, d) + C(a, b)$ + From the induction hypothesis, $dp(z+1, b) + dp(b, d) \leq dp(z+1, d)$. Also, it is given that $C(a, b) \leq C(a, d)$. Combining these 2 facts with above inequality yields the desired result. + + - If $z \geq j$, the proof of this case is symmetric to the previous case. + +2. $b < c$ +Let $opt(b, c) = z$ and $opt(a, d) = y$. + + - If $z \leq y$, + $dp(a, c) + dp(b, d)$ + \leq dp_{z}(a, c) + dp_{y}(b, d)$ + $= C(a, c) + C(b, d) + dp(a, z) + dp(z+1, c) + dp(b, y) + dp(y+1, d)$ + Using the QI on $C$ and on the dp state for the indices $z+1 \leq y+1 \leq c \leq d$ (from the induction hypothesis) yields the desired result. + + - If $z > y$, the proof of this case is symmetric to the previous case. + +This completes the proof of the lemma. + +Now, consider the following setup. We have 2 indices $i \leq p \leq q < j$. Set $dp_{k} = C(i, j) + dp(i, k) + dp(k+1, j)$. Suppose we show that $dp_{p}(i, j-1) \geq dp_{q}(i, j-1) \implies dp_{p}(i, j) \geq dp_{q}(i, j)$. Setting $q = opt(i, j-1)$, by definition, $dp_{p}(i, j-1) \geq dp_{q}(i, j-1)$. Therefore, applying the inequality to all $i \leq p \leq q$, we can infer that $opt(i, j)$ is at least as much as $opt(i, j-1)$, proving the first half of the inequality. + +Now, using the QI on some indices $p+1 \leq q+1 \leq j-1 \leq j$ +$dp(p+1, j-1) + dp(q+1, j) ≤ dp(q+1, j-1) + dp(p+1, j)$ +$⇒ (dp(i, p) + dp(p+1, j-1) + C(i, j-1)) + (dp(i, q) + dp(q+1, j) + C(i, j))$ +$≤ (dp(i, q) + dp(q+1, j-1) + C(i, j-1)) + (dp(i, p) + dp(p+1, j) + C(i, j))$ +$⇒ dp_{p}(i, j-1) + dp_{q}(i, j) ≤ dp_{p}(i, j) + dp_{q}(i, j-1)$ +$⇒ dp_{p}(i, j-1) - dp_{q}(i, j-1) ≤ dp_{p}(i, j) - dp_{q}(i, j)$ + +Finally, +$dp_{p}(i, j-1) ≥ dp_{q}(i, j-1)$ +$⇒ 0 ≤ dp_{p}(i, j-1) - dp_{q}(i, j-1) ≤ dp_{p}(i, j) - dp_{q}(i, j)$ +$⇒ dp_{p}(i, j) ≥ dp_{q}(i, j)$ + +This proves the first part of the inequality, i.e., $opt(i, j-1) \leq opt(i, j)$. The second part ($opt(i, j) \leq opt(i+1, j)$) can be shown with the same idea, starting with the inequality +$dp(i, p) + dp(i+1, q) ≤ dp(i+1, p) + dp(i, q)$. + +This completes the proof. + ## Generic implementation Even though implementation varies based on problem, here's a fairly generic @@ -38,13 +92,13 @@ int solve() { for (int j = i+1; j < N; j++) { int mn = INT_MAX; int cost = C(i, j); - for (int k = opt[i][j-1]; k <= min(j-1, opt[i+1][j]); k++) { - if (mn >= dp[i][k] + dp[k+1][j] + cost) { - opt[i][j] = k; - mn = dp[i][k] + dp[k+1][j] + cost; + for (int k = opt(i, j-1); k <= min(j-1, opt(i+1, j)); k++) { + if (mn >= dp(i, k) + dp(k+1, j) + cost) { + opt(i, j) = k; + mn = dp(i, k) + dp(k+1, j) + cost; } } - dp[i][j] = mn; + dp(i, j) = mn; } } @@ -52,11 +106,11 @@ int solve() { } ``` -### Things to look out for +### Extra Information The most common application of Knuth's optimization is in Range DP, with the given transition. The only difficulty is in proving that the cost function satisfies the given conditions. The simplest case is when the cost function C(i, j) is simply the sum of the elements of the subarray S[i, i+1, ... j] for some array (depending on the question). However, they can be more complicated at times. -Further, note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems (found very rarely), such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $opt[i][j-1] \leq opt[i][j] \leq opt[i+1][j]$, and thus, use this optimization. +Further, note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems (found very rarely), such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$, and thus, use this optimization. ## Practice Problems - [UVA - Cutting Sticks](https://onlinejudge.org/external/100/10003.pdf) @@ -66,6 +120,6 @@ Further, note that more than the conditions on the dp transition and the cost fu ## References -- [Geeksforgeeks Article (with proof of the optimization)](https://www.geeksforgeeks.org/knuths-optimization-in-dynamic-programming/) -- [Tutorial on USACO.guide]() +- [Geeksforgeeks Article](https://www.geeksforgeeks.org/knuths-optimization-in-dynamic-programming/) - [Doc on DP Speedups](https://home.cse.ust.hk/~golin/COMP572/Notes/DP_speedup.pdf) +- [Efficient Dynamic Programming Using Quadrangle Inequalities](https://dl.acm.org/doi/pdf/10.1145/800141.804691) \ No newline at end of file diff --git a/src/index.md b/src/index.md index 3bffc18cc..ed00f8538 100644 --- a/src/index.md +++ b/src/index.md @@ -72,6 +72,7 @@ Full list of updates: [Commit History](https://github.com/e-maxx-eng/e-maxx-eng/ - **DP optimizations** - [Divide and Conquer DP](dynamic_programming/divide-and-conquer-dp.md) + - [Knuth's Optimization](dynamic_programming/knuth-optimization.md) - **Tasks** - [Dynamic Programming on Broken Profile. Problem "Parquet"](dynamic_programming/profile-dynamics.md) - [Finding the largest zero submatrix](dynamic_programming/zero_matrix.md) From e179fdcdccb60cee3a3754bf8bc6a8e1408e5d04 Mon Sep 17 00:00:00 2001 From: Oleksandr Kulkov Date: Sat, 7 May 2022 23:32:02 +0200 Subject: [PATCH 7/7] style changes, add proof for complexity --- src/dynamic_programming/knuth-optimization.md | 215 ++++++++++++------ 1 file changed, 140 insertions(+), 75 deletions(-) diff --git a/src/dynamic_programming/knuth-optimization.md b/src/dynamic_programming/knuth-optimization.md index c69785f44..9dba356c3 100644 --- a/src/dynamic_programming/knuth-optimization.md +++ b/src/dynamic_programming/knuth-optimization.md @@ -1,104 +1,61 @@ # Knuth's Optimization -Knuth's optimization, also known as the Knuth-Yao Speedup, is a special case range DP, that can optimize the time complexity of solutions by a linear factor. +Knuth's optimization, also known as the Knuth-Yao Speedup, is a special case of dynamic programming on ranges, that can optimize the time complexity of solutions by a linear factor, from $O(n^3)$ for standard range DP to $O(n^2)$. -### Conditions +## Conditions -This DP Speedup is applied for dp transitions of the form - $dp(i, j) = -\min_{i \leq k < j} [ dp(i, k) + dp(k+1, j) + C(i, j) ]$. Further, -the cost function $C$ must satisfy the following conditions (for $a \leq b \leq c \leq d$) - +The Speedup is applied for transitions of the form -1. $C(b, c) \leq C(a, d)$ +$$dp(i, j) = \min_{i \leq k < j} [ dp(i, k) + dp(k+1, j) + C(i, j) ].$$ -2. $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ (the quadrangle inequality [QI]) +Similar to [divide and conquer DP](../divide-and-conquer-dp.md), let $opt(i, j)$ be the value of $k$ that minimizes the expression in the transition ($opt$ is referred to as the "optimal splitting point" further in this article). The optimization requires that the following holds: -### Algorithm +$$opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j).$$ -Now, just as in divide and conquer DP, let $opt(i, j)$ be the value of $k$ that minimizes the above expression in the dp transition (this is referred to as the "optimal splitting point" further in this article). Then, we can show the following monotonicity condition - $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$. The proof for this is linked in the resources for this article, which contains a sample problem and itss editorial as well. +We can show that it is true when the cost function $C$ satisfies the following conditions for $a \leq b \leq c \leq d$: -This simplifies the problem a lot. If we process the dp states in such a way that we calculate $dp(i, j-1)$ and $dp(i+1, j)$ before $dp(i, j)$, and in doing so calculate $opt(i, j-1)$ and $opt(i+1, j)$, then when we process $dp(i, j)$, instead of testing values of $k$ from $i$ to $j-1$, we only need to loop from $opt(i, j-1)$ to $opt(i+1, j)$. +1. $C(b, c) \leq C(a, d)$; -### Proof of correctness - -To prove the correctness of this algorithm, it suffices to prove that $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$ assuming the given conditions are satisfied. - -Lemma: $dp(i, j)$ also satisfies the quadrangle inequality, given the conditions of the problem are satisfied. -The proof for this lemma uses strong induction. It has been taken from the paper Efficient Dynamic Programming Using Quadrangle Inequalities, authored by F. Frances Yao, which introduced the Knuth-Yao Speedup (this particular statement is Lemma 2.1 in the paper). The idea is to induct on the length $l = d - a$. The case where $l = 1$ is trivial. For $l > 1$ consider 2 cases: - -1. $b = c$ -The inequality reduces to $dp(a, b) + dp(b, d) \leq dp(a, d)$ (This assumes that $dp(i, i) = 0$ for all $i$, which is the case for all problems using this optimization). Let $opt[a][d] = z$. - - - If $z < j$, - Note that $dp(a, b) \leq dp_{z}(a, b) = dp(a, z) + dp(z+1, b) + C(a, b)$. - Therefore, - $dp(a, b) + dp(b, d) \leq dp(a, z) + dp(z+1, b) + dp(b, d) + C(a, b)$ - From the induction hypothesis, $dp(z+1, b) + dp(b, d) \leq dp(z+1, d)$. Also, it is given that $C(a, b) \leq C(a, d)$. Combining these 2 facts with above inequality yields the desired result. - - - If $z \geq j$, the proof of this case is symmetric to the previous case. - -2. $b < c$ -Let $opt(b, c) = z$ and $opt(a, d) = y$. - - - If $z \leq y$, - $dp(a, c) + dp(b, d)$ - \leq dp_{z}(a, c) + dp_{y}(b, d)$ - $= C(a, c) + C(b, d) + dp(a, z) + dp(z+1, c) + dp(b, y) + dp(y+1, d)$ - Using the QI on $C$ and on the dp state for the indices $z+1 \leq y+1 \leq c \leq d$ (from the induction hypothesis) yields the desired result. - - - If $z > y$, the proof of this case is symmetric to the previous case. +2. $C(a, c) + C(b, d) \leq C(a, d) + C(b, c)$ (the quadrangle inequality [QI]). -This completes the proof of the lemma. +This result is proved further below. -Now, consider the following setup. We have 2 indices $i \leq p \leq q < j$. Set $dp_{k} = C(i, j) + dp(i, k) + dp(k+1, j)$. Suppose we show that $dp_{p}(i, j-1) \geq dp_{q}(i, j-1) \implies dp_{p}(i, j) \geq dp_{q}(i, j)$. Setting $q = opt(i, j-1)$, by definition, $dp_{p}(i, j-1) \geq dp_{q}(i, j-1)$. Therefore, applying the inequality to all $i \leq p \leq q$, we can infer that $opt(i, j)$ is at least as much as $opt(i, j-1)$, proving the first half of the inequality. +## Algorithm -Now, using the QI on some indices $p+1 \leq q+1 \leq j-1 \leq j$ -$dp(p+1, j-1) + dp(q+1, j) ≤ dp(q+1, j-1) + dp(p+1, j)$ -$⇒ (dp(i, p) + dp(p+1, j-1) + C(i, j-1)) + (dp(i, q) + dp(q+1, j) + C(i, j))$ -$≤ (dp(i, q) + dp(q+1, j-1) + C(i, j-1)) + (dp(i, p) + dp(p+1, j) + C(i, j))$ -$⇒ dp_{p}(i, j-1) + dp_{q}(i, j) ≤ dp_{p}(i, j) + dp_{q}(i, j-1)$ -$⇒ dp_{p}(i, j-1) - dp_{q}(i, j-1) ≤ dp_{p}(i, j) - dp_{q}(i, j)$ - -Finally, -$dp_{p}(i, j-1) ≥ dp_{q}(i, j-1)$ -$⇒ 0 ≤ dp_{p}(i, j-1) - dp_{q}(i, j-1) ≤ dp_{p}(i, j) - dp_{q}(i, j)$ -$⇒ dp_{p}(i, j) ≥ dp_{q}(i, j)$ - -This proves the first part of the inequality, i.e., $opt(i, j-1) \leq opt(i, j)$. The second part ($opt(i, j) \leq opt(i+1, j)$) can be shown with the same idea, starting with the inequality -$dp(i, p) + dp(i+1, q) ≤ dp(i+1, p) + dp(i, q)$. - -This completes the proof. +Let's process the dp states in such a way that we calculate $dp(i, j-1)$ and $dp(i+1, j)$ before $dp(i, j)$, and in doing so we also calculate $opt(i, j-1)$ and $opt(i+1, j)$. Then for calculating $opt(i, j)$, instead of testing values of $k$ from $i$ to $j-1$, we only need to test from $opt(i, j-1)$ to $opt(i+1, j)$. To process $(i,j)$ pairs in this order it is sufficient to use nested for loops in which $i$ goes from the maximum value to the minimum one and $j$ goes from $i+1$ to the maximum value. -## Generic implementation +### Generic implementation -Even though implementation varies based on problem, here's a fairly generic -template. Notice that the structure of the code is almost identical to that of Range DP. -It can be shown (link in resources) that the 2 inner loops combined take $\mathcal{O}(n)$ time to run. Thus, the overall time complexity of the algorithm is $\mathcal{O}(n^2)$. +Though implementation varies, here's a fairly generic +example. The structure of the code is almost identical to that of Range DP. ```{.cpp file=knuth_optimization} -int M, N; -int dp[N][N]; - -// Implement cost function C. -int C(int i, int j); int solve() { + int N; + ... // read N and input + int dp[N][N], opt[N][N]; + + auto C = [&](int i, int j) { + ... // Implement cost function C. + }; for (int i = 0; i < N; i++) { opt[i][i] = i; - // Initialize dp[i][i] according to the problem + ... // Initialize dp[i][i] according to the problem } for (int i = N-2; i >= 0; i--) { for (int j = i+1; j < N; j++) { int mn = INT_MAX; int cost = C(i, j); - for (int k = opt(i, j-1); k <= min(j-1, opt(i+1, j)); k++) { - if (mn >= dp(i, k) + dp(k+1, j) + cost) { - opt(i, j) = k; - mn = dp(i, k) + dp(k+1, j) + cost; + for (int k = opt[i][j-1]; k <= min(j-1, opt[i+1][j]); k++) { + if (mn >= dp[i][k] + dp[k+1][j] + cost) { + opt[i][j] = k; + mn = dp[i][k] + dp[k+1][j] + cost; } } - dp(i, j) = mn; + dp[i][j] = mn; } } @@ -106,11 +63,119 @@ int solve() { } ``` -### Extra Information +### Complexity + +A complexity of the algorithm can be estimated as the following sum: + +$$ +\sum\limits_{i=1}^N \sum\limits_{j=i+1}^N [opt(i+1,j)-opt(i,j-1)] = +\sum\limits_{i=1}^N \sum\limits_{j=i}^{N-1} [opt(i+1,j+1)-opt(i,j)]. +$$ + +As you see, most of the terms in this expression cancel each other out, except for positive terms with $j=N$ and negative terms with $i=1$. Thus, the whole sum can be estimated as + +$$ +\sum\limits_{k=1}^N[opt(k,N)-opt(1,k)] = O(n^2), +$$ + +rather than $O(n^3)$ as it would be if we were using a regular range DP. + +### On practice + +The most common application of Knuth's optimization is in Range DP, with the given transition. The only difficulty is in proving that the cost function satisfies the given conditions. The simplest case is when the cost function $C(i, j)$ is simply the sum of the elements of the subarray $S[i, i+1, ..., j]$ for some array (depending on the question). However, they can be more complicated at times. + +Note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems, such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$, and thus, use this optimization. + + +### Proof of correctness -The most common application of Knuth's optimization is in Range DP, with the given transition. The only difficulty is in proving that the cost function satisfies the given conditions. The simplest case is when the cost function C(i, j) is simply the sum of the elements of the subarray S[i, i+1, ... j] for some array (depending on the question). However, they can be more complicated at times. +To prove the correctness of this algorithm in terms of $C(i,j)$ conditions, it suffices to prove that -Further, note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems (found very rarely), such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j)$, and thus, use this optimization. +$$ +opt(i, j-1) \leq opt(i, j) \leq opt(i+1, j) +$$ + +assuming the given conditions are satisfied. + +!!! lemma "Lemma" + $dp(i, j)$ also satisfies the quadrangle inequality, given the conditions of the problem are satisfied. + +??? hint "Proof" + The proof for this lemma uses strong induction. It has been taken from the paper Efficient Dynamic Programming Using Quadrangle Inequalities, authored by F. Frances Yao, which introduced the Knuth-Yao Speedup (this particular statement is Lemma 2.1 in the paper). The idea is to induct on the length $l = d - a$. The case where $l = 1$ is trivial. For $l > 1$ consider 2 cases: + + 1. $b = c$ + The inequality reduces to $dp(a, b) + dp(b, d) \leq dp(a, d)$ (This assumes that $dp(i, i) = 0$ for all $i$, which is the case for all problems using this optimization). Let $opt(a,d) = z$. + + - If $z < j$, + Note that + + $$ + dp(a, b) \leq dp_{z}(a, b) = dp(a, z) + dp(z+1, b) + C(a, b). + $$ + + Therefore, + + $$ + dp(a, b) + dp(b, d) \leq dp(a, z) + dp(z+1, b) + dp(b, d) + C(a, b) + $$ + + From the induction hypothesis, $dp(z+1, b) + dp(b, d) \leq dp(z+1, d)$. Also, it is given that $C(a, b) \leq C(a, d)$. Combining these 2 facts with above inequality yields the desired result. + + - If $z \geq j$, the proof of this case is symmetric to the previous case. + + 2. $b < c$ + Let $opt(b, c) = z$ and $opt(a, d) = y$. + + - If $z \leq y$, + + $$ + dp(a, c) + dp(b, d) \leq dp_{z}(a, c) + dp_{y}(b, d) + $$ + + where + + $$ + dp_{z}(a, c) + dp_{y}(b, d) = C(a, c) + C(b, d) + dp(a, z) + dp(z+1, c) + dp(b, y) + dp(y+1, d). + $$ + + Using the QI on $C$ and on the dp state for the indices $z+1 \leq y+1 \leq c \leq d$ (from the induction hypothesis) yields the desired result. + + - If $z > y$, the proof of this case is symmetric to the previous case. + + This completes the proof of the lemma. + +Now, consider the following setup. We have 2 indices $i \leq p \leq q < j$. Set $dp_{k} = C(i, j) + dp(i, k) + dp(k+1, j)$. + +Suppose we show that + +$$ +dp_{p}(i, j-1) \geq dp_{q}(i, j-1) \implies dp_{p}(i, j) \geq dp_{q}(i, j). +$$ + +Setting $q = opt(i, j-1)$, by definition, $dp_{p}(i, j-1) \geq dp_{q}(i, j-1)$. Therefore, applying the inequality to all $i \leq p \leq q$, we can infer that $opt(i, j)$ is at least as much as $opt(i, j-1)$, proving the first half of the inequality. + +Now, using the QI on some indices $p+1 \leq q+1 \leq j-1 \leq j$, we get + +$$\begin{align} +&dp(p+1, j-1) + dp(q+1, j) ≤ dp(q+1, j-1) + dp(p+1, j) \\ +\implies& (dp(i, p) + dp(p+1, j-1) + C(i, j-1)) + (dp(i, q) + dp(q+1, j) + C(i, j)) \\ +\leq& (dp(i, q) + dp(q+1, j-1) + C(i, j-1)) + (dp(i, p) + dp(p+1, j) + C(i, j)) \\ +\implies& dp_{p}(i, j-1) + dp_{q}(i, j) ≤ dp_{p}(i, j) + dp_{q}(i, j-1) \\ +\implies& dp_{p}(i, j-1) - dp_{q}(i, j-1) ≤ dp_{p}(i, j) - dp_{q}(i, j) \\ +\end{align}$$ + +Finally, + +$$\begin{align} +&dp_{p}(i, j-1) \geq dp_{q}(i, j-1) \\ +&\implies 0 \leq dp_{p}(i, j-1) - dp_{q}(i, j-1) \leq dp_{p}(i, j) - dp_{q}(i, j) \\ +&\implies dp_{p}(i, j) \geq dp_{q}(i, j) +\end{align}$$ + +This proves the first part of the inequality, i.e., $opt(i, j-1) \leq opt(i, j)$. The second part $opt(i, j) \leq opt(i+1, j)$ can be shown with the same idea, starting with the inequality +$dp(i, p) + dp(i+1, q) ≤ dp(i+1, p) + dp(i, q)$. + +This completes the proof. ## Practice Problems - [UVA - Cutting Sticks](https://onlinejudge.org/external/100/10003.pdf) @@ -122,4 +187,4 @@ Further, note that more than the conditions on the dp transition and the cost fu ## References - [Geeksforgeeks Article](https://www.geeksforgeeks.org/knuths-optimization-in-dynamic-programming/) - [Doc on DP Speedups](https://home.cse.ust.hk/~golin/COMP572/Notes/DP_speedup.pdf) -- [Efficient Dynamic Programming Using Quadrangle Inequalities](https://dl.acm.org/doi/pdf/10.1145/800141.804691) \ No newline at end of file +- [Efficient Dynamic Programming Using Quadrangle Inequalities](https://dl.acm.org/doi/pdf/10.1145/800141.804691)