From 4baf852b30df7c22982a92e09524c23cbefae6b8 Mon Sep 17 00:00:00 2001 From: Eric Book Date: Tue, 11 May 2021 21:35:42 -0400 Subject: [PATCH 01/17] readme: removed "in-progress" --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 0254728..633ac8f 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # Nested Cross-Validation: Comparing Methods and Implementations -### (In-progress) ![](images/ncv.png) From cffc2696e4ae3a32d0a14c83d394c06638477d38 Mon Sep 17 00:00:00 2001 From: Eric Book Date: Tue, 11 May 2021 21:39:26 -0400 Subject: [PATCH 02/17] readme: added zenodo badge --- README.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.Rmd b/README.Rmd index f5c3143..df83961 100644 --- a/README.Rmd +++ b/README.Rmd @@ -3,9 +3,9 @@ output: github_document --- # Nested Cross-Validation: Comparing Methods and Implementations -### (In-progress) ![](images/ncv.png) +[![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Examples of such situations include: proof of concept, start-ups, medical studies, time series, etc. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better judgement of generalization performance. From b86dba7f52edf66e05fab8c7c11fb84d2e14283f Mon Sep 17 00:00:00 2001 From: Eric Book Date: Tue, 11 May 2021 21:40:19 -0400 Subject: [PATCH 03/17] readme: added zenodo badge --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 633ac8f..be74ffa 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,8 @@ # Nested Cross-Validation: Comparing Methods and Implementations -![](images/ncv.png) +![](images/ncv.png) +[![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to From 3b78909038685f80ec1f8ee221c416bda0ee3d78 Mon Sep 17 00:00:00 2001 From: Eric Book Date: Tue, 11 May 2021 21:40:54 -0400 Subject: [PATCH 04/17] minor edit --- README.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.Rmd b/README.Rmd index df83961..48b51cd 100644 --- a/README.Rmd +++ b/README.Rmd @@ -4,7 +4,7 @@ output: github_document # Nested Cross-Validation: Comparing Methods and Implementations -![](images/ncv.png) +![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Examples of such situations include: proof of concept, start-ups, medical studies, time series, etc. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better judgement of generalization performance. From b5b51ab093cb5ccc58cb0e0cf1c905ab754538a7 Mon Sep 17 00:00:00 2001 From: ercbk Date: Tue, 10 Aug 2021 21:53:56 -0400 Subject: [PATCH 05/17] updated packages; finished readme --- README.Rmd | 66 +- README.md | 159 ++- README_files/figure-gfm/kj-patch-1.png | Bin 16818 -> 15927 bytes README_files/figure-gfm/kj_patch_kj-1.png | Bin 15493 -> 13694 bytes README_files/figure-gfm/unnamed-chunk-1-1.png | Bin 7395 -> 7163 bytes renv.lock | 1065 +++++++---------- renv/.gitignore | 2 + renv/activate.R | 575 +++++++-- renv/settings.dcf | 2 + 9 files changed, 1023 insertions(+), 846 deletions(-) diff --git a/README.Rmd b/README.Rmd index f5c3143..04f177c 100644 --- a/README.Rmd +++ b/README.Rmd @@ -7,9 +7,11 @@ output: github_document ![](images/ncv.png) -Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Examples of such situations include: proof of concept, start-ups, medical studies, time series, etc. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better judgement of generalization performance. +Experiments conducted in May 2020 -The primary issue with this technique is that it can be computationally expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). +Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. + +The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. Various elements of the technique affect the run times and performance. These include: @@ -21,7 +23,17 @@ Various elements of the technique affect the run times and performance. These in I'll be examining two aspects of nested cross-validation: 1. Duration: Find out which packages and combinations of model functions give us the fastest implementation of each method. -2. Performance: First, develop a testing framework. Then, for a given data generating process, how large of sample size is needed to obtain reasonably accurate out-of-sample error estimate? And how many repeats in the outer-loop cv strategy should be used to calculate this error estimate? +2. Performance: First, develop a testing framework. Then, for a given data generating process, determine how large of sample size is needed to obtain reasonably accurate out-of-sample error estimate. Also, determine how many repeats in the outer-loop cv strategy should be used to calculate this error estimate. + +The results from these experiments should give us an idea about which methodology, model packages, and compute specifications will produce lower training times, lower costs, and lower generalization error. + + +## Recommendations: + * Use {mlr3} or other R model packages outside of the {tidymodels} ecosystem and code the nested cross-validation loops manually. + * For data sizes in the low thousands, Raschka's method performs just as well as Kuhn-Johnson's but is substantially faster. + * For data sizes in the hundreds, Raschka's method with at least 3 repeats performs just as well as Kuhn-Johnson's but is still substantially faster even with the repeats. + * Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the r5.#xlarge instances ran fastest. + ## Duration @@ -39,7 +51,7 @@ I'll be examining two aspects of nested cross-validation: + outer loop: 5 folds + inner loop: 2 folds -The sizes of the data sets are the same as those in the original scripts by the authors. Using Kuhn-Johnson, 50,000 models (grid size * number of repeats * number of folds in the outer-loop * number of folds/resamples in the inner-loop) are trained for each algorithm — using Raschka's, 1,001 models for each algorithm. The one extra model in the Raschka variation is due to his method of choosing the hyperparameter values for the final model. He performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. +The sizes of the data sets are the same as those in the original scripts by the authors. Using Kuhn-Johnson, 50,000 models (grid size * number of repeats * number of folds in the outer-loop * number of folds/resamples in the inner-loop) are trained for each algorithm — using Raschka's, 1,001 models for each algorithm. The one extra model in the Raschka variation is due to his method of choosing the hyperparameter values for the final model. [MLFlow](https://mlflow.org/docs/latest/index.html) is used to keep track of the duration (seconds) of each run along with the implementation and method used. @@ -48,9 +60,7 @@ The sizes of the data sets are the same as those in the original scripts by the ![](duration-experiment/outputs/duration-pkg-tbl.png) ```{r, echo=FALSE, message=FALSE} -pacman::p_load(extrafont, dplyr, ggplot2, patchwork, stringr, tidytext) - - +pacman::p_load(dplyr, ggplot2, patchwork, stringr, tidytext) runs_raw <- readr::read_rds("duration-experiment/outputs/duration-runs.rds") @@ -88,8 +98,7 @@ kj <- runs %>% durations <- raschka + kj + plot_annotation(title = "Durations", subtitle = "minutes") & - theme(text = element_text(family = "Roboto"), - axis.ticks = element_blank(), + theme(axis.ticks = element_blank(), axis.text.x = element_blank(), panel.background = element_rect(fill = "ivory", colour = "ivory"), @@ -104,6 +113,14 @@ durations ``` +#### Duration Results: + + * For the Raschka method, the {mlr3} comes in first with {ranger}/{parsnip} coming in a close second. + * For the Kuhn-Johnson method, {ranger}/{parsnip} is clearly fastest. + * This was my first time using the {reticulate} package, and I wanted to see if there was any speed penalty for using its api instead of just running a straight Python script. There doesn't appear to be any. + * {h2o} and {sklearn} are surprisingly slow. If the data size were larger, I think {h2o} would be more competitive. + * The {tidymodels} packages, {parsnip} and {tune}, add substantial overhead. + ## Performance @@ -120,7 +137,7 @@ durations ```{r perf_build_times_kj, echo=FALSE, message=FALSE} -pacman::p_load(extrafont,dplyr, purrr, lubridate, ggplot2, ggfittext, drake, patchwork) +pacman::p_load(dplyr, purrr, lubridate, ggplot2, ggfittext, drake, patchwork) bt <- build_times(starts_with("ncv_results"), digits = 4) subtarget_bts <- bt %>% @@ -159,9 +176,7 @@ b <- ggplot(subtargets, aes(y = elapsed, x = repeats, coord_flip() + labs(y = "Runtime (hrs)", x = "Repeats", fill = "Sample Size") + - theme(title = element_text(family = "Roboto"), - text = element_text(family = "Roboto"), - legend.position = "top", + theme(legend.position = "top", legend.background = element_rect(fill = "ivory"), legend.key = element_rect(fill = "ivory"), axis.ticks = element_blank(), @@ -184,9 +199,7 @@ e <- ggplot(subtargets, aes(x = repeats, y = percent_error, group = n)) + scale_color_manual(values = fill_colors[4:7]) + labs(y = "Percent Error", x = "Repeats", color = "Sample Size") + - theme(title = element_text(family = "Roboto"), - text = element_text(family = "Roboto"), - legend.position = "top", + theme(legend.position = "top", legend.background = element_rect(fill = "ivory"), legend.key = element_rect(fill = "ivory"), axis.ticks = element_blank(), @@ -213,7 +226,7 @@ b + e + plot_layout(guides = "auto") + plot.background = element_rect(fill = "ivory"),) ``` -#### Results: +#### Performance Results (Kuhn-Johnson): * Runtimes for n = 100 and n = 800 are close, and there's a large jump in runtime going from n = 2000 to n = 5000. * The number of repeats has little effect on the amount of percent error. @@ -263,9 +276,7 @@ b_r <- ggplot(subtargets_r, aes(y = elapsed, x = repeats, coord_flip() + labs(y = "Runtime (minutes)", x = "Repeats", fill = "Sample Size") + - theme(title = element_text(family = "Roboto"), - text = element_text(family = "Roboto"), - legend.position = "top", + theme(legend.position = "top", legend.background = element_rect(fill = "ivory"), legend.key = element_rect(fill = "ivory"), axis.ticks = element_blank(), @@ -289,9 +300,7 @@ e_r <- ggplot(subtargets_r, aes(x = repeats, y = percent_error, group = n)) + scale_color_manual(values = fill_colors[4:7]) + labs(y = "Percent Error", x = "Repeats", color = "Sample Size") + - theme(title = element_text(family = "Roboto"), - text = element_text(family = "Roboto"), - legend.position = "top", + theme(legend.position = "top", legend.background = element_rect(fill = "ivory"), legend.key = element_rect(fill = "ivory"), axis.ticks = element_blank(), @@ -319,7 +328,7 @@ b_r + e_r + plot_layout(guides = "auto") + ``` -#### Results: +#### Performance Results (Raschka): * The longest runtime is under 30 minutes, so runtime isn't as large of a consideration if we are only comparing a few algorithms. * There isn't much difference in runtime between n = 100 and n = 2000. @@ -328,12 +337,17 @@ b_r + e_r + plot_layout(guides = "auto") + * n = 800 remains under 2.5% percent error for all repeat values, but also shows considerable volatility. - +## Summary + * Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error if your data size is a few thousand rows + + Kuhn-Johnson's runtime starts to really balloon once you get into dataset with over a thousand rows. + + The extra folds made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka's, it was minutes. + * For smaller datasets, you should have at least 3 repeats when running Rashka's method. + * This is just one dataset, but I still found it surprising how little a difference repeats made in reducing generalizaion error. The benefit only kicked in with the dataset that had hundred rows. -References +## References Boulesteix, AL, and C Strobl. 2009. “Optimal Classifier Selection and Negative Bias in Error Rate Estimation: An Empirical Study on High-Dimensional Prediction.” BMC Medical Research Methodology 9 (1): 85. [link](https://www.researchgate.net/publication/40756303_Optimal_classifier_selection_and_negative_bias_in_error_rate_estimation_An_empirical_study_on_high-dimensional_prediction) diff --git a/README.md b/README.md index 0254728..5600219 100644 --- a/README.md +++ b/README.md @@ -5,25 +5,35 @@ ![](images/ncv.png) +Experiments conducted in May 2020 + Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. -Examples of such situations include: proof of concept, start-ups, -medical studies, time series, etc. Using standard methods such as k-fold -cross-validation in these cases may result in substantial increases in -optimization bias. Nested cross-validation has been shown to produce -less biased, out-of-sample error estimates even using datasets with only -hundreds of rows and therefore gives a better judgement of -generalization performance. - -The primary issue with this technique is that it can be computationally -expensive with potentially tens of 1000s of models being trained during -the process. While researching this technique, I found two slightly -different variations of performing nested cross-validation — one -authored by [Sabastian +Using standard methods such as k-fold cross-validation in these cases +may result in substantial increases in optimization bias where the more +models that are trained on a fold means there’s a greater opportunity +for a model to achieve a low score by chance. Nested cross-validation +has been shown to produce less biased, out-of-sample error estimates +even using datasets with only hundreds of rows and therefore gives a +better estimation of generalized performance. + +The primary issue with this technique is that it is usually +computationally expensive with potentially tens of 1000s of models being +trained during the process. While researching this technique, I found +two slightly different variations of performing nested cross-validation +— one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). +After the nested cross-validation procedure and an algorithm is chosen, +Raschka performs an extra k-fold cross-validation using the inner-loop +cv strategy on the entire training set in order to tune his final model. +Therefore, the hyperparameter tuning that takes place in the inner-loop +during nested cross-validation is only in service of algorithm +selection. Kuhn-Johnson uses majority vote. Whichever set of +hyperparameter values has been chosen during the inner-loop tuning +procedure the most often is the set used to fit the final model. Various elements of the technique affect the run times and performance. These include: @@ -38,28 +48,45 @@ I’ll be examining two aspects of nested cross-validation: 1. Duration: Find out which packages and combinations of model functions give us the fastest implementation of each method. 2. Performance: First, develop a testing framework. Then, for a given - data generating process, how large of sample size is needed to - obtain reasonably accurate out-of-sample error estimate? And how - many repeats in the outer-loop cv strategy should be used to - calculate this error estimate? + data generating process, determine how large of sample size is + needed to obtain reasonably accurate out-of-sample error estimate. + Also, determine how many repeats in the outer-loop cv strategy + should be used to calculate this error estimate. + +The results from these experiments should give us an idea about which +methodology, model packages, and compute specifications will produce +lower training times, lower costs, and lower generalization error. + +## Recommendations: + +- Use {mlr3} or other R model packages outside of the {tidymodels} + ecosystem and code the nested cross-validation loops manually. +- For data sizes in the low thousands, Raschka’s method performs just + as well as Kuhn-Johnson’s but is substantially faster. +- For data sizes in the hundreds, Raschka’s method with at least 3 + repeats performs just as well as Kuhn-Johnson’s but is still + substantially faster even with the repeats. +- Choose compute resources with large amounts of RAM instead of opting + for powerful processors. From the AWS cpu product line, I found the + r5.\#xlarge instances ran fastest. ## Duration #### Experiment details: - - Random Forest and Elastic Net Regression algorithms - - Both algorithms are tuned with 100x2 hyperparameter grids using a +- Random Forest and Elastic Net Regression algorithms +- Both algorithms are tuned with 100x2 hyperparameter grids using a latin hypercube design. - - From {mlbench}, I’m using the generated data set, friedman1, from +- From {mlbench}, I’m using the generated data set, friedman1, from Friedman’s Multivariate Adaptive Regression Splines (MARS) paper. - - Kuhn-Johnson - - 100 observations: 10 features, numeric target variable - - outer loop: 2 repeats, 10 folds - - inner loop: 25 bootstrap resamples - - Raschka - - 5000 observations: 10 features, numeric target variable - - outer loop: 5 folds - - inner loop: 2 folds +- Kuhn-Johnson + - 100 observations: 10 features, numeric target variable + - outer loop: 2 repeats, 10 folds + - inner loop: 25 bootstrap resamples +- Raschka + - 5000 observations: 10 features, numeric target variable + - outer loop: 5 folds + - inner loop: 2 folds The sizes of the data sets are the same as those in the original scripts by the authors. Using Kuhn-Johnson, 50,000 models (grid size \* number @@ -67,11 +94,7 @@ of repeats \* number of folds in the outer-loop \* number of folds/resamples in the inner-loop) are trained for each algorithm — using Raschka’s, 1,001 models for each algorithm. The one extra model in the Raschka variation is due to his method of choosing the -hyperparameter values for the final model. He performs an extra k-fold -cross-validation using the inner-loop cv strategy on the entire training -set. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter -values has been chosen during the inner-loop tuning procedure the most -often is the set used to fit the final model. +hyperparameter values for the final model. [MLFlow](https://mlflow.org/docs/latest/index.html) is used to keep track of the duration (seconds) of each run along with the @@ -83,32 +106,47 @@ implementation and method used. ![](README_files/figure-gfm/unnamed-chunk-1-1.png) +#### Duration Results: + +- For the Raschka method, the {mlr3} comes in first with + {ranger}/{parsnip} coming in a close second. +- For the Kuhn-Johnson method, {ranger}/{parsnip} is clearly + fastest. +- This was my first time using the {reticulate} package, and I wanted + to see if there was any speed penalty for using its api instead of + just running a straight Python script. There doesn’t appear to be + any. +- {h2o} and {sklearn} are surprisingly slow. If the data size were + larger, I think {h2o} would be more competitive. +- The {tidymodels} packages, {parsnip} and {tune}, add substantial + overhead. + ## Performance #### Experiment details: - - The same data, algorithms, and hyperparameter grids are used. - - The fastest implementation of each method is used in running a +- The same data, algorithms, and hyperparameter grids are used. +- The fastest implementation of each method is used in running a nested cross-validation with different sizes of data ranging from 100 to 5000 observations and different numbers of repeats of the outer-loop cv strategy. - - The {mlr3} implementation is the fastest for Raschka’s method, + - The {mlr3} implementation is the fastest for Raschka’s method, but the Ranger-Kuhn-Johnson implementation is close. To simplify, I am using [Ranger-Kuhn-Johnson](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R) for both methods. - - The chosen algorithm with hyperparameters is fit on the entire +- The chosen algorithm with hyperparameters is fit on the entire training set, and the resulting final model predicts on a 100K row Friedman dataset. - - The percent error between the the average mean absolute error (MAE) +- The percent error between the the average mean absolute error (MAE) across the outer-loop folds and the MAE of the predictions on this 100K dataset is calculated for each combination of repeat, data size, and method. - - To make this experiment manageable in terms of runtimes, I am using +- To make this experiment manageable in terms of runtimes, I am using AWS instances: a r5.2xlarge for the Elastic Net and a r5.24xlarge for Random Forest. - - Also see the Other Notes section - - Iterating through different numbers of repeats, sample sizes, and + - Also see the Other Notes section +- Iterating through different numbers of repeats, sample sizes, and methods makes a functional approach more appropriate than running imperative scripts. Also, given the long runtimes and impermanent nature of my internet connection, it would also be nice to cache @@ -118,36 +156,51 @@ implementation and method used. ![](README_files/figure-gfm/kj_patch_kj-1.png) -#### Results: +#### Performance Results (Kuhn-Johnson): - - Runtimes for n = 100 and n = 800 are close, and there’s a large jump +- Runtimes for n = 100 and n = 800 are close, and there’s a large jump in runtime going from n = 2000 to n = 5000. - - The number of repeats has little effect on the amount of percent +- The number of repeats has little effect on the amount of percent error. - - For n = 100, there is substantially more variation in percent error +- For n = 100, there is substantially more variation in percent error than in the other sample sizes. - - While there is a large runtime cost that comes with increasing the +- While there is a large runtime cost that comes with increasing the sample size from 2000 to 5000 observations, it doesn’t seem to provide any benefit in gaining a more accurate estimate of the out-of-sample error. ![](README_files/figure-gfm/kj-patch-1.png) -#### Results: +#### Performance Results (Raschka): - - The longest runtime is under 30 minutes, so runtime isn’t as large +- The longest runtime is under 30 minutes, so runtime isn’t as large of a consideration if we are only comparing a few algorithms. - - There isn’t much difference in runtime between n = 100 and n = +- There isn’t much difference in runtime between n = 100 and n = 2000. - - For n = 100, there’s a relatively large change in percent error when +- For n = 100, there’s a relatively large change in percent error when going from 1 repeat to 2 repeats. The error estimate then stabilizes for repeats 3 through 5. - - n = 5000 gives poorer out-of-sample error estimates than n = 800 and +- n = 5000 gives poorer out-of-sample error estimates than n = 800 and n = 2000 for all values of repeats. - - n = 800 remains under 2.5% percent error for all repeat values, but +- n = 800 remains under 2.5% percent error for all repeat values, but also shows considerable volatility. -References +## Summary + +- Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for + a similar amount of generalization error if your data size is a few + thousand rows + - Kuhn-Johnson’s runtime starts to really balloon once you get + into dataset with over a thousand rows. + - The extra folds made a huge difference. With Kuhn-Johnson, the + runtimes were hours, and with Raschka’s, it was minutes. +- For smaller datasets, you should have at least 3 repeats when + running Rashka’s method. +- This is just one dataset, but I still found it surprising how little + a difference repeats made in reducing generalizaion error. The + benefit only kicked in with the dataset that had hundred rows. + +## References Boulesteix, AL, and C Strobl. 2009. “Optimal Classifier Selection and Negative Bias in Error Rate Estimation: An Empirical Study on diff --git a/README_files/figure-gfm/kj-patch-1.png b/README_files/figure-gfm/kj-patch-1.png index 7aa6b3fa2b9c489205396236a842f335ec52edda..a94122bb3a54876a74c47b0df46a6ede518cbd96 100644 GIT binary patch literal 15927 zcmdVBXIRtC(=VQc1VZmskbp=RR8Rp0z979<=|!cZAXPw0=zkg4e}nh)yPp5K&Wm$iJTH#d*CcFqXJ=<-XJ$XMNiZ_dq^IGbfx%$( z+FEKRFc=aGgCX!}3eYl0v+V@_&^FRDQwM*-U<@!AHw<8~w3edL-@UjX> zvPvqmD(-Z4_jRscaDl;GVCP*FU@o|L7hJDPU71TAcwO_-*Z1-QkEd7C6R)IZud-q9 zvoP;)DerKVpaA`#W@4-+Lu|N8oV#zFdr@2;DM?>H$qPK5NoB)Ho#0u+DZp?p;5m#MGYv}>{=>bXUpGmpa+_^VS=7vkdS-n84ULe(4>D5+$ zuXBObxty=Vov*8ludC~=8w1Z;9ck^omEQXvQhUB{JBiepq~Dq3*-2dMB(3!n>-Q{_ z^^iJyNW>lzX)s7{aF#TNJ3oeV9K*$r)j5vU#gC1Zjg9q=k;=wMYcuJ3Gxhgo<`!m1 z12ZJ>N+Ond5Y$nZ?k>;963uPn{c$xu^BvLPlG(aK^lStpeBqY)TiL?gD zbj><`1ExfMOUud+Skyc6e|U#?39vK_Oj}LGEGTz5??C~tddR~$g3A4RnVF5Nn#jn& zD^vrgQZ080b3jXp_`!rWi+N?@2Q4FI}lv3C}Nh^?5?Z)Nbh4wv4y=*2hjTA$%{Mk-h*G{c4a(w%YYQ9EfW)aKN01GbvmxAB_q1%$p6%H^W|E;U>HTpA7%sBkuBzbX4*hI_ zgiZQ`7-NH;v8g}LM_Bx;HZhwIPMAP573BwDZY|f3T!fQfTT?xDL`XlBNw%MS5Bj`I zy}oL}Z|*K*_!bp;VVDe`Xo3fw^_Vcv;En^Sv{v~G>L<0^%kYo?fHkIqHG1y7FTd3_ zme;yh+!-fTd$=%`cPyfA;mo=5Q0J*~@87gqXwQhHY3El+n0Uv?Pi$}P`S`TeQ_mkc zsu;d2OW09@+l_H5qJhrNf1uGCacb3bwQk+^HtgZvnPPPP5MHNlmm-J32)rn-j;DYX zy?Ztd|6lcqM#FR{{`X4P5W-8rpsRE=CA@Iec8tM4)M&Kc|L|J?2jQcru$ZGe8|$Tr zF;U?gF)luA8Ykr+8~7kjMqybX==yWhTqps9M=}vC7uiIN1ks{!q->@_J9Ro5p#y_I zc>2F|4Z`64%CC5SzAnbt=-ctniMOuw6a6R5A&Zvn$8Q?gR!9Sox(QOKtV;PgmJL23 zCrC4*(1?vpO-1?NED$ZJmDg3NSTlwiP$tPftB&p|Y-e}rCC}kP?uSx6!W_LxD5Z9D z5*nmiMUTB!-Z74zPMNiGM>N(Matt&GF_i63*`T#K2=Y6(tfH$?LbUTAnAZt@8$-vX z9IJ|);ObN&%@~UyQT#qa%;^S89*=g;8F~K@OzeGw$0F24xHH=R>nUc9STA1V%7X=) zDTB}vXxu5}N+LF<-+2TW{nSMR@0C+(LNxOqQa(}i9TUe`K^AqL=kKyQ)?*9F>(GuR zl@!yx?qWWq7VUs|*mc(LJP;JS5<5u#TrI56r1eBlY%=xji7pEGO?uAC8^+nhjEjdx zk~cCu_u9MpUs($(IE{Dey=V zf?&ohGUrrP!Y;P`>FlLJ+5oAbvc!D$H&41Q?j0v|ou({$r@@ZlcoI8*c*Cx09p%JK zSQz>BAuC-H|Cywf40jUkWItM;P7n6J;Pc*{P@b10=4g zjOUx~_;`AfUiy!h4BlFU%hPJTnZ3JqeCt|lvKA-d>8*K!24s9aM+$FX5SjiGJ3t|3 z2U$^wour^S_E`?z#nez|_)3Z{!`MgWk`b2@#GlnhMI1x@+)Db?1hry`o~c&Y}Y>KvDrqQ*}V#Z=X}tu!%U>VtjA z$ieHm;WSA_1E@jyQ^~Eo+iBA8GfrLhVLf&+ec~79{9{*jP2GwQKw;vT&+b9Z!@5s| zl`oqh{VUkN+^NEL6;HEz4B*}W@KT87z0%SVyo`BzS_Q*x@wD#qno6=wt%3hI$!+Q| z^5YKUV*yopc^-B3KaxHj?nBq7WY@4S5NBJM&MY-PvcFrBU-mO4WXQsovgqy1W@ir3 zf9D}EG-=#`_PruCAjfQNgjl(FtY#_|LWii6NlWa&8AE50Tak(vm;$>HsXi5^Q$Aoj zigGQc8*)iG;)~LyMi$nr4+o=cr)_eJuCZz@6sen>eIA+qQ7AI~sd%i0$rDt%Mk3Zk zn=htUFYXNf+WIL2lP(L$k><$rw7-_qM!|OyRuMID_^f8$3{t147ZJ7pKkPIA=e_d( z#0*KOSLgr%({b2HB-T+jdxr8K1|;%H7uy9`AO^#wUtBMimYtnlQwIC;@1M!s`d^1& zx=cn-Fepr`Fhn*Pbt+*8YtiCDW-e{(nS2qQGhypn%l(fo{pPd$$-L%lw ze!Vb~Au(3OtI5-6J)u&~V8a%Yk>M)gz~F@oB~Q%$BIdo;^>3B&m(^Nm z@1DWuV6i+MXUZR5Pw|LKvieyevdT75)gB$<2iC)gG-G5k`YiW_X3whQw)T`H*mUJh zQB?CtG1KvMYRXaybjej-+F$wY?IW_5WdWa8_g^}yi>MvamPBJcU zoD9|_@Ld#*k1sugF8>G0pn{)EXvkD;&={X&x1|+Qjx3Cijw(+D90Vj{&E5mEh4 ziI14k3!yfPKNg?-ZkDLQoWf^p50Cwpy6sO%l!B|0nQn7C+x`iCK78|gZ6I!@Q&Q{Z z#jh(5;0=1%(?&x+lK7heZ&!>4AHA2OMzPURQZt7|bZqQCQ8`aM3&xvM!5?&xzm`O> z95Odo-5uKgJ@_T<;srH9{}FTj(?(BKpzqZe*|vo5VXwTEuj(Cr9jJj|p~n4r%`)js zk0_NzyIl&5e>~~LMDQRM^bS#c?%|g%k$T7rx6ZS-4-m%CnYl9&f|eN6o3%M{joVcd zN^e>)wofXkO%Be;BZ#sTU=hU$h|$A{<0%(RCsr!+AJo~+2P&u2zMSy-?cDr{%$+~{ zxhLagcvsou))NO$ivP(?f?vh!e1_l?zq(ybAk()>riGEkO2N<#PRg7N8w!fAe2F37 zJ6zjj(IszX4I#nx6hz}*!>?=3_RflJ-Ak^?;|JsM&E;gy{|EsdukwiCpReqQiQ$(y zopNN#QVVa6MKC+5qD4=j(YXYWufM{k`xU>!7Ax&dSTx#Yp!!igF&2oo(9qLr#K1e{ zHVom#QpUoP$a&eDw?!A}5eN|sronmJ;;MqI^r3g_EqldYu##h~NT6EX>!`tD)aT18 zW72QNFPwGycYv$~PNG-B8t4`)Dh%7yG9tnJl`=2}Z!B*bG);WR{AU};{1&xQa z%&yA8kVL{6{Ea{4p2j&6n4Ln}g#xz?UHy07j~F?bj938orz5d%9nNb#mVHi)d~(@V z_YEr(WvMVL+O7Nn76Svsgn{lZ9yEpa1jd>2@=lJ<6adpXnGHd0(#M2M>SS_fexQe0 zkw}A1Dg1mEeX8XKKaz-GM8!mwY6Zf#4rzl)7Yao-9c<~+zkEMoB8XZj^A z|NK|8|2LkbRv_ezYY4b{>kSvZ0|X+DoOKu87p7Bfn%RV?2Zdh1Nx zNiMWPVF#WW$NGjH@W&CrY7|=E;QcwFO2TCrB49ALR40%{R5^w*d8ZR{?kTg=qrRSF zl0BqBQqfpCPy*_An>|Ek8nL-%of1cLzH zwj>HLiUQi&j{9XljgdukqyBbDT2ygu9$&U6eOGp+6J-<^z|awu&#@X+QgKwxRSx3* zLoFNo_Gh@@!??b94>X!0*zgmQ?HIy@Qz-6dV0n}PS&}XYnhMo7q}b%5$UmYQUS?_# zSLbaHVZ6`C-_-({2%D)xeD-0jK9U$WbVdT}WW+wOeroUuNxT5$TVjywh^bV89908m zknmvOIatNnM7TO#fkuo9rxxX9GR447~oKpll9K2eeTC^D%;KuVfS z>Ixv`O(s=KCPhmo#Q{iJkV#boQY|x<&^@EGft3y};=pAPCD0`cs~nn}gI%q*hd9C}0#T z;zg_RkV;yO?gcrwp|}t%giql%($tTdq?J>5&PCN z;tOYtXUH(Jiy6-%0}ScEhIL`^4_+=Vx}L;4%GaYC-<-i9iO2rSf@VNG=IWIAZ?*-p z-cNUyMycRKz!z@N0P_a^5BL>UaY;aw6__;e7)N`^3gSZ&83R}eH-NjBuTvs*QH1=g z%pW>N9-YAnMn~sRyIl!#5qH5quV5p95P#zl4V}}l5}NFELa66g=9JhX=9KA%SA*4Q z=JPFj<8KRKG|jAmHS#Z>!hbb9R10XU6TpDIbxXAwDN8_&9o}=DyA1y{fb!F>1<~#e z*zPRb32~3q2mF+{Vlp4~{d^F(91SCTm2B(}sh{z+5o111zyFp2VEW z7R!T+IZ+P29;ygqBK!I+7P7A&#WXo?B=e?Zj?^+D#HE3xsLrtx>UJt}R%t4CsXt(U z+)+H!w5`L66wTVqRIdP-8{UFGl7)zYgghVzC=V42aSv zkpslrV(%5Xx995_%sB`${I>Ti^O@X@Zj{+w#$=D8OIRqy?Ufep+%`)Qr=8csZERKo z=k|A;sD;wPIrj=|ehZ-h79hQSv(^(WWXccUxq*owrddindMhr8$M>-B%=wb8G{r$@ z(W0*V{)YSp;^tT3c{{U)GlvawRpe#pJwOr*BE>QM>MudQNO6&U_ZhGR^76V9N;@^L zJJc@HX!SM69|pU+9Wkar-Fb|{Tvy4fqyl^GGE}|uXY$_}k`-k2lu{RIMC&Klp1+p5 zhtv|7@DtC;c}?a`0-oWF&z{$n;aNDY9)L=Z1B>KF5?!7>$V^6p_yVj>j8avJx+{6BJf0SA@f`YmH$x$|ORxIC=4}^Q_Emos^NgaO>EC zXkGZ`=lJ(im%)J`!2Zi)6^NDNewcCk>z^7?J7EPcNluI3xW_Btje%rnCF`&MKmt zUymKk1lxaq{QN1Fmj`{|RHT3~$@zNhKAewb!x*6;ets#Kq-Z}Ir2@IP;|ui6G$9?u zJChd8r_N!7P({OAVS(r36H1<8bGguadaBOIBEJGh(Oujj%lyjEI|24ZhoeKh!U*`u z3~cyWQ*QWj`1h@BdrrN99_)51oLyfZ<>Zc8k~x#XjN{qv=cH&yzUMO1#^BHNsVk=$ z^TE4zq~z)OKgBFqRXv%w@z}c~H(kn1y<`nrsc{lG2vM+myD+YVGw&H5QrgK#_9~zg#C~ z8dgjanNIn4p&_x64sLRLKyHy6>Ok(ukdo4nFCF<`lZQ_FoV_A7-*JxIB{=u~Chc|? zoIKMd0}WBi10p#HEFKr3$h`VRQJU-*e#|G%nqHn5K^I{98At> z9I+E;q+=~}ZO1N{0H@Y|wYH`O!*Tn~7x}ULLzQPQeSBR7GF}M~=(oFoWy1qI-muO1 z_^y0;t$FFOA>`rQ;pw^ju1xi^=jWbSvw_5Hf$j?9Q!3DL$J-xu3#BPM#9H=e2u~`G z*RR^uR*hbfPd?qZxarHzg87g~3lfZ@+4X#Ce9r0xTp4xno@yHJ=QcvI;%huj9hh)b zBYeM1I@v^2CQs-Og0s1~G+ew%Du5Es{+87t%xGf#i^l}Z4TTn6e;Y50dt~NzXNb8= zkvI85^rYd|q{F1yEzNi9cyS?!Z^@JnU4qVk^GjF^fBk~8Bu8yz((3NO1&+B_NlQt= zVw?1;J^clqdvSEHzA$KDWy_XVyN;#6j&gy?bFWY&HUvT(d%Zl9jj(Qab~?LBOe=0B z@d|k;o#8tN9-bKc{twJA!aQAtFe$Kv2#LqaKQ5q4n3#+Xc|9k>^%8$1REA!kuaNH2 zCo>A5kztCTwu+~l_t&Eqqd-0|v@mPFiYj6=3uIpe;R$&XuzAp%H(h1;PaBUD^u5vX zC#}xeMHh7j)iQ=`UU;b*B{CL^rBR`FGYEX2wAy&G&gy69%EK(xIF zxR499Bdy8*l#3k$fG4I` z0YALL<*zDizcO&ItJg)~E}jQZ(C?5g(0d|G%}M>!0KKTH$C^fz#>da*WF zXN2ZkuvtjEM4m>|8)&f*7Cvq8y!3k{B3iLz+K^Kpa)vfW#qE}A6jse~LarwGy=y3o z=$=`H@QQu!TSq3IT%05(I7|`I&39Yc_39A~CQt{ok-izC7x=!$bo;rjer|J4dmBbp zPQaaRzB}{FkKToZjh{E2XBHnq0DH>)7Q}NLelF&;!65q8;#0#xb1H&RGQ8_i0)iHP z?%y*S`RI0^ozQTlQz)8%E)gK7JRy{_%)z_Pd$C@>-#PB&1CXOO@VDR0_9cY)(W$o= zvpFcjh+aifj)J^sXnCUVVDanCmyuaZr3`80GJT7%L|*7h#iK#Q5sK&^Z~rvU5H^!Ds+37l5}~*^>LrVoG51K6HQ0>tDwzpGr0>tmhg}`5EV`c ze@wG3(BSoX#q%mY@-$jsH5e7B<};p`Nc{L$nnu~~L3t~i^y+09Unh-lc0*TL$mhPZ zh{m~$tP}LH4at_-%iAV`O{MRIEW@3spfpiXU;A>{)v_X`48Qs4fEfAh8m8?H?{>Db zw4Q&ggP-q{k@!k7!~JOKr&Jwz3$Nmu7c*r+G86 zEO{j;VELb}1@HX4%beSJ{Wo$5Vpl;Jttxwt?+qcZm7imliMA{!c5Sy1jR z6cbmP9QqW61n?YtUktJ-_zWZ&fTe|Vv*+RtEY4IMe`tRg1R#bO_n=#Ek;*+U1uH3Z zvQdC7GVYMt=(v$J4f5rHlbg3qDq=ML!!4cLM&(|><18Y@%@!E!*j@5nfM?R3eYfox zz8C~F6|^?I@>h*mEZbv1Y`}Z^=n!ta5q$BiaK&G*y|=Fz*hRQ6yr1)q^BAzMDH9q= zTgz6#F8vY3v%ZxVFm$M|f3-PilXd63Dx^f#Mp7$dX9Cv1?)V53&3rbal#~EQnu`R6 zP~X}QgA9=VKfD_!R@&=RqNGN$PoMK03ORjp2MCor``aG2=G$9kLx(@JhA}q;De}-a ziyE(^J82)rQHg*$CUmhnk__fgd2FK}%Cq>D0FoAwQyTUpsHIDE~w@1{>i=uHE9XpHoudl&oXvNmvr6-d-!? z`W8cn*yUGGhUcZEtxRKHE!_rruARwT@6Ik8qnEZJD7e?3+)vo$rb0jVWlA z8#p1Mr)Sen1v0-#EL+nQ)8yA*!+n~^mc%K&rpZJO(OOOyec*hJ zhxNBo_oK0wVi=!9vFM}Ih|GPbaPB~hv1wD^5($|CTw9Y?J zqM}d2MGz*N1NoSR{tNzXmq5kLCxc!VAiMWR$im$2U-=l zocp5g$tBmYs~u$)j7vfK4_yO7vXN*(kiOvr436@mGSj?s(~ki%@=k$55FQ0EJK>^^ z1SW$)R2qUsRpG@4tQHmz#LhbK-RWTOV&*VE`@jOSxthKCBbw9XIbs78IB=zJW1E#r@nAET{e{-_)c7m zT$oMfeC1zpobb|-FN~hBxVa&YKu{yPlO_tEb7s4=A%8bFf${{aE!ANE3CIazio(`h zzwNfD9w+yeoQBLzB(dOZGc8nq-otZKL@^=L`N*`nB#PD`tDaln; z{UPRUcz*F_)&(&ktEKS5=zsOl>ikhmwZ|Eg*utXaYH&k=gR66eFh~nD2|*p!r@|Ui|G6%&!+rK28-2jfx^l_)s0U8F9+yiu^(vX6UJ6( z>V7Ic{7)wqBwKGgm2k~P*)1{(%R~@`yMMns!HgVJ9+%0PI4hM(K<$iHcje`HZepzvh^6 zu7I~FWke2pX#!G~^!3D8or4_zs1_sg>%YW+*U`Tnix>@8F=*?HqacK^!-J84rJxg- z;Az7P^?zK)(wD+f5wF$#)&tEcvdHu<01x2`J)Q$H#k7*=Rz?jTqsX!NwN9tH2X3Z` z{1L%_EGE;+Xodo~Y%=)-LqC6mGlnP*%TR!d!0OKfQ54-cj*BcJsVF3K3e?i%RawnPjZZ?8gqhkg@ z&RM%z2?<`muidcfnOF068?6w;0T?~k$cu9+OWBy#wW9Ua=0_^d>$K=jKXTTm2ao;i z*cXbqX!oq5l?^a6MP|kXRTRq_q_zKmg9*0^VB4ztddooe0a0zpT2aKmg0+0}W^Eao zNQ>AgPzN=8t>PDS^B<)nYC!(cCMtU!%AkPnFd}qhafbx;TMLF49G#8lOagytcH@3L}xYDMY3v4hK&ZX z@jz|o&rb~49zht4vSEoY*NJ1n-y5RHC+S9PsA6HF$ho0)Yx3wdb*LrZ0+mu*FuPi< z3Owvr1W&+^2Svn?A6;tYTU#Z-wS#=OhAGWLWDzEOYQuy)s^ksUq@DORf#*c^DC;Ge zDQg$<0pFQcRGET1?`!N;8}(Hli(D*2K$F9zDVvF37Rih^Y5{AB(g+^g%G&tx!Jf)M zc47SGVg7=zV={!53l+*rRYP8VMxsw?MSn8s+g&2ooNsxhZIX@+Y+axH`0+qGl6h{UzF%R4I{X;JK=nEvGu={gQ{$?K!Jw>XWaFs9VU#Z?22 zibq1kj6ZwTkOa#o8Fdhd`BI@KyiT~b*&KVa$JK0tc2tC4 z1I2pweV2}nu4YYf*6?V&MSEuL!fbAtg@VFuR5 zU(YaBJiJt>!1!EmL_`t_NiU#iP@xnu)g(weK!%!|!9haP6{6R|QkL8eMbMP9gLq!k zp71e>Sl$^Eavh@A0pDOlWl>+~fz4#kI9*r&{YK&E)KvAGLpL1#IcYf0KYn|nG-Vj_ z!1oDj$@8)5FX~#K)#Z;j2(T2slQ(~%@W+#1{|p1!UNhyasOSDs0KuP|T1yC;nbHZe z@LRWo+C|ZTJP^#e4a2#=q2G*CF9}v|AYD01>*ns>Rd+Xa-9|ZaoIM6q*d#x%`F?8% z3kp-I*RW$P+P&_J>{O?GRa7m39O#m#C(jg;Tk`+AhG2Uiq}gvf&B+@=^rAs#=4qoy z;xqEDOXm|ompE?dM3XmMV6Nm<{Npaoqi%`@r;VtI-;O#R7t!S^p`&DTsuUpaR3;Iw z5;l(bD)HWGY@DZsQsAkm6tI1%Da9$WM7V$~E-@R$3ze{q=Ux4~Z@Ee&dH;Ua0QFPZ z+}xvfP4Z{kT*q2wLUrbkibD1#jyA1JEKHaCotbk(t~d{BK#3x&Z=8eLE7VW3R3g<_ z%s93?-q<5@!0n*vqu%>0OqAj%PO9cBh>)4OG0hhvwP=l*Y*0_-#^B)$A6W5jM}w!p zZH7c=u_S$!02XqYPs!t`c@_EI$lq9!{;H{U9@2kFcZ#d_+OQy3LOeo{y8F@9EszXQ zd4~rLCfeO|6jB0}?b!~w-=`@Duj?_=!Dl10Ky_ChS9Bc7HV1MN9#94bV+%l|f%fa9 zgTjgw><2m^_)BYM5S{u(!Poa)aXi$b^=oS&RpQu_(thR1eE+=-JOycG@-Rg991B5i z$r^cL-2#+V9!5=&!RDwAKrPKQE<~0rNKgiMW9jxwCS%X`w08^VZ=^3B2SJSW5+{?k z(OIPu&F_}b*vIOzGi>)(!NBip9g1XBGzgSlWMf29-n!MZ>P4QS{I55nU`r1CkpA`Co%W>X475+<52fRUqZ&s zU;LEEekiL!8BhG-&dc?Cp&Jdw*H(<#Cypyn+G!e4~y21lOEaSVv8Fmezm2tr0iZCI+jyMUb!-& zn@M3vj-6iXr2%Tx_6Fxt8r3dJR=NgUndv=&@1D-Un!I}!M_KCeK}M@^7>@evXv`_e zLkFVnlN;~{iLYWbEq)PsFWZej5jBIX<(gIIEgk6K=Cu5%bS||aMfK}ufkm#DP}Eu% zU3KOZ`oZ0;D@}ZttJ9_MYYO;}ETw-EtS|o-P3#7hwqOiXU)@OT-TQEJc19#yO20~m zqQlR{EI*vHQpA7(DWWAzV9^ zqUSejgZ7H?5%#kE+7#45E-*x4OYgTL7KKDq5A*vT_$7neDhKl&o@lhJJ#9%E5tH1z z_;f^5xOCz8-mk#NDsImZH814S6IT(b_kD2EaLWVM_v(uvsKd-{`7D*Y|5@)F30CAr>Ro7tx88T3n>C zb7Cz#+1gCt{#Z>(E>__>uh|XuIFSgu{Et#y-}pq#1NW_}w4@E2hn54S87kht9$iLv zKTSlWrU!g5yrq;S{v@Gvt{R&k^7Ge)EQ0{)-<2SL8Dmb@HAfAKji_kMa0l1TCA#Y` zNo1yB^LM`St^Wqqv)wD?s0;|bMWhnL!kY#R8WA*Q#V_-=(=0gkd{l)ly_0B*nn=Ri zii69&N(lrt5r6N>v`e3~pZZOy1p2ZQxbuF}y~q`yC{-HbH0sEU*6W)yzL;C|VsMmk zweQV~+)h(SZ%=)KH9$W3 zbRf}xG~J>gzF+2hSKmWoOp*a?%)_AOb60hFp8O8As5}W&?TeKVCTA{gq>sByMlSFt z-s1b2^x}KFx@H6QyxF*Nzs`*^GSePH_|tA3xldU03A&ewHan$>B0|#eyv3H6vTs;G z!T}gkpf35n9`xj8@Sn1{#MC|v`-Xv<*D$<&qrmN{_g^9>@qak|x#pOi$6R53&8#VD zSc3reg4P`}5GB#Cx**MHX(A1$4^`ZCXmZM#-3jRp(6HVzF0^7Ftr<617swTYeN{#$P zmly!}A_JhIQ{>B*i%G)@pq8`qr58T(jBA6PuvbY-t@vuLvVid=rfHz<7{C|);uPqK z_nEQ5SDs~+;b6`rTQ18KKgvzQ#}(xWC0>wTY=PwFi`s{mJ~d%-EiYuqB^Y!G2AJy# zDG}?Ta~@pV7@lr%mZvm(fGl}(RP-=x`MjoS&Z%b*GeKI-j;mx{@TuVex4aryPC!O% zJJec-QwMQ5IORd}ED8bquH!>^z#!X5oK_6OoUKX;r7bnR@WTi2=d%mW>)z_g2A4mR z6%>+dt4Mf+x_7GQZ-oVkB$s2IS{d9xzEXT!AIA$9B;ygFt$}6tMdPj&wWuY7RFn%H zT9ua;&4chk064($|B1I-V)zIAl*$p=tUqa&124WV*tB!NDf8vp9P?ifKBscHG2dMI z!{VT7zvS>ZT8qDb9Q`PFiosXOPDo;Tb=I=+o?D<8ev%P8_s;sFCh+JbrE^1Z_&;xQ zs#ww=S!})Ac(R)%jsJG{#|NGK{x2!YCV9UW28^dOpJ8p)US5WmOD_d&Ek^P>)C}=D zR3sagI(C-PtALrWhFuM@LT*mKDdo?cDvMY^&?LLrtLgXNI|AEJ^?g>qC}jw5xa$%R z?mY~3ZJo6Fn+!{--wTPbIsPekl!>Fjg7qj@e!TQ9xEXnMr+GK)6n_5h5A{LfrJbJb zu;A^sD`Pt}o$9g|IgIk9vwN+61^0~nIl3V^e-7fW%9Ice*v^l<)Q3-JGCn81=m>zaau{zk>afuq#D#9 zb&^{?QL$Ud{`taG&&xl2m9yettZgBkR&~x8^CuaK#rK9;wmBFBj+9Sx7GZWRsza@C zMB+JlXwjGyP*M4i}QW#LBQW+Y0q+sQup zs&AG*cEZbx*Ee{d-)8y5000H3<)xzqqn>oc8=N<%Wb1ik``loRc~}kc%0bC~)!`N= z!K3A{>tZe~6E2OzY(W8I9`0qeZe0P~>Al-i$N+Uqn@=XQCI^Qb&Ig{-_>!)M6mPls zz0TVlA6rlcl`%?>R-fk8?5`59v?R@qj}j!5;)P$-M!!7yZ1HzS<)XtKxVnBXeeg;D z@aY%e2Mbb^Pdc5;oh66B&~HT@9a}&n8O}2g^lvZc6Q)1~{4A_gse6AU)7STsxb672_tBjDWk@(lsX%(TTwDR7vctEBz@AEB-)Zu5C%p;_yM&5Sbi+aipE{H5Ze(mKxXAJk)7Q94Hrj=@ z#Lt?Zx|nu;pm_3|z6-p)pr}Bd^F{D`!BceT3Dn-`z3IlBNA<77*o^XKgfil2L8bN6 z%uwbp^GAZWJ1ruNpbJTNxv0O-Zhl!u(9ovE#u%E|9Za%%sQg`6EWH0%3de=($NfcmLCWL_y7PKylwdWKyh5?nqH79LSiQ zxT*04HMpGVhRgVv%SpPMgR_nv3guYxkvdvxx)R*CoapGBG&P+x?VX%7ot(fu!pZ5aQ*4A&EV$P@ z6@Y8hsd~#<7;+8}bPiDP^3e3EUwEcQ@hm_g%+58;?p;_fHda$J)=4uq);=~iBDSDD zwqP{24P0CC(onoLxWwbF&Eu`Z-EHfa!> ztU{MOxs_qWl;OaW5g?cmAej-MkZY)sYv`0~7?bN&kek$&7wVeVi_Kfc7C31Z#M&3c zMiea67c6X*#A=odVoO%B6^0rWCG(ZmP^GnaWjwgdD=Wh*D+?+sdnzXiDks2ws}j3a z?L=3NEw09HH5+O)cVb&!yS8AlZLykdvG#2R5p4x;+ZMnz+J@cg%+>6~E_7nCLtYv~ zi`a>H@riiziAr#VPgE96RQ60vgilNqOiX|uTNBuV3GCKnvf?CmYc5G+?sL@K^6DIR zU=9oJ*o6Z7g@TBMwulAn)?&TW;_AX8wrvp$?%3@Tr|rf0Z7jI3*h+J3wJo+S0^1pb z?R|$GjKxlbViYVi(M_iV!>66#nuD%vDh9gwik;Xz+y+S z*m*z@i(SQHw*X4UtXF2!ToKoyG`s9h_gj;2<$P%1dXk2SfPn)?bx8}+H*)VJbWF1aNGF;8L1tB4ka zj}4M948iH8<{+?|run5Jmq(Csh5wsXXSN8o$-l9;*xUN)QrhxSKx6N&UzUyBCd03u z-CsRjyIldxiOa2BPKCq}{(CVNNQVrLhq;DFgnCFsh5kZ?gq~eQK_p6Oh%O9*@oCtg z_eW(JaDRU8i^mSBHu-4J_%DK$&HJ@xZV1s@byK0!r_fD|8rpRFY8CqQ ztYl5a?36Tyc4xQG`ug(6;b{RTWPcVR=L`j;-pq#BDRp)A3OdAQhip6>)8v%Xj+Bh# zj`aUjJWAgNRl9NFzI2dn;K^?LJsS=O%6OuYy#!|=3l%6yyLc3~ zUVN2Et6|jA>6CQZ&g-Dr&ls&$NYWSy!EWa3qm?y_Qqn~#%H&FnUngL_o>>2S|M1{K zBpZyT$%yd?aE$i+ir^!I!F3q_?`9SUkcvcULT#EhXp(l0jxZNE7ZH`-H7#>ISTM;q z;%FpFr%5u18jpydk@s3Kq<}wUPDZK6(s+C_m9dEpdIU73j}*|>jgk)vOx!>9N_fPkZy9N0)r;4D zllJ{$5OS3d^#3gWo5hjA&mS&|^KzX@q4<1dR7@RbD( z4a}Q;YT&xucgFFFPQIcD8=S*<^Qs9JzcY~AkP0)UlqI><^|NfAu&2(2_JRG=!C=$T zNXq!r_tpA0?wrIR%!*Aos;MF<`vMLo7`{!SvY_^RpQ}#D-KTZ^z%14rHk@pn3^d_M zSU!(|bvPjLeH@f|8P#92PxPG;>S9_|-Jb{1t&~t_VsG-xq4MsxQ@glI1`JELYQC^< zA+$YU2^?f|o&INnE&89S1Q;s|=4dqC6R{23p>J3t@2brSfYr#Ya;8~4M3=JMoC?!j zhTU9p1=JyHE=%2K_(PB^Fjbs?~If~3Dv6! z{ycY*xVHIn4&lCY+MPhivmd5WB91a7dnX6eWl^6Ur6GHF2m$*3@T5N@vmpIlc+lCK zcmL|)QYS|u<-Fkm$9EH%lXHZ{+``t_fqdEOZ^9{;GGo74Q|v{eBA!;w`G0~C+L47{ zS4LTgV_`>MRaLQ}ei3J+vl73@@L6cS`glF;A_qT7Up_p?iQ*6fOHHSse$V^`i*U;q z$Vwvq81JSdx5~@fP1JKrozFz-0q5JWjz1IoZE$YpvNzop>TSgbr{^u9Cx%@K^Gidu z;#J}_U+veQdNx{NCK_LhqNWKx))maU!yfx7oZ44k+aRF9TM3mply)|szPd-;*H=7q z&@zj!?$k!df=C)e-D1SJb6s}bYd~C9K+{mX3xIX_bB|FWw-J7i4Sqf7xVmgq<-$b} z!zwXN;OhABq~fV>?Tiga#=%}R!rfw}24Ti9+D{yR8m=vmI-hf3{|#A*#S2sDV1tI_ zQR;-cSJ7`EOGajwe1`YbKxC zrb~7NQ@X@a<)sG@^pE#2BM^_=uFTa`^t5Jf#Ed~+5^6DD539!e3W`GPB88|HIl z{*^;K#E(k47OW>Rd16&v^MC*v@^>$bj?}Nm8`UiFViWZI(AE-}dy$Bptsm{PQIxVb zCZ(LhdK<9+Z2b5>+s8{GyUvzJSA|H!sh3cqdmt;N0bT^ z@(56=uMQ`9+|M_1%zoVX?BD??bCd)YyBRt^B}>v<&~>+vWdP#{B;uogPY~=JO*g=Lz~0(S0VRE(qI;s1?Dn z&#-bTq`Q&1anAY)R_9LQhOBIvW9nl?eoETcRp0GtTUl8lZ_Qn+Vq+Q`^I8z^4qDZo zY>}Bs3g_Eu4!Z(VyfyQc`4`3g>S#2tPDz+>&~k_o8lmCIEHFjS_HW&OZf0k%^fOnD z<$MH~G1m`v-Oon2xU$A1bESF|@$x5$sA_JDJJpaBaCv?m!DEKE8 z3%b?qj;6mCv7~gzju;`jh<9S-n|a0Qy^OP`kanYUUNz4}RE{dzXV8z2|EeK6HzUFJ z@#I^@9lrioTpr7+ccT=8I-;Sy+#?-5qijfXG_&&24H1?pc1=P5@`k1SM?bGo!T3es zL+LmI>TvGk`Aml_nWc)>3HYlqNpZ1Kx3Kv>TPlp?D@v6d9-_W3)mA1pHh*l-YE4A% zyOV6BH`&TcOt!Lp>$K^+bd!u{xLaXWOCde4xC&bIx^Xv9%A>dNASMih>+ z;g0DwN*I4}6KVYStpr%juh&Wz79q96^V%Q%)|Waryi3INci-fT7Qg{A`ycS~ciy&< zWPNjAT}5niYOj~uZdaXMy$DaDf(26}7?1$XLD zQ#w7PMlvSwqlg1SZuj?JQ$J}cyhe%uGUHQ0e`&O*viTB#*9O7yxODOJ_dLmc_-h1- z8NDY7GGk!bD4wJp(K;-6dj-5*k+(&nu5nqRPaEy~jQc&*QCNkLOaQf*D2qw}sNSyB zaGS3rusGlmslJmYr9U+ZVN@CAfpNu6V9G=QXYK>Pf-BBULfD@d-G*OcuMAQ{n}@C* z1Ya;oiHtPqw%#?w?&MKMMJ|!j(t65Wmg6|Eeq3~s`!`uI|17tlX*`t~dqkkKJsU z)({~t27reKs8TX%{A5@4+R=Q1)OiqQ{rtKu+%mWx;Lp*mfGXIWVtdBfH=f%8u7OBD z&?7fdyZh$68m_%9n~&K@1|BHgq$iAkC#w&`7o0b|j8TP=CbRam#PQ%%q^frnwcd)J z9NhMet>U1=!HVNxZ!W?`OIl@iHyV*`zGh*5zN#XUGP%`Z@kCV7`KIV&9N>K%AR#zr z0jl84^_>l;cBF$aBPZVO3dKAcya)~-CUDjQk^%>>XN!PgE8J-dYQk;2mAKxxL)zco z1{^>)DZnueM?epK-^=;vvD}hok-d1}so`e!lF^&QGu()E-b2+zKBvCZ&#q?6=?At( zC4eScAf#c^z8Zb`#ejyN+sg|>l=1?+qN604x2OPGf@IRZ;Ti9q+D~7m&UWO4$uz-2 z%La2h=M3S)b|{GKiY`beUj);D2c+lRdV7Zg`;iNk_IHYns51^oC=_Wi8cCcey0(CO zqeR@Tak+*`lAXOlWhrzj6DZL+HpiK z03xM0B9{kY%xWc~fdT4ZX)B=lRd<(D^u~Z46j1s5KnFO-%eTNmBm>jYruFZ`AMg(I ziBhA&r?2bEEAIP#l`%5=e0G)NMb=2w(&O{f7yM>uzW8J!9b({&?36Y<&0UH6{Hfv| zMn+IdlfbtAkg;p{^oU0V4bJrZK@+|8?vveJl~%4Iw(Du``0#5iUpRFo;+el8i)6#la#w6K``A z7EAr4b7hQx48x#%(M!ET@9|)dGx5`9=x^|dm$^7Rwp=D_ebbLbCBMauQUTqZ_ajH% zqSv0^?@nJoOG}2B+8Q1aUcM_%g?NZ{w&s@8;vSSZpLb?QNf9*9*j19jJ}`D?ndw^j z38jW>A`7T6&-?!>M(8SZwKQ|-xasaRbO$v2$LOd;0yFNQ_+O)AMFEx+ux;g~V-9k3 zY13OBV$+jZlj~acMI5q!t>G0Fo0rNYB$TFrvjCt&a^n-vCVFaQSqB@E#4_eNe*Pl{hwZA(X(~Kh zo?3GPuo&)CK#|9I;pvO2y%({_@(lmB%sd}P4w9e)pnc!XzFr^FLMMulZtrXF~J{SZIG`GH@;-T#8`gSFb8tFw8Z!&jm;SuqI z5Uj?$*ySLHqOVB5 z@mD~@+K~;{-6orEF$Gq5Qq_a;#2@VBLP}`}_tD+fw}}%i);jJ*ORtbxR&c%N@I{WS z^8+Vag$`ECjq_Zs6H7^bH%+Q=Xffye;cv4SP2_!flvU1%U;V502>)3_-zgFWTaeC z6NSzM`}ZP@d_wgr`(Q~tJ}#@sIJLn34&a?m`u1hnEdQmVi~e|7r+V&@syBj83Xe{T zgCnh=`ci}vy3`;H78R#A4ILFr)a|x2G!w+%uC&o#8>+*NCGmQh2367#nSQ9W9mqoSIGY? z<{Hm$|9IfK7@=JSsNRZ$4GJGN@R>!`kreDu9r%1HaF1yHZvwDjX860Ogk&Q)SK=ubxPCF5gPvnlEWNw+uzIO57}oI-HN>mx4KG*x zz4W2g7~iNr31)OK-~0S_-1ggUCfF^Hf`z=40lbyWEMG0Z2M0g%PFD-qQ(!*J%a#e1 zesLU5Js%h`WF*S|$CowHMdx|Fm72vTvu3mypgQlW)B-G$FQKlOo(;C9cQ13^82=(@ z;KI*@32VQj>`w!FjEJ>Nr&KkSO-@|grBE!k#W_9!ZQcb==q@s@okSgP_=2_dvp_Q2 z%!oJ^%vFnTz@>@haO)w`X-vN;ADP69a`GuUkFBj*WT$!o+sQps8pMh50~3-9KU>Z& z=@M$XGA@wMC-dMoCOxijt1_xC^IErx8VMfX27Q~hMGkS_y*T;Gx(d=u6_x1wpYW5a zo)g?G145}7z-znlYrdY5j|LpvFN8vSp9PQ9^JHN!mTMR64!1N?J?R~8i^xh9SFhZB z^8uzSNVRw8ZZ@=+1D=DR?nZjV9T;w!2yW(}$do*C5ur^wQ!X#WIfvA6(?=4QsCV(A z{41abAf?2uO}y8y20W3DdG3c=ZE?i7nUvLff%j~eBMc}*{%^_E3A&$;fUETUR$QW- z;YE31SM&4tB_l7B?p==pu1iI$dhh3WjzJtkzYOYj;~GKUBH2sUag(oFX&&#mDftVlq8%_t7zQLW#y9-@P;T}%;Y8zNTfR<8cS3Fk z^>y#BC9B7dYN}7g*{iR%KD$@y_c}Wz{{>MzxhThPxUy%{{ox0B^4}Lv_NDFXNE6SW zz0WzxU3+x*{Aofu&MD{~QeyC@uXx)1`5p>`F7 zP+weV91hwv-&9HUkhd~#T*uU(2c)&MY0DAm2s#CJhEHlQk36-rb)w?;1Dkd9v`LW1J_q%eIp`H zGz|4{N8|l4&*QQ$>joYJ2()^h7 zgJbZBAz4LHhZ@p7`STkIZ1s1# z4Bg6|{L%3UcoL}uYHcz0d!?0)s`ZJ~fL|Xe6&vqKxLm*OY1TbbnZ{Q6g~8OLwhME+ z2{j@+O+Oa&K{)>+10ElPDSGm(uy9&6PDVnKC-<*ttQ(PL;a>f9N+^`sUy$EkYFSfh zvm2>6<=S@F^v53=-85#FMGiTmcoJw(fNyzqbz&f5aj+G7j9z2t!t(^8H z^}b+ANO$1mg}na&Z7tvMY5Uo@;neiP;N3Ucy92_zWnHw5Qnaeg__hazlo_&E3HeH7 z_xgo1JwaX(`A%6EDrVJYyqgVQJg~t|#ZO}H{JvYAp>Za<`&Ge~k(vW3RS72Pzt6R# z`*+P)$7O4M&ucRt(czFN$#=2-L4v=MGhFd3=1eJlaB@1EGtK*EJVf~Bq4Lb&R@tgk z97Ik4PK6e5(UZn0|Kc&Boa!5JQO01Gxii7o6c&>}Ho`vbNre?NS2ewwAQd!$k}-Oc z5|4<)hhSx>LSEl@9$)e5u!1e<1CCuQOqem})9#~l82zjftx7TZ8!`%4^QCt@ z3?F&JalM6Jr^kzpDmo%iC{>os{mY9o^DF7u!YNbpjzdMsA#~ln z*OlUkkHP7fQxKI`H09^&;m9}a?AtDgnqJbHcw_5nVYQ|cpB9csOoe3ZKX41bZwo=} zPjR&XqNkm!PC~Vi$@!Mj!?0jSj_W0*W8>Tdegg3s8@>cEeLwVT^#kD*)m$Ov2S*)Z z=TSkS!;NO%oY^8TPaTlKi@uM-$VbD(vy80T1yg0e&%gFQ=m~YCqNQwS zyT(wA=t%p6?bjNb{&WgtwzR(X-k6i*;*g~e^fe^N7n+Zw@agyOMgF)Iz#QNdE+YI> z(mAdXr$6p9B1E5$30Fd~_oG*j2mI&0CU2*s@NBP@vt7x!f%GrSw^E2~VMB}ujsymd zF6D(Ys9I~5;^SSG)@qRJ63PkB{@xgUj4xFrNOG`lBQC9#MUMHV3=C`2_&NW#7S{8i z>_ecxy~^0{gOg@w4I;vTB`N&2Nd&~h7Wfa14lN5N<@`r5>Ga@!-s}`Y!G);L{D;un zFA<+Xl8?OpL_f^Fe^#t40eaKVjw#Nf19`i#277`4$e+K#2Fqd690vsoCC!!?EcM_gLid62bDA7;oK ztCx6@qRsJgbJM3OYkb(D0eG>^Pd81>x^0N#@&AZv?rGDh|A}7hfhyd6v{API)%U%h zDZG`9=gRSygl-dX!15!!dy(-Eg5^6CpC>?$4Bf33Y zuyRN)5JOXJ1@SV^8-aeo&=}^xmmrAl-MAh9R<;?IYiR= zXpB&AY}$n|>i$Ql(VZws49;h6Ok!94P1g4zEAXH;(8|Ei>%e!w!GspmL`B|LuM~*W z@>j26HT%K;JhRg(rolA*1Kv>Q23}->F_WF@`ur)EFuO&G3;1o_UVVrQ;00X15&elD zWgKw$@L-n&E3_qBdHSJsae0#(Ie~)-)r1-K|3)m-{B-HhEWoao&60KVgs9`NJ-mZ% z9?=p-t(l%B*}gbMp?`hF(WA$=oub4_(*R$6HzVVqeq_!=8?g4mtY ziNjGK&>ktcCEFhsE}=G0C*3-^U^O5FI2q^K^KuzPIE>~O*1jq^oT8G&U`D*Uw8VYS zIjUmiP6&fV0nTUhme@XH?32_P6#|t?;O0(UAntov`BecNGds%zT&5HQSgs6>(O^7g zZwEe@W;YPk!fWHNX{^M5GDd^cw=Zj*S=p8=A+_cJ8u!6fd&pMeq%hDYAOX40pfpIoswXg zp)txf+wo|62)G}v1&6Y${%{cey`cWBJxN4ZTG}Us0i)i^MZ^e+clSL4dg z@E5@K#r2i8|I+CuqwJU4ywo=|;6J!`efxT?l~N>duW!<0O15^oR1;W_9{wlt6jjYm zKm;6U?s?1NUGdiKgMV_@oY2iZDInM+A$cuS0l4d&hUkA?#z~U+46m6UffRBc5P9KT zm2~+!XiM2Q87rptvnOWkDr-jwaJmP8qxw6v72Ukpo)b4R%IYCWhPMl2E*9=?&Amkv zOmqsur{inyR64bCx^7L1;8 zQP-5b`@qPopnjHZS_8)j*h-NoVevfw{IlSj{pn{b9~sIjkMRfm96p5N;@F}M758^* zXiei&zFx|dkZW)pOWO!tdUTfuI0{sl!iwFx@lb|$dp4>K&!BD)!@+|wJ3DTa3q}{8 zjo|{K@+GfLK56{{-gbi3%)bDx^#xbbaxsB~|6iNfUvs4#T@MVNy*`mRTei}Ui~|5rHwBgR__|MH?i zXnW_SXbSlfSOfnP@W&zhPh1WkAbE*z`}NA`Vak2+^zQKH$Mhe#Dj+daR+n6~880J1++i(Lu_kP_MQeZi76qqq1*D>T9 zM_v?)#vuDfM2c8wB;b=gX}I|PDi?|#5aLupFE!|TsyeN^`M!cZx3PGJGqlu~;pL;a z(6i?0I!JKivJN{S*P60nuOX$+9#xO?ae%(wq{e_8@W(*(rRa+&ahxx=sIXFh_OuCj z8z7^>X@bsWEn%VWe1QHvNK7-Dr`f>{0NCdS zQMkX^*}zjR#=r00k;eBegG#qQ0I2~&20Q=!dMs`h$SK_KSGf-VldGV9hHpDsVD|bu zJDqLH=T9bKymi3bD}`Qou<(^kv;}I(bbXehxFDm?AIMHM%KQTXRrPj^&1$8R>;&|-Q?;x z@L#?Ha912S3sOijAiTMC>kl3NknQd5Y=$>k+3<*cYg-urbO zq_RnaHK>v4F`j%r-WGo}zw&2v+#)#$H!7eVhPdR!dE4C4FYWv4>=z=Su}k7|`{EHG zPX=)BGTn<{FG!g({5T)b;?qQnez6>;tqAF0HcUf;jUk6pXXcvWzH&^2$uGLmuBVr2abZs{ zwD6Vq0TCIAnkd_;jzva{NB2^k{KTueV-ES@>-xJz30(H@q3edL6lsQnwIf#=p+eui zyPf*wB<{oW#Q`546)l-?skIRD&0c@?!M$x^;Crc-F0f-@1AZwKxV+r3tX*U)ZiOCS zdOR)WYN>=SPkOJk#L)B_Z_$^GkF4+clG~NZtC*Ds-LU7S4r@Br2lnbAJLG*e zZ3>wC=#diA`_jqeI&EPTR=zg(x%qmb$f=@b&bt94W~mPW=<(R=7y91F>e?s- z?@Z&L%!V&J!|?w!GKWIlDf;0%5!$1*9V=QE3#ZZ;P%m;H9bEm&GlrGi8tpgsl&|s7 zy|z(f^Fvey!hO$^eT@bg_-9-QHl!H=TZfF?)P{JRj6?fXl&NKQ`i&ini9h@j>9pyE zll{JJ@-SWD$PwmvG#j4Mxa(HWaRYfPy^m)PWFRjat4AyWt zlsNemDEama7vY0s+y*Xx-NZBWN}SjwvnOH9C;E4Ti+`XHNQp0p&i?)3ikF{$u!qs|HAs73spmz zRkbO3GgTsQTln!i*XBDMVZ>B=+$_9zdv1z~ZQOaI`sm-RlE%vdIa%UF%3ikJ@p(sBw*zZB^O)4z zKvo|M3&n#=&#gQI9;EStg`*PeB3#z*0)#Wu=vE^*C$ieWcat340?V|E^eedjiT?g z84ApOJx^37CBuOEW&=0{?kkf~1=j6Pww242v>r~0QsfCSWi zJdflx@PL{D>F~XR23M!3E4j?+xekP#h~0AazTELQ5CMI6?S-dxIL2j$!4b z9`bwqC=hBgP5IM(A3p-VXlcQoXN;GA_h@?=laSZSw(n+tdnH;kE z!&x%Sf{Rb2e*EioOvHw-IUXPBQ-TP18bb;@d}wLkO=w;GO||OEw*io(OFY(B!zL8z zqH8$doG8OjVj%w&-Bsx&eeW{roRqTas*IcL@e5_#2u27Zl=@$}-j2(=1^cOhlRyiQ zzz0{SIN&=!>xNlK2vl$m%wI(5JoG^G>5#Nkw1Q0bmksC+Mw#R<5U%)O+u>$hD=2e% zKUin80BQ|g_-Yt@$LE0^d>DxSPW02#&UsRreISo zKuz~MnGJFI6S6CYz7ML|5hU>P!!4z0acUTU!45bDlIVTg|LjWMcPkK=U|d}0vB--y z`SgNc7FBp>_`tYxTSJfDv##}gxaT_R_N#8Ca~=uhjaHsQkQ6wGmNlfqoGazAYOCO% zeYhL!(c{hVE7^V=QLf;Ce^t&e0!ofCf&G;6`=F00b=r3CIPgH7+kJY%ARx=AjJ}n- z1$nU8U7s=Ov9a;IXBT&62J&vQV+)P_6-ABFEfGe!gyc8(_P-opUtPUg40(4O&N;4y z;)?AaPmAn;lG)%<7*C8&d)E644slniu42fGAIyi_+*x=Q!HVF&%mSyoHG&zCGbCo=Lg5%>J`%kL<+Y0`jtYNXNm)LU-Q>@8P1j6#kll+ru%QjqB5IZE~(!;9v50 z&|p+Y$uE+InR{nlsP}Zbhuf|1pqZy$i?qZx3Cmqvk`KcpdUjbK{ceN2BTy=McBYd1 zNJ!P<`s_s%->oC~&`QI_q)nPG`2+i%b_dn?d!3H7j$j)jR^fL;GmlBM={-H>MOL+lKFwPfp1^l=WMGlqT$a<{QC+L9TwK6__nMfr)s?rWWQkMZ)~3}ZxlZE~oymMCgIsy?~AIJ}c= zg|0kNOP}EV@`vEPODL1x=vFhXZdz$pY)w?!lGfm$iL?h30bjsLIFxMU0bigPE7ERP z`6+X4U5!(N(nF-*(6S{ewv@E5$TsjxWZQ#oU&RD|I*fVOzE^CnpQ3hG;`)~qG1;Il zczMCmujbQ8;FbW84|JFj-@D1LyLSTyb)2?W^|CiYOMTgNEKhhd{uN^mobK#`LdOKP z%4lB5;uNvf1&oR2=$iKA(E@2<R-Eh})R z3Fw!|Owa^I<6K^7}g7(om?cm;jA{@C< zd2z=JZ?_=-+Z)km6Iu1|x8EL|cg;kR+`;JHyr#AEG-4WeR#|Fyio}C{T%SdN^u!$mT3u2Nm!z z@(y}bNm&FII4ZIu#@8I_4P#FiFk6r}!xyFgj3^<(J*YkMV7=?UaT0G8Re)c4q2agMFK^9fLL|Tg zN-bNm&4hqO(tM?_*z33O+<(`MzWX{-v=SRwbQ#e^G>{#iS9mwHb&rBDdpXBcF9R>1kMtvo~k^Uhe%DMedqKMte?= z?e^CR=c@tzu*WyD$7~B-UwU3M=`DRDoNf5GKPD~4!f}d{DXq(4?eRu|*)D;x>KfH_ zfcV-qf8di>_r8qAjc80qCRn>>;(5?2V(f?AH`$oemq+#A>$z2_reVUQrspe=vLcXLTg2oH zv#O#{_#tdaxNTXj8PM~1I|L`qMFibP*ZY$>y{^2Q|EcUr!*A;{5I>T}ypJ{`T&Qp3 z0ffd1mAvQ$LOkNbMOvQtVtb`i(C=OePH`(>=1hHQfR^FecthzuE0GYhu%$ zvwVlzBQjWDH);gUN zE7**iDCbYa(GGodm63wC7dyOvXK$dw!b^4~uKvkKvM>dTzF!;Q6FY>c-|-n?bNNAW zJPW3*IeL8Tfa5R~D5%Edn&Z)D{q@5SMDc-VKlV>2wu>uI97@OqZMY|V_=Eljd3bNb z%oNXVo_3+pR(Hyxwv_#RA(&+xFb9=3nIo`!WQF0`B_6G%gTZ;7_A@j84POLn^Bdcp< z4roESCn$CqEqV;CSc>nK6sT7($Yf;q={?)&z_Z=@GuRCz)f?jQ@JajveB+r}112(1 z%uk(2tCXHdV7rn?hk93 zCuemx;edjrk+XBJtW)2P~twO%s7OZnQ{B#adq3 zFRQqZ)C6)DQ|)4so38c2T(O2mCM?IpZ7)0DYg**_nW7hn2PI6GP4u-6$aJ3gPUCn| ze{D3?wT33pwf}hZ_076y9V#|1)*q(bbLAx z5eLC<6wu>TNc=jM*Dl#XkP7g0|5u;g?!xh)>Qy2tXdxDSy#0Uq26*VX91EfOle0j4 Yk{zZeIxd6Y6Lm;kMN7Fz(LDJ707s&}umAu6 diff --git a/README_files/figure-gfm/kj_patch_kj-1.png b/README_files/figure-gfm/kj_patch_kj-1.png index 2f224813db705691ff25585822d4ba4eccc7f93c..5dbc346067956f7b4b3411a63be6f77af27171c3 100644 GIT binary patch literal 13694 zcmd6O2UL_zv*tU)kfTH;j(~t5C{Y;%1`!F86eK6f86<;*SCF76prA;cL6E2fB?t%z zisYO#C^_d0v+wBl-T&Wz&)s`>@7;5j(=+ExhwAR?>Z+%znkSm-ij-u`WB>qADl6T* z3jhcZ0N_{>0;r~!Y}XR{aZ6KGTLJnB08{{Q4ge$ozzkr$0Eh?yC%2B5+LYpCRL z0{}Nb#SNU}R=kD+02CmB0?be-0ELo3S%;&n^H4ERX_o;28GuU0K<fpyw#3@2RHm>8$S=r5}~2pV??-=VtYJ z!5RRp0SRjqU>y@~9n)@InrB@KeXl#KsX04C#lbl$!Z|A4IjYJz4=MvLB7jSvkV~MP zkEfbX)pV#LRcN4Gn4MdgT}D_3E=o-;${8vSQF#MVjZj&SK>;z=P?3mINE9LSqqFN#ttTELa)tCke~DzyemVy5bemQ1x4+;j^L*XyI&+k)$z`PDmv8;Ow^i7^|A z2_Gpn8z~JR8Oa;LL8W~JxBfFu_2=ivpEG$sag9H5-9K@^e&QB>;&9V>4%2xN(~S|+ zxb>MT=NVk%3~qV`hubJ}-k8bTnEADVYuvy=-#A>U8Llw`*YX_Kk%8-p!i|LEM)GhY z?YQZ5+;kOgCJ#4PiCf6S;h<6lEf$As$KkqhxB(pQ7c>tJw}8W~Lu8H!UHKq6B6U^L zcZbC38~y`paLIN6lY2*jvo&HY9jevNFBNPZeUnB zHjD;g^CEK0WHmYRSpJybnf8L5ojjRBm{>92$pgaSQGMypAC-cX;nOk#rZ2b476n$a@?19(Yj0GogB^8;S&)gyXX zAn?$ejl;<2Wcxz__h@5@#gX~k-Fyll?IP^%i6mWR3;^S+od%4D9`;cGVxmO?7v7x3 zKCj~FIw4hYJK1&)Sk#am8QOU``!NU?-J0_Ti4Hm%QdbNb{VrgNyPDFTA5O*^|ADZzD5y4CDOam-^I6>Q?Bf}w1tk(_u(Hb@ z!P2=tphs-ruyB@F&Bsrf>h<$LPLL+cHHSsN*nEGhwFc8WLCfsb!~23T+t>}j6NOv< zRC{LL2i$&<=Fspn%iII>Fphg=Y{O%G&^#Wr-fJ03{+6`N0leV2)t9~NeN{i=N}fuM;LHax!^ zICDrHJXe0G7~fFvrR#vVVA%-wsJ(r^4=C>4=BV#1Antys&vD>*7tCi{dSjfIHaEb# zQl5}C+BZasUvv+go|WiuVZ7Z1S8%ZbywSBpcJC8W!4=k;kG|)BHkc;(hy#N48s7+9 zp*8-8pG*!cJD_bXhIWQWB;Hp?1_QN3?(j1R5{Wl-5lD1lZ891G3#Ob!aFWnCUg4#Y z1Iv?z_ z2&naCNIsHB@WEh-xytVFa1u>8THYck_#iKaDSoVXn*~B!<+46Ug zAWLOBXMd7vp;X=wM!%!%{#v|dW*1AJa7=m0GlqyKkhn!yiGZ#q*^K1+fA@V*=q|| zx$D0HudXV+$YJTqiKTz@kupU_=|vMumoL#p5i72WLuEI1>{3?SK;TTjE^UE?$khZH z{tM{bc|XT|O#g(Gju*B1_64H(!-X*}j;me-O|XiOx)a1D@MjZW2L&$>+kSh7%cv9y z#5~?-!qO7kb@SEZ`s`ntN5-I(VsBmaig%;v&QN|j5!=k$6auV7s+$ZslyVo0o+>VW zmkC(?jM~~!S5ym6WvO~6xeX*xSDZXMxE3%o$WS~DIC<yp7V(DZ~NE|EH^`ZzRRFR9}o`ZEy$ONDJ?6vIU zquZDKe&N2bjmMz((#93j?2>iEd${zvHQPgu@chEqsk53_BxZ?vn@*pc*4T^?CIu4c z!;5K;J};C(Bwq!wvJy19%HKUE*L)HZS4wCpt{g(G&XI8m)-0iP5tbYHa+H|w8c|mD z&!DWD$=RdMjVrY$qKj)I9dD#u!URt5OpVSv&A$+^q|huh%{yJj%;O%qfp62@dR&n5 zV6S%v>D4PHsA_%@;$P2`cwcH0Z*p&HH zmp8oJt=u<*-qiG6%B~q`-F;v)v93tyZXrRbSfJ?!_Li}VkH=Q)b%}Qq6rW*3U82RF zrr13My;PVpUJ}H~C@sL6QA!sL(OoHa#^82@kyPfYu^4~sto1SazDMZfgL73k$}7!6 zsw|F-3n@jXL%nO?&e-h<>0zJAJyms#ej%a!Xv2wPtGtI8lB6|(2Bbn}F|PrZaV0p^ zu-dO>(0`5j;?)Ww| zAEb@z>b9}a9c*Goj$}=RbW{j0G<87tt{>Iq#`KU`=DxDQwwOP3VnR+YUs|5cpqC_S z^1H))BBJwti7qF7l4VrSDP#oghz+A_-Q2)?dYmoCR( zI4!_!5+ds5;l{xW#7Z5$g$q(EL@x>G*%x~`t^^WrmBoD|FQu(|ExQwzu`j`!^OWK@ zt=FjEv|cOZc)T=V!-`#msl4)|L4BfV0CQ(Xk~8?Pq6-y3S{iJQe^RVAqEVxoIGY5n zor>2__HLji`+r?o{$J?V|38L2b_Ip77e(4AH7Q_qmI<>&d||--kE-qj^zz^p{mk+N zM~A7xL-!-Yu~jBhYN03Uri<-?YAKaIp9kNh36s$9w1{1$b1LxiG}Eb6(Mw$4w%+SK zOk?tvu!)a+evUZ=C}|stmbv1_>1AErQ$CeC`BK$gw5gIw&AXJ5s>*feadTVyWHEoo z;j#z|dY%g{=)8k9=T)Xq^}Z}i&EnP>2H*IObT$T2Pm1Y7L_C?Y746f{^J{pDqSv9Sfr z=PQ1hwxCzt`1=Mpy=>bV3!?NR~}}Ur=6cmt8R%%s#1D5WA)59H@j#ftDZ&tMD^l zzofTZECVl=sr;f2V$WbpmB}@?#q{u86+J~^KM?jM);a`lpFx&_H0RMyvCP;{Wn*bL zXwR{Pu;FS;lM_cceJYhHqD>filSFfb8^Ncf;p3ZG4tt94UQFnLYoj*47Q}sx44M-( zhq_P@^ZE5@_$-{HsLw@BVnqMlgT<40bE#((sHZ`yLQDhA9}|fE$}{&SPbZKUQ?2ID>VgK}Oi#1fThUR7N@qcTap*kT8B? zh|Da)R;ZNROJtD>#!m|U&O|7!I92t_T6i@DN0Mqv;wfH0`!Ow7bv))JZb&E9RK{2F zqKlT3Su-MlS0RAh%@DZ90FzjW8E}!8q0*`Gz!rU9} zSyKa*rYgtY@Lt+KeYf1hNoj7ZkkXLy-`fR@rG+bxSK*arox}7 zB$~Iyq|u6spG{JKG@rq)+=j03%OiPk`0W&od4-ygKXSbz_#MOShUqci^_@YhlS=n- z0|UQptDU_hM7BvH8&=Q5J_XBs$+HCJx4e0D2V}=3#kWpGS05+EzfSGHz=;;`4g)At za_$G)7xlg(ntx(az=7qh(~8Dz-rJe$sVz6hu=d$`fVXr<6B)A^$&IbR)cLA|o3dz) z8u4w78s8^CpK>gLSoL_n8&@`ai_wNKRt`)(2?~*yU1T*x?{rTBYDP{D@z@I18{Q1q z1|i(yQLdEBZrStKB$_E=GH5NKT|X=K?2b=9?WN#P8w^Rb936Jp))~yqFE}k|eX()8 zUpeSZK+pOp@NJEN&T?!z>N%xmcg5mC1#99<4vqMc2jI5*h6XhiA;Zyhwz?KU+{d?v zU-*f`+~`i%+=Gn>`9^o|*kD$xhH-PhcVi3Q9k?4bU}@oI@9!K_X;Zs>A7eA2eZkLt zy`x@6GPh?Qu=@PeTi~NamB=YiqXNhtF!ViZpl3uzV~@gwSo?Atg=T4APYZ+OMOh2n za@Sj3RxJJRfsKHbiFlRabFD2=FE-QHDM0NlI)XTh`fVric4kJLLWZHynWyVfN}ySK0_@`{no0Ch;i*p{BQ0{!Ru6otnwE6& zwZTK9f(YpLCo*7z<@HWjBlD%ybrtY5$uu|4zxRr%A7Bl4u&{zv>|L%olj2`EaCEem z*b+FtR3#5CW-G&9>aFk_J#0}9%|qN{z-r*8Px#Q(xmtS*@Ugw6g6x)w^JwwjFF@0% z%EC7z6P{cBdP6Lj+_%8kz>)&&a9d;in%}n8#6!gs+^siUaBV3{vY5d`o!SaRxAUyr z*@;lUhrg;%8DlaDJl0GD$`(g&sJO$quYpztl?=W2E$amrK7H5rqOiRN&1zhDG>(hK zZ2P&;U{Us}pyfcZi0)W!ou*SrGGbn)Cxv*&Bw?JM6TQ<@2q;ga9;@n>8{f4GW_5xE zEIzC`Z4Q~uDLlGHq^+n%8gtCA)BM!XD)5KQ{Txis9dNtHtafVdVbG}sTw9isRP$n9 z{Jx)|Z{RIiyC;}|d!V*=@a-`DN;xqPqz#htqw;e1p?>Y-CSj zIXJAC-e)qO!P4qpDk|wWhR6q8B%d@RpHHeeRas}o5~tH4vb~t;u_7*&^D9#A9rK)V zh4g(yd_L_TFzH-A79~j6hoqCG)#&%cI{61 zTs$wJ3-927d&0hiM5C1@=^5fW*U+|4hC7rMmP%)smy838@6z6cWMkL;?ohK+(GR}_ z4vS*BdM3O8jm>4$l}Nd$9=An%c^IXQa6z=TBy+rW+CSwVrf`G+wdH;t$}<8!{Us$t z2g1{2ctYduKg}pRK9=@hkY$Di-!4g|u z)!j~)ueb9KB%xXE^;GxQ>xovM^-4234=t}pa>LS?S2myyRspIo9N9ByE&=5xPuX;9 zh==46+5JxgO31QpK!1agrf50P{==Yd9%w++nAevAE$^>p3-G$(l(ZqMB8ldQ-z2^P z2c>mKrPrNlkeD`aREm_jjq_*4lYrBR3J(oe{h|ULSP5m!(z}Q)#|OzsG<%Y6t^PAI zj~bNwZmE=ZQ16q2cEQ;?bJGv-MJ6QfFoX?0{b@_l!)lfKuDf&1_K)PG-EEcMmN{Fzt8oN)+I$>N1;LCFox#aE*iwVJm!{NDcg$COgiF zK4nIYz9;Kl7~P+lAoYR*SZA=R^U*aVF*UcMq#SHy55aNj$U}MX<0KH%0M!NyV(V?m zm6B=>(oP>Be>3iouyNg;|G=I(m$I!H;4Fj2=>`NIx>NlyW7*R&kVUud-h38LX9 zqsyrVy61+IjfEZ_P*4E}(U63fN|yWmj4eo_TQdICmfrkHrm~fY<`B}tmWi<|kzJYX z_Iq?=mGR!t=IyV7{0G{Qmtn@$ll*u@y5i8gkc#NYxYl&BgCOX1dXnR>5r*VEBC`ee0%(SpV-~p3 zGcK?$OhvX9s7^G3x)1twupLkCVb(a#@t2M^Ui_A9BrQzRx%N?2>IK3m;1B%8+%m_%R zkdecP`N>KB{e;-6apD>rCY6FpmYC%xBHKWQIO4Qd$rvuiv=sed(uSINL}n^&-n`{` z@^vbjlbqN@gl+!b==_DXfZ6=HozUgr@<|u)8lWw(L(=)ErD%m?t6P3c`_0o%)pc_u zTw8boe^X-aNW;YjL}mQDug`f7Y;{GOvo3FlePJ0Vf+o9N4N#gG$tozrn$eR*Xs>E~ zrD{IlrGeu$tr=chbKvwj853go6Q!we_`JG3@gp3-4o zuS=lwp6D^x+^L7Sh4F!F<7qD8XRK0S=LYE z!gud%9#-w)eqkO^WX~RxLZ?{r6i(0fn5u)QfSP|dXY%D$h^;$1YT%*9H)0>=>Ro@l z!m!Uz1a?d#_6KLm4m;}Tu*_QmcsBJnQ+K9ct|<}QCw_F)NW^$Y*2^=`Z*dzzKsJ;UfH#4g2D2#6BJiL63wwQ zze#biNL#x%BKQW`xX_gNj@v*eaT>%!;!M&%eFB(N8L|f+;xtY_RnI3)rP;!@A42|z ziYy7+XFSlGAmBoKD&j|GO?_kHT6Dsk<~<_OT*1>n35}S27yeBVl7JKZ?>R-`I!yI` z@7{P9;nP4Nu2L->JSQOs3y(>l%|U@{IS_f{8rqihzdsmr%TYZaBeUu(sp*cl{KEC@ zcQoo}r-y~%2t!1+$M2tWmMi_b$%4f8QNXMGL9WG|G(l|CgC5aUMq*C|P(^q7%np46 z+4CubX>L=_VlAPvpC8aZ=@>W2Ky8z9uHtZ3=k_?edWo#s`pqJmc%zbQ9>`O7QI z|H&42M5ug$EV3pM6kMe*QE zIeI8A5SKNzRsFRUg(en~%{om!nw&}#m5+ZqWmmfi?m$oc^$O!j4y2Axf=NxPoY6f8E^6ehNiNEFACBPP+|(=(5LPpPo9Q0 zh9!;eU251S8Kjgt|5HH#{ z{9U|l#xMl&rb({W^LNu8Qb2&4la~6%#9>x#hLRISCF zL%{7^n*99NQ=I-p5H+eLmjEkd(A<5&ZyglQf*zG)@|P2Px|WUp4l6#jx=CqC-MOnE zt}zzZbRI4EVoGny+M5R3XXYhQnbqHSA?Z`gh0E-VoOVo(YHPs@d+)C1%ec*lw$V+Y z)#O0~PKa~Xb`#icmU>!+E^Gi zP&&sBg>W_`m3;_1{q}Aj7PRSse?z-X_*-^}mrJ#a9FND3jg~71#URP+4ie2xh}Q$R zg^})lUiv3xiJOi@2AoGO^`HIBy$;3*;RwMKyZ!wyM$%`llb6zcA+ind@=XEcS~eVf zTOB0trV>IC)fZEq=&3J`g4zA=cTMFpqk$sA$4cEoZ%*6RLyWI%+4o=0_(Gcql-vx% zoExpSoUKPFVI&IPUe$Q?fP&-&6bC(a*}DxIyq)ZzfWxNBejy=}>!KlOFDt}+QA?F+ zfGNgj`1sOM-tFk>(WAE8h-`dBj--YoCaM)OwH4djhFuaJbh!#0)%K^H=r-XK8=^<7 zk0+=b9f1=JhoJOnBnRP%?{bHU2vZ?>|D`5bQ4-6zUnk(l&KJJN1vM@e7V~Op4Yt&lak!M^!^p0) zS;e85+hu|7sLr{(KY_@$yA}9oX4{(({xWV|>=;_%Icf6U`5^Hj-`B>4nNwxv=$m1H zyzX}a0HE#0zXdocOKt$`24D-%v96Vh#_Oli*GR^=A?eQ~w!N^gATSU$BTdpKkY;Fh zI8q8os#3-z@&~v+u-Ip7>ZZZIp7cc12n(W7GEyvMr`Xt~P)Ml=`Y^L|W)e0o51I}o z?Hw^gvbeqXm`Fs4RC9b~F}+7wP}o0a>S{WC;U$5`Q9|-nMbN-}pD?f7tLG66!V(KE zD%6%}2yldS(g;hV4<@2L!+{b!(3s+=hYsk@`tv0afWXUm96LS{yq?;`laBr$iEP0FB%ED79_ zzH&hDCI}_Q&(*Jbl0)zViN5t$Hv>BCkxQWMcI(gxD#b9E{OQ;dBNWi>dJPDAnZwB1 zFXhU%uTYa||DhG}8Zke5ss{#(;w^O6oQpr&^dC*Or?WvQ5>40N=6Qe5e6J!D68mB< zG`|D{zNQlXve9%#I)72qK`5;`?GVsIcp&k$+6)|z9 zjueD^7ka}6>G2iH7|FBrU(QAVGq)g+b`SP6b&JfQcyP+j#}J|qal-O{VbA}Ti!@Z7 zs%PqD#B+c1_HS`vN7&lgHSKWj=Mzz%f#hutW}cNequ^u2;+JxAEE96-8+mM^eUR%X+D<_K@I_?2fX=JT@x4L$Nr!M&LRQ6q8G(>y;dH_<)O$@bg2b39kH9i2L zaHCc=aigtysLGeC>L~31FgX77CM;1hXZL&XDGio6M*(~`tDXH;>P41(y=+hv;AFb! zwHA2(U=~VTznwIe3tgl`rtdQ31`d7lP%^vdhOjNkWAzjMoh0bJguZ1gFzxZ;gtW82 zZh3RJel_QN5@a~;GysH)_s*NXE-r>!QFrz!${bV-lc`X3exKn&w`olP=Ss3;!pY}F z(23(@%{3q*+t;2Q$@Sre(0~_im~M_AJ(4%}t^9TCrv^&%L>i=pUuZIA!R3}0D&}3A z2&Emxl~5~vZXAacea{=kGv?`^Q<58XnigANLeBk9 z&WN-uY!Ik}bp7*7?joC1iBQNW;r34PeDSjDN>O>IXgYs`DKn^8xXTtt4 zX(WyEF@cANc2QDDfk>Ko%g@7m>HD3-JIsbvX40)=RwUxJ)!7qhYwtbv6I zo#k%B-`Pt1YN2H%0t>ndL2`Qy*9m{%la!Fw|77-0rUvgQ{Z1VIi;?(8M)AL;p$>NA z_J8HE)4y+QAhj1t%U~!H0ojRv_3;1AU6@h>+c_3IP!E5!w|b+lj>>~yH@u@<@&1+&m+EngwE3HG+@EX_+GB zJenPva7q_kr7i`-I$A%G&vH=1#Y9E5R(Z{4`6uszj7N%T#5Rma;hA`#E=k>~iv6PD z`Y0zaSOaE25iJ*;EEzOC8-A9sW1kkgNk&g;ZifL96=8|RCv?eFyR}cDQRP&ft=Dpr zQc>8u7(n0gUXUo&L<#%DC;{`kTkpN+|G@`#<}vw9sqPFRxHsVE$o~>E40(g@HvBkJ+Gox9m;($Lc=IybA+J=;& z5CoD;h_f#{`0^C&UNq#^ck$n}X-4inAK~9|h=JhFElPNtRVHmII_#zP4MEm=9)%gx z3(z1LV!oCuGefjZIjx?^^t6UruMh8MBQLOG;|kP@Si)t$P-wRI<$fo*_vURz-(sEK ztz{X-i>4p&SPKdi-Jd_mr_%KqhpM|FOB+*P8xg6m7;dNiWg5=<*#1B2_@7$+|3v!F2uK6= z@_DgjLy-EvuBN!YB|8S?jhy^k`j&Os)21e~-B|UAVZxRin(N%GC1h~fQB_0IC4uwx zDQ!bDZt+wlkV4@KQ2+ZSjN0-)J%qvQ*ChK$nI59RXLh`LgpN)Va)X-P`dtk?lJ9SAnvCuCY0j0nA~n`cJ0O6%iXp z<~`W3iQwLnQ)c=12{Y6BM(5X3tVwX{7iQp}BfR;<^dmx-Lvp(=GpXal+HNKJyTF2E zd#%v5F+wKq5nph;N^fmY%<$$a}?CUGx=nlKlgflb7%I$*(4z zPl}TGA|?vU6JmE8fZYw*J-ExUpBJ5N$A6S2=xaG}d;Ge|pVuWtBQ#VoNhP&Vz$fIbPP>DP(b%iEAU`PM>CxOUJd8A?k*m~!yGYdhnR&A!ukYYkJ`%s_!<=4!AC9N_hx5aJ9v$xlDwGa*WS;tnq z`+f|RK@WVigPKoE?=Chv=kCWHU3yb$vMwq%eYSIUF1fzL)7x9e*rSXPW+1pFSx9!P z|N3*?*%yB9)Yr*6`{<}Jgi^wl%eJC7WyxoAD-RRsuTuMN5fqy^2Q_mHbrW#AQ8PpD zwI|&xC24-GYn2^%qAw6S9Ya4h?riRye0<=Gr zc&cySPJDmIzI!+fAk4=|5V%>bZ4?EmL`9dDK5Tv^;T)h~Hss&iFWGeY{>NMIGz_+A zu;^<%QUa9?rx!ghZHKi_3*@MAJ+j)mV>riM^SFsw0UVanD-ssAz@C0~JK|Vb@rc4d zl3&7%Wp%JqOT2gS`}I*;l;|}UlhU*rmxOXs z+@k5w@kx>E2w#QDRx!ktNHrZX@pP?i??V=SF}uH(aMfb_YF@3-U_u%YYSYHyP-G(! z4Ajfo+Pv!-lKwozW51L<&6M=s*1k0+yXc&1SJ4Ae)hsFK7~ zcP@|8Vf%kia{`3U6|jwX@2S}JJK ztz;?AwP`W1R1$sb87w*nC00*#9hP-WPXZQ~^8+7ji#>xLY}<*8b{)T8W+9CkD&sJ! zc`}URJ*ejqNKCAio1gr`o&o*OMS?O*tGX0)n8gLfj522qJA%A5lV7#3b^!_UkzG2S zVZO{V)r9oR-D?yQ6)agF++Ujdf2_+9*7&wIKD40G_4IHl`d&b92Bmct8;RzFo}iz> zOq}Myj82mooM?D|rnFPn;J|qiUIiK~49;yf97ZMk?5Q$ICmvn=C-Nc|FgV!!3_$|q n-U5MZ^8fNlha-TfD6Fi(O^;2V6oHCP}6wRtwa2C)X@pm zady;k2ImZ&sA^*#$XEh0c2d3Pp>xl}`JPAAy{O!KSuKz3T_1g0wt*lUNZJOnw2_Bw zb|9F9>@ z;0TS1%81IXj>;X3Y5~UvMjpc0fI}K%V~Mc|!C<5@7)uN$7gHL7DFx>)Oz8$@v49+9idvr8ZEhjdUpn9G0b}A*H3c zrKMe^qq(J{;Ji^v*eG{qDJOg>Cu}s_({8vI+0agCOmS@_EH@GeEm1lxQI0LSp)I)? zEiD->bKn?kA#Ai~>9iB(+6jbTUfRFr38NV4QHc@;FYvGj!NR0_Z!;0LSbxdYsC~r&gCyc$*A@2bsfua zQ>*}&LKumOrq*|fCQ6F(G+3*1;(&A9J7cspH?>xtY|_&wFrkmuQOCAYn6r-~~*%Vd1_le;@R@ZU_B zA=-~uxFJh&7~GQf|KeZU1--{Zz!18Bh85<@vx_;&*}HUj@R+*k&W(jKkfYz$!I(~l zxymsRtfl@t#G5ROuUiTxr9_@4CtKQRV5(Iq5~MMGdz6iLdi?lLa@`};hCtm%N>1$J z;D7^ukq$Q%%+T*(Wh7mp@|lnT8ML7xrF*XvFa3lro*K^S&u7YZxZ-`DpR#9~ir`wf zD7_R^f5*tfZwT!O*FSdxJMt_^7|RGJZ*l2mE}8n(TOFx1WAK6xu}NPT@Ky8q7?wd< z{^`K|O&k0DkX;LA^o?a2{k*+|k}XPiDI`87&B`%7n;!=qdYsB?Dy2V~%3%J2AEvdqXa^PnJaMhoM z9B1L~PC6HdN`6HMwuqks?; zM8Tg@;yD!p%Q>Gy$MBs4PWr$7e`knJfb`!@K%A|7X7x!(8aOFy2URfwsbBYBeGwr+ zVb_na&$nzwoKr>}BF?$IDE4Fg;Mkj~?FJ8`)Tf0@8N6R0B|nAbAhpYp=%k|MM_#7v z(SMY{%yAX{kP(~KCMA5S9{ z{59n$x?b7V0NnO9+*oI3E-CbUmP{E*sZUtkHLDASE9k-;F04 z6+;}@G4hXAEiMHyuYXA#up5V`j(nfAJO}PgxfIL@w+2Ewk$%=k%O%2%47!tJ#mbHk z+It_QGV>t_*=)~$Pmd^Mr{G!z$H&4y*ue$R%mj7nl zR5sgw#Dz`-T|#F5`n8|0t#6{a#HF~pEpFvr^uBJk@zv4M%1Bx6mT6JR)1Bf}Dm>kS z&9w5yt1e5bKe@wajYQ>e2;x8s2dF%K_p>q7vPtLT$rEq}s4o(55F)q18eUyJr z8L6o^%5|94a6d+DfLtV6mFpCuM9ZO8ybjSwA<~He^8Y)phcr7K6gX6L-o2v|cI71$ zcf8^yrMn8t0rpQ9mKwAe6szYCLITU*Gp&CU$FUVFx2GmH?dM*3`%&rH%gQSfuX}OKpPO7R)5^O1@Bw;@JM$Fm zc4B6WYcx;h0aU_%rnOr|8QG{8O9pdL6!Oq4f-2OZ^Rp1U= z?M|S6R(pcINn(EPc1H6KarzY+8BD%rx)7Y%7h@WTkJu>5noGw!`d7%uJk=is$9I-%pfU# z2Xw^obURSg;`!X=P_Mjm|}I=%GbFu(&H@ib(gtn8n$5t!wE8G z1salg#Jres#tHA|9Wi6)pkER>N&zvi1$SMJrMSYt>f3126KF(~(Aw)^d$c6@5UA11 z2NPt%Go?uxo_C8?MBi~Arc6{y#FAfMMYtb`K5)wYt-3TYBq%q-&WB~+lm1coaZ7ax z2h_&WpGo`nO_VijK6^VeS&js0%x%ei0v9Dd%b z&YltG@J0#PqZ@2f$9;18hwV>?$_hOjPfarBBC4&F$Mb>@DofNpU%;vkakLG);ye|* zGk#X*0+!V>_l*&D()c7;_@`*CLH%$*-b(skts}E|TTqQ^c`vNx{b+o0h2Jr6dGlh=g_rw)WVY&}U z^ry4R$cMIn47{~~SVY!1FtSQtw+fG52AvKG08edyvGoATPl5<8M6s1F-Y;lW|5@3v zAoDmSc}%YRsht$1{_ZPk{S{{*UrBwh$z!MEG?jn&ukusE8dP$6m-)WW<2*wJqZ)xtxd^Dep6Zod6twiNa*hn zFY5mD0=Vjlemi=8c2!LVPD)e`MueRz8VmG;80(NynlK`A5XheN3ow&$C;XI{x2wh- zpCJ{uy42cKXdVrzM#PfZ?e0(3KG(=a)@<{6>*BZl;BgdSu(nfJaBXwAwY}z_@7DSo zq4>}$6q*M^N4S!TFw*sq9c#o~MWx(RMY{r$8e#Ba`68NUR=EnKB7A^y+*#y9mzDQ> z?=9oC?`!|l#vVSm9_Y89H{IH`X$OyE0Ay~kV6Q)O{*(Fd&=O=zfhP>@%*jfZ;Bm}< z*+9HcdFC`C#E&els5qNnByI6jtW@yY zrQKwO7d!Ny?~Bm){|GPk-r0S>)}jBt_D<+|6xvWLi#_AdZBo0kee!PZLW+T*7&@3M zDIot5?!Kwxu^zcO7=IaQ;>^5dbOhy@Xas~44SD%dTeFo48oEmYTAm}3s92Jeyz#U(Jx!x!zVF`=sqjI(U zNJHNvyjMF$-6KrcYp<*2BebdbdpB46hf!h_J~3kRCF#$$gnbZ`MtVvou0rFSDMxdY ze(F`#Mk(zF{kP(7Q{;FydK4pdFSw_{D!v~6Nd}FMSMxnjX5*TNt&{O&+&BtqtG z`H~ef8S=vuJm^nC-JD=@9~xj>-$vx)5k~ZB+%$j-Lj8Kgn&Rk$^oS~&=k^gqS3dOn zO?iE??`Fqim`3*m=jouc$b1nO^vnD&dtW*(o*Wi>7Sd5%KOIo>wOaJ;W^RVa2Ri++ z3WviA9_UOL6s-}`) z!bYQnLiN5byBv>=(r(Bivg0I?t%iD{MsweAJJqA<8shRu0c+fEksIeo;=KBcov)mJ zpIQ@L_-L4l(IDaesNrnpgF+X{P)Jq7n09+KZo5=2{F|EGIGPZdxUty9dFgo3H4FG`Oh%-25y>_1aJ4<|KvWdvpFnf`mW$uVVcc1m>#bpU z=5E^-=TiC)eW##Tq)S^^g~4OrNBAs8n5&*WJk$4LyJP1?*UHQmQqB>$PhWQJe$7fy zs;}V-}H>1>Q3a&*^C>SS>?4yi^r3KHS@$@m)f+Pb--A3n|bN6{vMei zQh)j=iRGJ5c4S!f;bz|N_~j$uLu8eO8}lda2kusGGb50fx;Ev``H<;Vl*(gL(KT@5 zK)$-PpQp7342`yC1(26om*p-Og-4rNn78Snp*oKE#fD!uPLHxLZuRwSM-qu_CA~9k z4>{Y^h+{;&o83ryXVyt!+ti6WC=$N}3#9g3Y}oLbNaga}&Hz@D+IF!4mr(O5SGK>4 z0T##wE(7oS^;=$xZv{y|YkpT4?~44PeIGA&RpfS4>RWkD66?-3U4J#w7JE*V$?q{* zj83k?Ar&Jc`w`CmqniZMFmHWM7>Q1LyP5r=YZ`igKb-9dFs|Ev(06RuWz)Y znKWdqaQWLMVl!XZ!=U;bq%HS7&3+p#n7mIIjX&g8;@10C7mbm9I_81@K;+Jh4zv6L z$pE+hk`Fzmc~(0&+7GCjDZTz zzw&Z}NmR3Nyn^H|61i1?rK=eJhp@HLOVQ($m_7-7@vm|w1U+;w>fdGppjqj^&EhR} zZK}SvH3zx5c@R2;LK+%h%Pdws;oWI|RzXrc_U;u;lmSymYrwABxJ#7FKq8Cn7iB4< z+WGHuT}>Hfb5&^Ak(ClyDq4%yWo4=<@2K=T@}ED?ukn^16Ki8XDdYo|?}eL0?Z>+b zoGxw~>t-hlB(GCPisiEB!+~f?fDQ;Iy&iZLU3+eayu0#9UqF_;SYb@Oo&Dq~1o~Qo zzQB!pxt4kUb^LY9<({iNc)5`tKK5@+*zGlE>BR2ht~_UH)RTDBOyEi9f*Wa$SL=ec z^aWkJ{L_Ch7bm_>BgZSxMmL6IRdk;SA{WOm27ayRPvCBRPqU{|_53h=y~gA4Fj`R) zO&xEi%eKB*5w-h*`>LcUU#^Sn(!4K6C<)RTbX0YI`vs3bK9O8hnQ@`c!>^fkS02Cb z`oPJ~^HDJ2_3(!BL8E$+k}Hj!f8WAzYx1@nUr3e;v1c!J`J@oxl#BR5M5X+C$G55L zLO+vU?ALYoYdp76bg5?l6T@3|&8%qJQfUw@22EL3Ecbk=E(dm@UbKcF8eJSs`^fA` zW~{aD4|U~85cFQaE9vi_ogA;&7G1YO?liNCk=bbsB|JfWKHT`7CDQBdsRCwWyYE&f z3(e7nz>Cv8Xyp|tFTw?7f@R4K6mv$_cpLLpX= zZoI!wEjX=>Hq<;~{x;P;DPHFHL;WXlqC-50qgh5HcyYzylXam!21NI^s8+0RB zRbo#Ck?en>od~21O0;X#%%@Dm_-L=9!ymbTg*?Zi33_oJ)S1`$-CtDc^*P@Ze%7;| z`++Z7TbzvP20ND1<=4u$ITkEw5lS4BR00Nc)?&pT>X1^xOaa3@ug@bZmEbSXABbA% z*h^|Ra6B&$3+(O!GF!}~?3qy#<>ThW^4c>ZY=?OeeK!MGKoHq9s+cVUf<>5ifdmkh zQY(lImt)mQDPto61^OGOu^a0hI!(1n;;>KTG$ai$1iKIehTys2U$-rG7Fi&R%!-FE zXZj*Q3WWJX!+7$P30Te*IFMlGak=Gq@ZsDf?&->0GVtX7!zxc*h~DPiVWDew`VSeUFpK z|BfqdPjaX2x)a%T=k2lMEj|iURaaBs)I0LQoNecQX>*vdwI@Z2+~hfjbnrQ;Tq0X{ z_0c2T%F{%zoTNkKqduHo--|H;lIyA=zFCoTKI~4-QQb0$;-DZ=e;HABj|IHdqICE# zc}m`tH_;4j2PKwKNe1cv0Iq5ZWXXc6{@O%*4Ax+u;_1$hEI$kRk!k;qK*>UGsTxKZ7K%_Ocq0?yAF^ zlMiTeNI{~~B7}4~+L1ZgcrL6Rg<3dTuhw&>tHvdg{>M{SUty=nF(&G=kcq847gyRG zT4FC@B#KumTNJs~)2&%ghk(z?f)vV?O27M>UV$77>Gid!Q;7Xj2)h^*`bSWYDtd$> ziwZMK?1^I6X2}_TeprSNLH*RMSN}=c;sLU_ITWMG-=NT)0T}csko@7+Uo!B$T5>rq z=A*lcHUq(_2A<(6?P;GOKLfA^qCu0Sl<-r^$1^&q_$UkPz_tXx!XwUFT3P{QPap_h z$Uq`QN_id-?cqfjko>=a<6K>s8)7wjZ%Qw#ON2~}n$~lY`Q3^Nf+1B_SerTVG96ao z@~3{ECUw-&`RUYocwFVH$y*8u$q3&(8DrA$q9QTc4|oyO6~X7fRhg}q00R52N=^9(&d-n!vXdr1w| zme(VpM{c7G=F(FgrI!0QYcBesqpBb0f|?ep>7S=xSHJCx<@W2Y$}yAYu>A@8@2qw< zB@7)`%=R)@cDK~4aOa-FHl-9sOK}k#JLR(8lFhd|dkqdYGrXaH4P7*1NVDc0 zzyBiG_G-YVZ39|R*ztG@Gdbvyi|qdOHq$3Ix|PNFMmIy7gd!)aNduXkLu-(x-ti%W zn2wuY*5EUm z(+H{J&~Y-~PTS^EG3^gNZ@(^=jH@O1-+uP@2~E$oqvN8~-mAx8?Dp-9MRIi-dNuWk z7|(j~%t~dr0Uc0>22#6e6#4Oen$(+aexx~ECT{!|5wpSM;>|FjrxawlIQ!wVE7!?UipwI`KttmL`JcKMp*&zq8yhD+)|*?Xw=OsCi#4F04(NeZUSuE@H<)t;rJmv~#2`KTwz%%;ckgI>11%YI zQ*z?=Qrxaw5E4MkdfC1l?gD+Y=oHQ;XMOA!MydUZlx-O)8t22}# zQTet~RhhbyWS@#aK5IQnQbjNx%VIyE_Db`TE2r7{cB@)F`NC_J=b^}~lTvl_*hSi| zex>QyClBd=RktQK)=F;ieUHaEA2j+_weF_uHyx}9Ap0?MazS4oS`K`vH6aJkX~T{a z+ue$PBTbe9Ou^(5#7mzVe%Z+4@SB&L7Vaz~?9#2MZ8bDZ0k%RHiaT7GS~n*< z`dySS9e&hF9pUmRB(WnI zI<%cAk=!SH!Tk_ay=vu7c0Aor><=uXW&l)m{V4vHDG4(jh-J8etQ=j(ojWJMmKagu zUV!^*vS-9V6~W4dupDOMUKMD5C<`P4B}){v)Ru zc1ZX`#YZtW8GL4f5h2BN^5_?+D!%QWN$aY%#Lp+%3VU|B@o+`n@~l55>?qRPspED} z0YAx-xm-Yeo~-|zVwhrWav=-sM|F91)kSsC&-ve|B9eb z7nU%!=*j2Mfts3WR)~J#TpePCw0q^n)3eD}L%^#T`|e@Jj{}&)zcVoR$WcV%k&*40 zXq3~^n;-J^sgjhL#OJZ3BG)@(bkWZag`=z17zQJwE#j7&u$KfTzYbK+6=8&im8vv;LnCdXaSAx{8bf zohvY*7z9$uvp$4dNjz_x@PNJJ)Oa)X$*IGR+g3Cro*%rGu!2maIg0_Id3TN*QrK&6FRQw9tHl>oM1%0ML=ZU*CNF<$n{a-+ z&#~{M8sI^J+v%olcW$0`l1i~?7vt#VX2V)G`*-IK4s>}v>B*)en$;EK=@6V%u&t}i zKhGoco{nuyGiMo+haXWS>EwC#HiIai>GC zNFg2lcUHk-rP^FrUBCX-LkIoo3>WseNm+46V>!SWnC!;oi?*5NzHgwr>VfPZD6aLL z1X_b7UDO3I1*Cq=Ztw~kq=D4;<4zOt^(l?zG>vIkI8H02%VNgcKADj1?RDj<=SKQY zPx=z%p~PHs67S{3i5UEQXA1JDX4793-`tQu=1&v**1>-JTxsZf)~&o(!&4v8s~(X~ z!s4g21V$U)eO<+dG4;CTV|VKbx&qIC?O1%Xs<;bfeq_$l!kaAT{qW(dNWL5iF6>%u zLwx`uqGUf@uv-x z0sx?WRO?IpCiRgo$+)^>u*LeC*eaW4HdNyT&7UaUL5L zM)9joRk`2$3;O@jTnJev9zl9N~$45G0`i?~f2BYNPA?^5+NG2=Caky{;)Row3_ zp8QMpRtyw24AocLEd;*+Uvx)x1X}- z+Rwzl_iNfF$wq~dq|hp4S)(9{ zWbzD9ss)l_b%|;YHn$lCKqn9y?haemFN9Tq{G3nDbJlb6{6s#0Ll~G5%^VCpQ6hab zR%yJ|+TuNo!Q@Lg%^E(cp71O_Y`@C4n@K+Jpz_9iXt?SnXAhyeaKF;P8f_?*7y~5} zLiqr!;@>sN<`V_l{DAJ2`?HFPDNFmOQq%T?FGz~=!%R-b30Kicj}(zl9TS!{B!wP4 z|9qs&>Psuagr&jHmA zQeB$=c@fJBiX}}85LuhqnQ@tR>kw^Unb{Zp&GwW`b*{Jk z8t7-f6F%Lej3{EY)#$)EoHDZle2m*Il1WT_37U2KBmGeo?iEn}$jXLg7k?h@RVn_e zv*yy(bQN~qGsv|wNUBq*UX5fTL(2}0HiLRm(a{!yD#4IkDD(*FCr?d-QN%HyCiSr& zZ?#}&jV0qp((@zVpF*U*hqNAeQ)0jpC7){^5;I1gX5&YS-H;GA7f+Kj+UL`F)|%GT zu@v8b)q?|IC6wzC!btjmAFt@-Th!uztGn%*(rd3dDas$+Q_6u$f~_W8g)|HKq;9<@ zObLPn^5%h>H1ix zx%uDe$$0-Jx?hHe=j3-n4m1u*qoT9Bb-G#$nSl3*5NqE#wjNwo=hxQ?i*odl$0>r{~j#tC}DcK`(v^cO+LQ;z=PYQ=N)l8ya>VZayb{NL^+i4KHX?#ZbG;Y)-O<3f0Z`@ERdfY`UvI;R>K{x+gdbTg z{_@-U+~iqg6aYWcF_1?sPqA$KWds17{%MYQS*w2S9~5O1HX=Nu{?G5}O3ON6ZRCMv zr&gh?b}{qOYZ52TAo}O0GK!|iH89+d7Lrp`m~A2-3m-YmQ8C#7!UJ#&nclTvYS8hY z-2iC{$Q<(ja*zCve-B=R9(Uhr(g?ctpIyX%$9F)m1adn7odE>GB=4ES zvLwNv5`5VC+s=cv$!Vg2cVQ!Jr-Rny+?qcKkA1%|)R+Lx8)64|7CcTUhgh(xqbgg) zV-)S*?0Q>)2{vM^ zP)2yA-7W1cB|n?4Dv(POC#~^YJ@HW5x-#$Nb(Y6(Bdy-4@s5o9D)Cjxwryw}xBmprr@kUn)9Kzx0k@iC_-)&Dd*R_dnD@_rAQI zzj5wItWPYnxKBlAvD9tx7`Xw`>+O1M8khO7c%^rOhgy=h)dJa41=Jqe&6^&e>I{4~ z*NrPmVYSGSRvn|XcP!^VUwQB1R09@>F)udi#XM)#Nc0fDR)y-715qK2Ki#dkm1s*+ z;M~PXMe43MIt923oeMGH2=Al}CJ?m*^dkKlXLD2VgQ8Kui;5b$Poj@5tB8)nxHjus zy;ci?lVLZ?Oy)V#;CEQc3?HwzhmH)k%!ddOk=8TF(aN=gwgNVLQQeN%-!ZPRnE}}n zG7h|eK8PQFJV(bFCCJJ0G%?C_9AY|o>R9{1@hpB_f7VXrX~euNn2P;0e(DV{6jmu=)%)Y zn9uw`M8l~0m-XjeHG?NHgzlRwlo-iZ6fKd9KHLUo5fT?|K%-4H6iW1!uE4jc^r=Zd zVS+Twl?+y7VVm3@mH|6@idtW}E{v?4(|Rh`j81x>jMS|fd;0(|5`Utp{~SFxo!X^} zjUOQ6M&eUAVXiOfU}{wF81%gyQbU2i2S$x;!L!cVBk(2_Mv`a;{2ArPNJa;D(TE`c z%hJpOi+)-Q1QJ2S3x=z&lm%m;zZcAUjYGnz;hSxFctia;m?~)tFHqM^>Yf>P|K*<0 zm77e6Z$#9V1$K0-xiTR{N~sL2M=XWi=IL$$qE#gBno&9I8~zC!`KgM&@d~sxXhu^q zs)1qVIIw?cC>((~1DX!ZFW7rxR!{?J;K}uX*Jy5T4l zElb>qR0Wcr-^Ze5;mpUl4Ia6}l~t7{5_Te3?S`(nkIO!}ZN1gIv^_Nqs=yar>GdZk zr>IK9=5mXEJRmZ{i=C*gW5%(j2z)uN_n59jd;@50L=wrbW8q7gvH~#qkXqh=_88OY-5F3iw?3!L&YX4b(8H~r`%jlD;xFHbKkY*P`wALU` zN>_1BT>RjHul*A!N0b6TU+BS1aGpX9uD1TdD5(sQ*W){v`B6c_xRq@Du)ypX-MSQ! zTL#7(f3f5LC=URj48+%lWD@tOJ`+tg$eR9}#s1$3vPBa4v$jQ`>VO&lb<)CQQHAdn zySh6MSRiubB1cybURs>KPnM$yX4xW+92X~>bD<}L&D=#3%NP+j_^*?Pp_)zNvCG5F z6Goh%L`d9Tv@jtl8Pl%sEk~ZSFTU0Cy3#*39)3PR&`o02dqv5`ZU24w(M5{0p3`A9}C z%0!^k>D6Q^NE-Y$qwX;L-TJp}E-#x=P&r)>wO#|MhZ)m5ZhoI`GPwgAY!DLt|23H-(&uHTs9l!>nb8Z=34OQ0;V9 z^2o?*Fz(SqyhE$+L-zH1hsOD^k=5$DZHcvjH^Rt>+T+}b%&=KKJWrksOD!is1OaPX zrI|ZQ?(}`wNPeTnv>5=WjbA2SjXnXHj3bPieeB^KnPYL(z6D|b2H1|e#n9t~582;N z8amX8=V?Plw%}OX+-X}n2I*@YoWBJaH_~ z+$O9~^M4y&kjCMQ(7V>?UJ8-*V`;b4ArfM~LnhuF5=JKJ(#M85c$VPL(?iZ|VLIU? z$)M^}?7Va>UI|$jQ@y)_x?I*@h6(HRWOusOJQcqX(^EUkc|3wTgt=^SNizJHSUs6R zNpA(`GGd>5T86(HN{WUZuJ{f#yKb9Z`hNu8^iD6`BN4Az{}dRXA^Z39n%*C86P`2$ zsZX5zLcKbk%gBDLB!3wp-JC!4>A}y>2K{okEB~B- z4u1qmIN6<|OCC7&RY-@2@V!`89CeWlX2LI)lMFWaQ#1<&~$g5j0y_)%{D=cu&uH@z4??6Z^t)~yI9N+q{lA&_ua!PEnxhOG$(JbV( zDW%V%&L7*#!>GkKa$Uif8uQ{M*ADbn>VqWfTBHFz!>;a^)-bWb?(}07@fuDX9Pcynl(j@W;AW*pR-1j!_)=CGFEv<>WXZ$y4 zOz%sc?2HuZqUrev+`h-~U+=h%TXybDFGN;>njuGz*7O;NH8OP~qqBT3=+`FgQdNA9 zK2klM;4A6+V?Y9nor@!_%^nzN@;_Td=I#_RbaR~sgJSrd`z0=Q)SpM+m;H}7A^%sN zB0Xi~l3lQRpNCA9%GHp@VGBMQ&AclLtg7gu*UZwC(J!vm_MYZ2P(@2|NZ(RqrzK&; zQo=Jyx?gI79nQqy<^M))I$azIJ~2yyV904oM#$=a{&&b)k0jAIC-A-9JIe4(DAFZ< PPOPP&do^FpGVuQZAbO_T diff --git a/README_files/figure-gfm/unnamed-chunk-1-1.png b/README_files/figure-gfm/unnamed-chunk-1-1.png index 0555528aa1437c4c70f5b44a3870030aa6c60972..47ddf7c697ed3a3e5abbaa1362719396e103f30f 100644 GIT binary patch literal 7163 zcmc&(2Ut_twvHhPhzN=jr7EbPC@6IZ0tZHvV(5s12!jwHfiMWt%S1*&kbodkq(m5M z1Pe_FA!DIR2}KcNKzcwD2)%{#g7@BU-n;kC+)?-c`FD6N`OFieBtNejv%d+cX9_5EiTy@a=$_#a0o;J0?~j#oFTGs zh^!Vw%MXIkfFPV9h-iq85d;+tK@ksGrX8{@Ib=yX2m--vqY)@)1d7`d5sA@=L~h$*WSM4^NHQvAnL;3@5V$Es z!Bkeu)XNRIePEEt7$lY}c}P50$S4Q|1<^o3oKXk}3gL`Oj7BAPpsKhshn9t)W#MR9 z1@vthx!xH;^Ez^9P!Dt;w^ksPTZLR1uw|l4) zcq9^!#Nd%Jcq|f+#o)1dcsvr1$KdfXcsv>3JeP=oBqB5tQIJFww{=cT)JRO^w#3A$ zjzrpAs&88A7%O!Q%)JcH^)kv$b0cDrL@b7gjUnQZM0_5Rh#?YVh|PJ#F*0#%t_lUI zO4O*Ta;~b1uBz&&qH$%e3M5v69aZ35ZDLw2I9Lm^YC(|fr9i%|MNTs!r`eIy+?w&o zW;~|3Ii`6muNfpagMH2594%3Umgr2Y;z~4)Mx@a?XdsaWa@#q2X$ifQL~kCXkG0ds zKsq=`2U&Cw93x`Jh%sYhd1D}X4D1^N=fEmwusH^7CV{kQkVXW@^1!in5F~Q90D>JL z*aw1xAjkr_n%skQPn7E{0zVpE^@Bk6yxsonX!0p?g+LAk7@ya-3Cd*B<0AL>1^+a> z<3`38*`xjMa7Vq98_$biUhnC}ifcbQT=|&a#ySjR)wPIr;oZA(JgEYE7CsU z1Ya*A{AjYeLC|vNS1tFBwvML(b_1LVhtdnP3+0DA13lmO^|_-0x-MDphFq}_!VMY{ z7@n}$>WUF+8=BQ1$AfY zR(d$Xb%WE@hRQq@>U-37!=suu_XW?+U^iytotqqnDG|r{rB|gEO0T-e#AR=d_vp{D zt7qN^aOPB(jTgpq((JmdwAzhb84l8NeYI9DN zx)&fH^RE(y8>CXZaT&ZHHql~Ta~)gs${Q97@9bu9#0~Tca^VAZ;dUqe{&ilXQcVkR zE^>3ue1)$V2#>rn}{Z6XF0A`XDwGqEy6dbtDaGg zsk7vL+7yzY&$J6n;H9wPkzh{K$6achntmhuf6PvhtU$z76Fn;W@Y8YVHp!@hKz1b% z9|>x>8#n9stmreBUA_!jmGGj?q9WERwB&aAyjobEF}9LqxXi6R9wyILN5~zxlKLp; z=;;Av-7SAfk4>{Ap^DeEOU2d^;7Opwmms`qoteWD*4-!{Fir}!-cn}bK~~ReR*vC& zG&6oZ2tGSl1WJa>@@;w_UsQt{n#)=I%M(VLk!9{=Ke_{Olu%T_ zGH!0<#&A4+m|m#VFGRq8*tK4}vBL5pvy$7q+TK=~k#^KRX$KMv^@e=iBY#bhd?GQh zd$D+|+3!J)mI3^!yankZ<%OK?PIb3|?*#VKpJxM1^KQ7>2EP?eU z`RMQ8SsYI(R3~4TDzw&__xxbteq4!5rWLFt*4Jcj(aFXWhJ=#e8GL9)=%S+@bJSue zYAxjQ28yvu)98KFm=8`|qbAdo`Fi9!)9{3a=Ce3tL^dAin?U(bT0e8I_gT%IFo)kp z5!wqb5HRJcCQk4uAO-4ppQT7)lr4M-;4F)q>pho{C?co7j5I!C3?+xoq%-0eE-<~A z3oGu+Ay+HmYZ&)l9Y=}kP$f>NhQtTQO%`=eFdNs$w158UYXPUjaoRKR)vM7Yf!=cm zC`VI4vd%>jhMr-dS^`fzYN9NuQmztPJH;#tMX0lEDemQ6=ZxpU4;j9Oar$8o_(DD0 zP@%IksiQZZzPUbDya4Der90_h_zv=kx3~F>zztJRJ%=kdpJO&kHe4NhAG;tCnZYBpz?S}y^ntv$1CllO%RZ*3E%DZT$bX0%Kv zbu_2su^JCK)N6%Yu0IzXSe306XAV^dm1CT&j7Bu412+U+)UU|qnH!9#F17XCa%aP0 zkN=M8|Gbm@|7ez6Yh2@VxVRv&uP8PQdfMjzY`_~fAO>Y>K)nTV0TQ^XI{+yjpuOCS z+iDf-4y%)eiKBN8RJ(~3aW^b$aRO>zBff%<6n+%V$Z`#EfGjH|JTvKui)%X-K;C(whUA_$-YVIDvhbx&rLmf{oc-zXJ znt1m47hH#wq*AAs#EAqwSMWr_>^Z%SW}|h1g--!cX6_yyM+CZRL3DUNz5Bt}`0F*s zJX!t0gQp(9^^3kRW8$gV&s|pdG%vfPCm!`;c%5_LC|ccRl9x>=3LtXrXe zq}c1BEJG->K!Zp9V)k7JHM@ewH{+?#NM?4lcQ~^QPRM>nTgT7}Up8BGyd_*RD!{f$ zkujhCfTI@(fokZ(FWP5a=Lw^KvXr1tYTUgS=vn)HKa+^kh%o(PbJr8=@jIFlk%YP_p33!SnWo1J_n$CR_^d%hgg~d(3IBUgT0wv^#L1AJrQspu;ly2r`(esIcO6V#;bl=X_Jnr#^Bb4fmlfor?9zn$*6rr_-jxhc)hw9w)lYmhqDT} zlP&TV8-Z{PuCa*#GaPR_3#EuLQr zC?HMoFeOSZ&*ogxD8%+R@!J?G;glq?>>dH4_ZH9>59F_&x}IWB_>1RfUM4t*<0hhk z10o&~{5CuOoi2l~_Kup2G9q7^@v-r{?VuQa$c=~l*4$7yYsL+ihX2%{tXgT*F_-8;+L3(mN{oa{tLGuC6coBg=Zg?)d|1lUvYo3vl8$Yz* z?fN+^CH$LSq3O-Fd{0qEYi}}mqh8gtWJ>`_Cpk<|`1XFYE zK6D8#k|y0p_XyVl@<`&x%E{}C=I;_jJdS*5tg$&Mm92q`@NO$DtZExirDqR%dA(p$ zh{ZBdjZz)Dk}|$#thnqV9RtFhhlkw9U-eTWM=NPc7Jl8XaYB>KmY%zOh^YiM&fdw- zqb=R7XFZ-Yj9x>aBeQbOkXCuC;}Y!5{pMW**%Fcr*E+LxxeoTQtLON^3M$ukw~kJ| zsnFZ0elfGGb+l(&K{#bi@;zkdc&o@j|`z zk8SV)6Xgln<~k3;e<9$oi&8P9;)Uro;*hxLSI$UtoY!rBoF-v8EZhgRCv z{b(l|OG$`WJX5+HPBg9Y+5-599}+r_Nnko9K5WP{c^XRx1O;sLdjLm2!Il&yakW>5 zP^qBVYeQc+$q}Q{h$S4SDOSdp-$wr{pvX4-c}BUVQ+7Q^&tD?c^U2tOcoq60&Yok} z@xgoNV5Qr#6FD>F0Pf_KP~LbY+5NmVD(ZWcYA{ZQLrtk-CS~ds_J2jF8>9E_mxP00 zC4{B3MN>Ua=M`F?SCxIl8|TSe_ldWj{2MxPdB9}+BGiN+ZeL0;U2cVL$r|WV;n?VY zVANFbSf6-_-A_RB_ehE{G=6zO0BTFU38y&YPjX&<3gRtwv@aT-_B2 zx7%BE`w_0}QhHwNn5@Cp!HSzBPHn8PVI#?>lnQECGSNZo40Y`2RBve;k>BRExH~)V z2KS1J)|KT!q54mJ&ciG3E1h=Du46q@uV^h0?q&7y4QhSSWE}xDtm#J-aOf||zo-k@ zGHLVd#}}*WR%Jvkww#s!_3>uVy>Q!~>IiuzHTP zhwoLp0$5+R@x~~4;>r3DK)BQEm08LXCdV2Ol3&X+cTH9HSc8+Y!17rZ~AIks|Q;bdO|whpviM_QuH6;#Yx+!~ubd{_Wow zW8YWmj;8prEKma6NoaX0Y9)?kK&bS~ZDF!&jR_9=?}5~0(uL?!7Q9r(gDKL>qP_2W zY*wByuj|3CjCfoe(8n;jDL=KdmuD$1|LVmidg$`o@s`7%vbA8xs4mlV^Q3`)<3BG6 zgE9qzH;oa%-?5ouz5#al6+s-3zsSQ}H2Zi5`1h^)EkFDn^Q#L~sGxlU2Ua`6h_AGv zPcKP1)|?@)w#Sy zpCcWyHonMQzjSx2zL1UVRBWPLepx-`E$soOIN9ixs3R7KcaREYUN+^zVlUf2C75=F zcTO-{tZrf!Xi8K@eIaWrJiptryiO_S5as~m*U-uJ^+5du=NeO3YNE7nAAYKL!R=lcYSOGRbKb^8;R zgKmZ6YL`?)-cHezVX+_Wa~d97=iRZIx@ULXO7k42`;KSiP|qcTqP6*!QxCw0e8@8^ z8MCsi+h<+Z{&QsoXl#`^j$Z!knc~4KRdSj6CV2YtP;tqv8lUH_5=&PcLb(W+lJQvu zQK6ZNU98uYxn*#>bHw&H%ci{lPB6gX=Vg&1u792W81FXl{a@L^Z#w^iM*K~Y1Jux7 z?8n{xtbDy?Ck!EaWn;5Z$Di5nyfJk%czO8nr{HYBGP$Ph*DDKs>Y!G}^W+2mjmv3X zt)s&Ik*%u2FK){!a#Ul4B3&^$1AF|rNP)J_)>gC1ria#UgTaFo&I#7NRTI1A+{>n{ zY1~$>9V>$;l0G&`=Bldp367B+9{JpZXJ0MtfRP%8t+EN9BHXXh=PHMR#G0 znSU2v=cP%-bnYqCFedm#>+bb3RW~6tOGjNuwO@c&&TDV3W*BHt?=|+JJhd$(T)mTF z2rjn?X2lf>>{n0!>U@Q}s8_|-2wc~~VJ0k(__Rub3msF)eMOvZAO8IhY5{Az+LxQ- zQE1#m`R(=OCu5lky2i2=y+lE^TltEil^?ta?)X|hJB%uI{d?-F_wkK_o|(Kl;n;`k zj$up_yRHu1@;DHs>(1^c*En9B+*7MLEg$6^$`pABbbb8pF$LLrG3yBKQ-{-^NxKC$_Lt_3LlmYZJ)LUMTjQxu2f-a$%!OE>*zakzdVAkJ`$xT==^A z;A@I#ADzcEpG#7Ot}~!gn}jNRZq^dny?L|7XD-hVRx1q?4-jFnhe%RNln_~N&L`8I?Yj^jAA59st{tZ?9i~2y0?fnw5KqJ2^V$b*9jFd!jPtxJ#Zw;c<3!fcelkk|0wtXz&EJf*9Yb$_XE+aCSef|%(u17x27Y(qNH7FyHlOe79N`# z)?kWXIuMOEJ&CKjK=?r(cO7I8-FM>=nv}mxd2URYT1o~SHSA0NL`GB_37UazGw4C3qjP zXd>H5l7%&W%pZjjQs7t;(D`tNn zUDe^C^JdQo%JPx!FLuZ1+)za7|$d(Bwb=DS>M^NjU7JD_CQKKJ6#GJ;8uy;+)7%UelQx%>qrDr z=C0QLeOYLcXB_p?3MPJ28++`xs?a|RK5lf~6y=55zWGlvMTKRAArOe@h4bewLm*H*1oG3cARjn`x~#ccV9tSZ#4?z>e4crf2N;`O|_~0edLEnypDTC?|h&n`C9ipnP1W{Kq zP&Zapx3h;sAaIBt9C8Z|hrr=r8wW?^Zcc>^C_hBr7Q~w!~!C10a3M3 zGO+M;LG8ReP^fqm29!J$j*Nyt&=5T|N4sE9C=AL2gNnysP#BB{1_KWAFt~UO4s6L7LLPr(tKNhIAchhopfBZ zL0p;(4u!&@JaDLZoTo7kgTi4va5xkW2e$Dz0w{gB7Em}Ta7YRo6up#`TPZ|B3b8YV z$Vs6SQs|v2bWW;oS}K#3$^>%lq;oyb=cc(3Fen1XgMdR3a2^CgJfWqJz$6oxV9Oz* zAw;wuF$I)cL?S40L?VGm=MVt`5#Us(q*Vih)c~s+0LY%IB{5nC8O#m_ z6JP*?41mP|04BkMNr-2*fRe{#g3`xia+m;_3G^`m4nVvGw8R4~c|Z#ZpvM7p0zmHs zn0Ww`3@|$YW*-0$z|{djCjfxsK>%O@;9UTaGZjt*O-ab-yn{akBG$O`{;Ao!@Gb-* zgS>Fg)HXPCo{=0S>6-O<;U~2-VfUWwS6{IAO8lthgKy;x4)ixggN_mwqR;ojH`-Vs*&FZ6GZ-knp$*K7*2(Re zldY_1PDBLjJ$^6O|iq{Yd=vJnfaOa5{z1^hmt0|*9lcLM7Hx?KX*9m2>UEW*7ji7mF%F7%-^lX&zoY!XbcE46-$RM9wj?ykE zDEU3REEh>ets(qeey1-oa*RBkIL-F7L*cr?yldCj)1rQttL8Cp?;~!^)Vl%X55j@QuU z7qOB(l2hZ!{V|lg_|;!~_lP+(A{m$=?(k#|VMzhjZ>vc4@N$8YnZ}$MLLIHr6JHB_ zt)$lo%xoPqJ$hAe`SP?w_#!xzNEy9}S<>4!8#%B%;Xt7;T4UtN19lThn{LukOP(2%5l>7WLwVKmGBViQ zSK_Mww7V*{Of$d8s~zY#rqIh{Kvdj+H5HOvv6q-iqC^yUb@R#x-PW~$7gbN zsFdv8((V3wHcO72tCCasr0}STvX_Rnb-&a(mgwv9hyKos<(ArOMm=&gn#uhxrBm&e z*5B0!=qIkYs$i>6u?2Cah|q6!HNdy;>iPf{c}YNBs3+&%5(~&=d@2w2fAGb@Xy!NT zCseASq-YYd>+d@a;Q(9s#{Xo24#8W5mNUg>Lj%Hi%yfE?Y_UCoRKfw zPTYt^eb3kF7=QD8{GJ*t)H#RT^o_S@EirX*;Ky3wwqpA2jeGr3@mJ&UjB=b<#2x&wiA3-w_pCpZ(y4lQbGiTNs_;49%eaJ` zJpxW!H-w|oKEA4m?)3;H{XyZv&wta%~b2Wa-fk63(gU%{H#)+Abc)&j35 ztMZhVS`L!fwE^}QKG%Ov%(0kHw)MV9I4yNk9&3Lp>R@ImRFpioyJsmdEE3+NcanjAG4S#!0Vgo?ONRT^p zoRgKUX?Yv@Sf^7lq`NslwqfT9*69Pm3;xG>uVyURCwPiLN#*nsT1VR}yMOFc`TBON zZoL&E{DR#>=nBP?d`Aq+(h{YTM@uHXLdtyJI4y^t=HyO(+uYYHuW;Y}PWl~K@xyxw z;)O@w9RCHMp7!Pd_Wy^;Z)p=lCC8qdpU2Bw7SgGU-0PMl?&c*}(J5G=2fOA9yCw@Q z)rOX$w)wWpS6(j$s0j9S30K_PK5PRz#5w%Vrja{df!*0TR0{MLm>Rf!L0iyi#0$YQ zC_YZza8{mZZgm=pA{?t+%U4sbiQtnbH7~&zH$;*{WRt6tD#=qqOHIT)Geithw^Wl_ z6{EHBG^_i$c`%(HUR$n=sD5QQ6jZRSHB`B={DaJBJ=X3lD^w9_Cp|?UTHLNW+szwT z7#Mu)_qN81Po5D@JCPwjMyj5_cu&8{I!(*uv+y-@!ax%pm5LaOYRI8!Uz&Jc>4ksc zaAF=a(BtqyzmhN&s}+%joGm0Gx&ALi0$9VpAv{(WMH*vVoI9=wQEhtMNImKkhj7 zRDCK%^pulnBOhh`VXau_=&bvFtmbYKBYc0dY21K8Xz-J}*B?ReTR&nim*AL`!!{^_ zNWgdbL@_kygh}|7CxYt zn%1nFGLUVvar-j%ViQXBsIj-prD+jLJj~AnXEoTXdn(40uAoXQ#o{uC-%m&n*pD1N zgTF)A)062`8*%G3hp{Qt;~vSgz0=)OyDm!#c#v@boA3~-i?9sb;5F6FP{uxGo$Nt0;=_UHCo{?9VKYc!azkkGODcnx6B1 zH>~;&Ozo&>fZn{(D1H!=0}S zc<=rdTju}IW6J*+X#UN~{F5CYYdyo?OY)E(6>U&Ou*COk+~D-`tT*kI)@jH|uJsROL8zbgI&WpHimc2Vu*U`Q*q^(NNdj%#G6`57 zXZhEj5i9AblzY9^ocyI=B&}AdP)jK8p$G!$F^EKr>poV zY0}G^SBRr3%gLsBV$H0FxYKb&0c*)aid0KA%cada=BkM^G-5%hGBIcF6u~_eIMzK^ zFGL#4LFnD7JF(=^qfC9pIj?OrDhF8BRQT;qzh~ARPIC1rn`|@1OZ)#aSz&Ph>%0RO z8JBzil+oziY<#L6Xu2BN6LdL_Fz*QYvJ~|$gwecgOV4t-l;z zwTP8q%v#?YuY{+LBMw`GLO^LYW9Y_`RE5?JtvDl!W_|=c?^v$+Dfy{uLtn_aXPD*= z^DMC)Nau&5M6vFNsXcrDx5Femyr@bn!o~}9nv0;x!_sA-y-0kKqK)W(9#&`m1-E2p zSq1yO<%xR^*o$Z#u@RMmX3e&a&*f`n#*UZ<&A)ivFn)cmT#CWN5^H>mi&nmm_>3Kb zFXFZ-35R?IZO`T-_n^3aabYpuUsXO~pdZt75LZ9k3t_zI-)h+SLI5?MBez>MgM?mf z@NSgpZ>7dZ8FNbT50xSuUlINYgl{W2$ak%VnQ$C)r#d~RyXTaerrI^>+^sk zcGTgoXIA?;m(T7X+PaPQz!xk(cLR)xg*DMMPPB#K_K?q_pGGV)MqNb8$$o<7*)LeJ zen3a8Vmm9>UcB&u8fm@1a~j_odkzn>t(pT5uA8>a50;+bm&rLIUv+I+4ccp?Lfzoi z4@;bR=uDMn@FM-JC8zGbi{^i;zSK6v!_u}x+dK>#lEPHOGm4mdpCPZ>8Yi+d#WWRl z=vPiB>JKq%;lqIo_0MDH%181ayr|yh^K!pg;oW8~OLEt%8?N0g-2*Oo`04Bay#@R$ zhC>w#FfKB3EJ^pn&&vHhvlg_yBGY|imVWfN%j-`|gt6`yRj40pS00_kzx-FY$DhO2 zA0*|gbvNT`@tjxw*Dj#|PkNV|_KX8PCXT&OgS%C}p@T&`<=>+ju9rN!h#xc5KcwNg zxY2dX@UMIX5$ZZ^eVA%lEaMh#QBe>UxMx#}-AoXlN?E&2ihhuB*s`z4K~Kx*`(rlU z3P0PB0G(&A6u=AA2b1lUsS4SFq;((Xi>NsYszkc>2kH+EQI@L1LS)TMGd`r!?RU$P zF_H<>OBM4Bsa^B{3Eo&KhJ7#ui%8GwdLKqw8p|!i5FKei~<#QLlD^g!I7bkZ2DG`7r`>-;5b$v4H3L*C^oV?}_9;d6<7=oc~DCE){uw@WzdBZ}YA3kV+wi z!#1{8j#I0$ZK+iFCK?cu+`*XLe)S zXTSd{JC&+%^C-x+taNpOIiH@%(ni*i>AD;xn-heu!tQL9X<%gGhGuOJ)3ZQC=hI0x zdrsXiEV3Ul)tOujlPSp^k)Rza&7T{65A6+Hc+>o@{fShCerT*h=8=At`?1*S746{0 z>6J!U!nmzSKy4ItT>34(>trG6q_>id|G8@#RJ(mHAok>vBhqHY2hk0S#W_tuR&~AQ z+>8a`^wWt$hK~h%^b1hEcGT4-Gm_7tDq6tZF8EMSb&KrEuMcS^7dE#mj9+STZ9nSI zq?+OjL2f>D%3{*OCFf;FMCm{yO!y%0L;PQoj(e8c{`OgKLC66SqJ&YIr}s}2lRq(& zZn0+>trx`$8|Odi?)@WyiNfb)e>5KW6sKu4;boLgNoI_NWLWQ0m1wx9xqH&Dig8S> zw%sv3{|U{WdWr5_)4ImE2ec&*whAXm_~K(~fxwT#ghB)ALYDkPRAjh)P$Y;o;X{E? zvxi{)C~o+oJ*(IbK^f0Z6MGw?i}?8@f|Ha<-t4`sD*%_AYiUeM>^>JdlE{E1NVFmu zQl$gW%jT|@qSvg_%=0{)M9-zlse+tCW-7KBE|GDXy5Oj=sCPtG#pZ+ozi3&&cp2@( zoER1(kmqvB{v0TQyUA`3Mw{7j@a9%fBN>E#BAyr3zWVo zH|0|m5nhTFW4r|87EP;B6su$_;WjE^&+2kYlF~6cm_T`$`}7F3Nx}u1wcF0dU$vS; zX+?I!`8AATNrUJQ-tZ(wGZ%kejnO1vU;P2A33FWl!&rX#`Y5@|(Y!gRT7M?a$lG}o z3`>{VblPr)b&Td&#UcN4?4w$V@qdz1{>hUAA)^TP6(xBB+WYxWDmDKmf&PcppJ#lI zL;8kCZ4Bm-eJk^$LbkZSX1;&E%>R}m|4)iQUnBz6P zH8pE1mD}HsuQb*xPYMN?3HB82Z~;xy%h~BHg*2b!n%<{$!%M7uP3rd`;Wn2gkxH=) zg--pCwNVL3`)Fhq4a#u#$ZR(Qu_(MXXHISHvh>trf^W`zV%?wl)qm0WJ@#<&rR$f?ML6I+U z$xM0*>pJHwA|2L7?%}6xZFx(NZHHWubPEf&y@EUnVeQQER!QIn zLf^L~^O1QXVY#*}o!T?_hbvE!AZCEqGp(4V!t1PXagwmrbC+>G$0Cyq=7Iyw9bUaI&xM8iC8g@sj?iVHM zdoS~)jY||dRX#E*dGh`jS*YVKXws*rts@`2_uHomR(OwfZS6SLrE=Zhg94f6`T=pE zjwq<(yz(D0Au4isI-3!LaonEy{2QXzdA~jP<&Bs7I?mwp6n;!Jv?sIu3~(5etUAuG z=ARU)y@#pi-$jdQ?#_AV9o)XQws|Gcf2WdhGU(Rb+n@BeEX!==c4`>Ly8~K|9Q@Gc zdiyN?qY^dVkY6`!9Z^;JXkJzk?Mwg-1IkF$93 zHSB}_A_12dC3td|`qy~q{A^3IltN7U&oLhZRNudra6<{|j9!~lsBp3RUEM8Ucc{;o z9QY+;5Dv3@AMdUh*t~$(RivKx**|2kZAW2wgG!?(BH3U3n(_5qeBhc%V2JN_uY2U!PdLyZ+9fl^4vc&Xt|H75#q!r5GOM diff --git a/renv.lock b/renv.lock index 2a4f0de..e284d39 100644 --- a/renv.lock +++ b/renv.lock @@ -1,32 +1,24 @@ { "R": { - "Version": "3.6.2", + "Version": "4.0.3", "Repositories": [ { "Name": "CRAN", - "URL": "https://ftp.ussg.iu.edu/CRAN" + "URL": "https://packagemanager.rstudio.com/cran/latest" } ] }, "Python": { "Version": "3.6.10", - "Type": "conda", - "Name": null + "Type": "conda" }, "Packages": { "BH": { "Package": "BH", - "Version": "1.72.0-3", + "Version": "1.75.0-0", "Source": "Repository", "Repository": "CRAN", - "Hash": "8f9ce74c6417d61f0782cbae5fd2b7b0" - }, - "DT": { - "Package": "DT", - "Version": "0.12", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "0e120603cc57e4f1d741f739aa8147ba" + "Hash": "e4c04affc2cac20c8fec18385cd14691" }, "DiceDesign": { "Package": "DiceDesign", @@ -42,26 +34,19 @@ "Repository": "CRAN", "Hash": "29a7dccade1fd037c8262c2a239775eb" }, - "ISOcodes": { - "Package": "ISOcodes", - "Version": "2019.12.22", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "8ca8885ffb01998764cb762962f9b71b" - }, "KernSmooth": { "Package": "KernSmooth", - "Version": "2.23-16", + "Version": "2.23-17", "Source": "Repository", "Repository": "CRAN", - "Hash": "997471f25a7ed6c782f0090ce52cc63a" + "Hash": "bbff70c8c0357b5b88238c83f680fcd3" }, "MASS": { "Package": "MASS", - "Version": "7.3-51.4", + "Version": "7.3-53", "Source": "Repository", "Repository": "CRAN", - "Hash": "a94714e63996bc284b8795ec50defc07" + "Hash": "d1bc1c8e9c0ace57ec9ffea01021d45f" }, "Matrix": { "Package": "Matrix", @@ -79,17 +64,24 @@ }, "ModelMetrics": { "Package": "ModelMetrics", - "Version": "1.2.2.1", + "Version": "1.2.2.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "f0d8132eac48aeead07b3a7d5056d059" + "Hash": "40a55bd0b44719941d103291ac5e9d74" + }, + "PRROC": { + "Package": "PRROC", + "Version": "1.3.1", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "5506f0a5a0661ac39bfcfca702f1f282" }, "R6": { "Package": "R6", - "Version": "2.4.1", + "Version": "2.5.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "292b54f8f4b94669b08f94e5acce6be2" + "Hash": "b203113193e70978a696b2809525649d" }, "RColorBrewer": { "Package": "RColorBrewer", @@ -100,10 +92,10 @@ }, "RCurl": { "Package": "RCurl", - "Version": "1.98-1.1", + "Version": "1.98-1.3", "Source": "Repository", - "Repository": "CRAN", - "Hash": "26b1263f36bd66a9e8b5c80753ebedea" + "Repository": "RSPM", + "Hash": "ddac9abbfba243f9aeab9b5680b968d3" }, "RPushbullet": { "Package": "RPushbullet", @@ -114,45 +106,31 @@ }, "Rcpp": { "Package": "Rcpp", - "Version": "1.0.3", + "Version": "1.0.7", "Source": "Repository", - "Repository": "CRAN", - "Hash": "f3ca785924863b0e4c8cb23b6a5c75a1" + "Repository": "RSPM", + "Hash": "dab19adae4440ae55aa8a9d238b246bb" }, "RcppEigen": { "Package": "RcppEigen", - "Version": "0.3.3.7.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "c6faf038ba4346b1de19ad7c99b8f94a" - }, - "Rttf2pt1": { - "Package": "Rttf2pt1", - "Version": "1.3.8", + "Version": "0.3.3.9.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "8c4137a9ab70de4787d57758f8190617" + "Hash": "ddfa72a87fdf4c80466a20818be91d00" }, "SQUAREM": { "Package": "SQUAREM", - "Version": "2020.1", + "Version": "2021.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "62cbc077443029b383cb0498bc5fd502" + "Hash": "0cf10dab0d023d5b46a5a14387556891" }, "SnowballC": { "Package": "SnowballC", - "Version": "0.6.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "c5d4a8b3df9c2a2403cd8a392de457e8" - }, - "StanHeaders": { - "Package": "StanHeaders", - "Version": "2.21.0-1", + "Version": "0.7.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "bd39dfdcb370ffbbdec3459cc6c4f40c" + "Repository": "RSPM", + "Hash": "bc26e07c0d747fd287c370fe355e7b85" }, "askpass": { "Package": "askpass", @@ -161,26 +139,19 @@ "Repository": "CRAN", "Hash": "e8a22846fff485f0be3770c2da758713" }, - "assertthat": { - "Package": "assertthat", - "Version": "0.2.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "50c838a310445e954bc13f26f26a6ecf" - }, "attempt": { "Package": "attempt", - "Version": "0.3.0", + "Version": "0.3.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "9aaae25e273927dba4e279caac478baa" + "Hash": "d7421bb5dfeb2676b9e4a5a60c2fcfd2" }, "backports": { "Package": "backports", - "Version": "1.1.6", + "Version": "1.2.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "3997fd62345a616e59e8161ee0a5816f" + "Hash": "644043219fc24e190c2f620c1a380a69" }, "base64enc": { "Package": "base64enc", @@ -196,47 +167,47 @@ "Repository": "CRAN", "Hash": "0c54cf3a08cc0e550fbd64ad33166143" }, - "bayesplot": { - "Package": "bayesplot", - "Version": "1.7.1", + "bbotk": { + "Package": "bbotk", + "Version": "0.3.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "c994e351b537a5d5101ce52915e83a8d" + "Repository": "RSPM", + "Hash": "71168758199a26bd7843170bd122b1e2" }, - "bitops": { - "Package": "bitops", - "Version": "1.0-6", + "bit": { + "Package": "bit", + "Version": "4.0.4", "Source": "Repository", "Repository": "CRAN", - "Hash": "0b118d5900596bae6c4d4865374536a6" + "Hash": "f36715f14d94678eea9933af927bc15d" }, - "boot": { - "Package": "boot", - "Version": "1.3-24", + "bit64": { + "Package": "bit64", + "Version": "4.0.5", "Source": "Repository", "Repository": "CRAN", - "Hash": "72557d88b5f42f01221dfa436de99301" + "Hash": "9fe98599ca456d6552421db0d6772d8f" }, - "broom": { - "Package": "broom", - "Version": "0.5.4", + "bitops": { + "Package": "bitops", + "Version": "1.0-7", "Source": "Repository", - "Repository": "CRAN", - "Hash": "c8cc938d5fd2d51c33c705cda6998328" + "Repository": "RSPM", + "Hash": "b7d8d8ee39869c18d8846a184dd8a1af" }, - "callr": { - "Package": "callr", - "Version": "3.4.1", + "broom": { + "Package": "broom", + "Version": "0.7.6", "Source": "Repository", - "Repository": "CRAN", - "Hash": "f3c7c37950ae9a772b3676e4c172f978" + "Repository": "RSPM", + "Hash": "06015476250468fc013c30022118ce3a" }, "caret": { "Package": "caret", - "Version": "6.0-85", + "Version": "6.0-86", "Source": "Repository", "Repository": "CRAN", - "Hash": "43c568c0cbbc66f2c1fe93f054b70a71" + "Hash": "77b0545e2c16b4e57c8da2d14042b28d" }, "checkmate": { "Package": "checkmate", @@ -247,24 +218,24 @@ }, "class": { "Package": "class", - "Version": "7.3-15", + "Version": "7.3-17", "Source": "Repository", "Repository": "CRAN", - "Hash": "4fba6a022803b6c3f30fd023be3fa818" + "Hash": "9267f5dab59a4ef44229858a142bded1" }, "cli": { "Package": "cli", - "Version": "2.0.1", + "Version": "3.0.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "5173d8ab28680cf263636b110f4f3220" + "Hash": "e3ae5d68dea0c55a12ea12a9fda02e61" }, "clipr": { "Package": "clipr", - "Version": "0.7.0", + "Version": "0.7.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "08cf4045c149a0f0eaf405324c7495bd" + "Hash": "ebaa97ac99cc2daf04e77eecc7b781d7" }, "codetools": { "Package": "codetools", @@ -275,17 +246,10 @@ }, "colorspace": { "Package": "colorspace", - "Version": "1.4-1", + "Version": "2.0-2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "6b436e95723d1f0e861224dd9b094dfb" - }, - "colourpicker": { - "Package": "colourpicker", - "Version": "1.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "98ca919385a634e5d558e6938755e0bf" + "Repository": "RSPM", + "Hash": "6baccb763ee83c0bd313460fdb8b8a84" }, "commonmark": { "Package": "commonmark", @@ -294,68 +258,61 @@ "Repository": "CRAN", "Hash": "0f22be39ec1d141fd03683c06f3a6e67" }, - "crayon": { - "Package": "crayon", - "Version": "1.3.4", + "cpp11": { + "Package": "cpp11", + "Version": "0.2.7", "Source": "Repository", - "Repository": "CRAN", - "Hash": "0d57bc8e27b7ba9e45dba825ebc0de6b" + "Repository": "RSPM", + "Hash": "730eebcc741a5c36761f7d4d0f5e37b8" }, - "crosstalk": { - "Package": "crosstalk", - "Version": "1.0.0", + "crayon": { + "Package": "crayon", + "Version": "1.4.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "4ac529753d1e529966ef675d7f0c762b" + "Repository": "RSPM", + "Hash": "e75525c55c70e5f4f78c9960a4b402e9" }, "curl": { "Package": "curl", - "Version": "4.3", + "Version": "4.3.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "2b7d10581cc730804e9ed178c8374bd6" + "Repository": "RSPM", + "Hash": "022c42d49c28e95d69ca60446dbabf88" }, "data.table": { "Package": "data.table", - "Version": "1.12.8", + "Version": "1.14.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "cd711af60c47207a776213a368626369" - }, - "desc": { - "Package": "desc", - "Version": "1.2.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "6c8fe8fa26a23b79949375d372c7b395" + "Repository": "RSPM", + "Hash": "d1b8b1a821ee564a3515fa6c6d5c52dc" }, "dials": { "Package": "dials", - "Version": "0.0.4", + "Version": "0.0.9", "Source": "Repository", "Repository": "CRAN", - "Hash": "880dd3606a6623f864880dbaea8493a2" + "Hash": "eca6214674f3c1ed7add92a28e26f8ba" }, "digest": { "Package": "digest", - "Version": "0.6.25", + "Version": "0.6.27", "Source": "Repository", "Repository": "CRAN", - "Hash": "f697db7d92b7028c4b3436e9603fb636" + "Hash": "a0cbe758a531d054b537d16dff4d58a1" }, "doFuture": { "Package": "doFuture", - "Version": "0.9.0", + "Version": "0.12.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "1a67be55e710661863877d3c9afcc023" + "Hash": "2e0dd0b1ec9b8594f7a0247b1b3f7657" }, "dplyr": { "Package": "dplyr", - "Version": "0.8.4", + "Version": "1.0.7", "Source": "Repository", - "Repository": "CRAN", - "Hash": "3ecb1deedd4c4ad6ebf4605d263616f2" + "Repository": "RSPM", + "Hash": "36f1ae62f026c8ba9f9b5c9a08c03297" }, "dqrng": { "Package": "dqrng", @@ -366,15 +323,10 @@ }, "drake": { "Package": "drake", - "Version": "7.12.0.9000", - "Source": "GitHub", - "RemoteType": "github", - "RemoteHost": "api.github.com", - "RemoteRepo": "drake", - "RemoteUsername": "ropensci", - "RemoteRef": "master", - "RemoteSha": "cbbb05973480b92e87cc3380ce3f3994bf3caec9", - "Hash": "ecb24a2b9844a618f43ffcb6ddc1e9bd" + "Version": "7.13.1", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "fbd3ca7f23e75f996f39a9d086823459" }, "dtplyr": { "Package": "dtplyr", @@ -383,19 +335,12 @@ "Repository": "CRAN", "Hash": "be6b351032f660488e5b19c06927e3a0" }, - "dygraphs": { - "Package": "dygraphs", - "Version": "1.1.1.6", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "716869fffc16e282c118f8894e082a7d" - }, "ellipsis": { "Package": "ellipsis", - "Version": "0.3.0", + "Version": "0.3.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "7067d90c1c780bfe80c0d497e3d7b49d" + "Repository": "RSPM", + "Hash": "bb0eec2fe32e88d9e2836c2f73ea2077" }, "evaluate": { "Package": "evaluate", @@ -404,40 +349,19 @@ "Repository": "CRAN", "Hash": "ec8ca05cffcc70569eaaad8469d2a3a7" }, - "extrafont": { - "Package": "extrafont", - "Version": "0.17", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "7f2f50e8f998a4bea4b04650fc4f2ca8" - }, - "extrafontdb": { - "Package": "extrafontdb", - "Version": "1.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "a861555ddec7451c653b40e713166c6f" - }, "fansi": { "Package": "fansi", - "Version": "0.4.1", + "Version": "0.5.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "7fce217eaaf8016e72065e85c73027b5" + "Repository": "RSPM", + "Hash": "d447b40982c576a72b779f0a3b3da227" }, "farver": { "Package": "farver", - "Version": "2.0.3", + "Version": "2.1.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "dad6793a5a1f73c8e91f1a1e3e834b05" - }, - "fastmap": { - "Package": "fastmap", - "Version": "1.0.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "83ab58a0518afe3d17e41da01af13b60" + "Repository": "RSPM", + "Hash": "c98eb5133d9cb9e1622b8691487f11bb" }, "filelock": { "Package": "filelock", @@ -448,10 +372,10 @@ }, "foreach": { "Package": "foreach", - "Version": "1.5.0", + "Version": "1.5.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "8fb3ff01ee7d85893f56df8d77213381" + "Hash": "e32cfc0973caba11b65b1fa691b4d8c9" }, "forge": { "Package": "forge", @@ -462,92 +386,94 @@ }, "fs": { "Package": "fs", - "Version": "1.3.1", + "Version": "1.5.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "0e26be4558dbbc713d7cfe4a4c361f38" + "Hash": "44594a07a42e5f91fac9f93fda6d0109" }, "furrr": { "Package": "furrr", - "Version": "0.1.0", + "Version": "0.2.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "c1f60eafdbcbea57078aa6f501974d2a" + "Repository": "RSPM", + "Hash": "9f8988c1c716080a968a2949d1fd9af3" }, "future": { "Package": "future", - "Version": "1.16.0", + "Version": "1.21.0", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "f25fad6bee82b7ab01f055e2d813b96f" + }, + "future.apply": { + "Package": "future.apply", + "Version": "1.7.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "20c20385da360f28c07364f2d099a6d7" + "Hash": "7fb0dc1961807da107ab2078366052bd" }, "generics": { "Package": "generics", - "Version": "0.0.2", + "Version": "0.1.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "b8cff1d1391fd1ad8b65877f4c7f2e53" + "Hash": "4d243a9c10b00589889fe32314ffd902" + }, + "ggfittext": { + "Package": "ggfittext", + "Version": "0.9.1", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "42aafda63a2012fb3b928fef01d96698" }, "ggplot2": { "Package": "ggplot2", - "Version": "3.3.0", + "Version": "3.3.5", "Source": "Repository", - "Repository": "CRAN", - "Hash": "911561e07da928345f1ae2d69f97f3ea" + "Repository": "RSPM", + "Hash": "d7566c471c7b17e095dd023b9ef155ad" }, - "ggridges": { - "Package": "ggridges", - "Version": "0.5.2", + "git2r": { + "Package": "git2r", + "Version": "0.28.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "b5c4e55a3856dff3c05595630a40edfc" + "Hash": "f64fd34026f6025de71a4354800e6d79" }, - "git2r": { - "Package": "git2r", - "Version": "0.26.1", + "glmnet": { + "Package": "glmnet", + "Version": "4.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "135db4dbc94ed18f629ff8843a8064b7" + "Hash": "d5d038d2f66d7ed2f95045c62d42e7b4" }, "globals": { "Package": "globals", - "Version": "0.12.5", + "Version": "0.14.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "e9e529fb7a579ad4b4ff65e052e76ed8" + "Hash": "eca8023ed5ca6372479ebb9b3207f5ae" }, "glue": { "Package": "glue", - "Version": "1.4.0", + "Version": "1.4.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "2aefa994e8df5da17dc09afd80f924d5" + "Hash": "6efd734b14c6471cfe443345f3e35e29" }, "gower": { "Package": "gower", - "Version": "0.2.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "1e7e711f2f87cc3a326f7f406815d019" - }, - "gridExtra": { - "Package": "gridExtra", - "Version": "2.3", + "Version": "0.2.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "7d7f283939f563670a697165b2cf5560" + "Hash": "be6a2b3529928bd803d1c437d1d43152" }, "gt": { "Package": "gt", - "Version": "0.1.0", - "Source": "GitHub", - "RemoteType": "github", - "RemoteHost": "api.github.com", - "RemoteRepo": "gt", - "RemoteUsername": "rstudio", - "RemoteRef": "master", - "RemoteSha": "9782e790daed8a903cb94451aabff54400f0ec1b", - "Hash": "5cadddcef4aaf49e1f7e6092f5b180b9" + "Version": "0.2.2", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "4eb28493ed31ff8f5f240d0fb5929ecb" }, "gtable": { "Package": "gtable", @@ -556,26 +482,19 @@ "Repository": "CRAN", "Hash": "ac5c6baf7822ce8732b343f14c072c4d" }, - "gtools": { - "Package": "gtools", - "Version": "3.8.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "b7f3a3bee8ec0858e8cbc09cfdc35ced" - }, "h2o": { "Package": "h2o", - "Version": "3.28.0.4", + "Version": "3.32.0.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "82e0902fe684ece2c4e0b8d12aa3b919" + "Hash": "d83b75b067cd21f62326a947e9f98122" }, "hardhat": { "Package": "hardhat", - "Version": "0.1.1", + "Version": "0.1.5", "Source": "Repository", "Repository": "CRAN", - "Hash": "1c5cdd05d2a9f0fa5e619afeeeb1c2f2" + "Hash": "aa8ad570d6c1662de36ccbe09b67d473" }, "highr": { "Package": "highr", @@ -586,59 +505,52 @@ }, "hms": { "Package": "hms", - "Version": "0.5.3", + "Version": "1.0.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "726671f634529d470545f9fd1a9d1869" + "Hash": "bf552cdd96f5969873afdac7311c7d0d" }, "htmltools": { "Package": "htmltools", - "Version": "0.4.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "2d7691222f82f41e93f6d30f169bd5e1" - }, - "htmlwidgets": { - "Package": "htmlwidgets", - "Version": "1.5.1", + "Version": "0.5.1.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "41bace23583fbc25089edae324de2dc3" + "Repository": "RSPM", + "Hash": "af2c2531e55df5cf230c4b5444fc973c" }, "httpuv": { "Package": "httpuv", - "Version": "1.5.2", + "Version": "1.6.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "f793dad2c9ae14fbb1d22f16f23f8326" + "Repository": "RSPM", + "Hash": "54344a78aae37bc6ef39b1240969df8e" }, "httr": { "Package": "httr", - "Version": "1.4.1", + "Version": "1.4.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "7146fea4685b4252ebf478978c75f597" + "Hash": "a525aba14184fec243f9eaec62fbed43" }, "hunspell": { "Package": "hunspell", - "Version": "3.0", + "Version": "3.0.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "71e7853d60b6b4ba891d62ede21752e9" + "Repository": "RSPM", + "Hash": "3987784c19192ad0f2261c456d936df1" }, "igraph": { "Package": "igraph", - "Version": "1.2.5", + "Version": "1.2.6", "Source": "Repository", "Repository": "CRAN", - "Hash": "3878c30ce67cdb7f2d7f72554e37f476" + "Hash": "7b1f856410253d56ea67ad808f7cdff6" }, "infer": { "Package": "infer", - "Version": "0.5.1", + "Version": "0.5.4", "Source": "Repository", "Repository": "CRAN", - "Hash": "5d74c75f99369d87d385a8d4bf316eeb" + "Hash": "b53abe478cd07312b8e09543f07c95f6" }, "ini": { "Package": "ini", @@ -647,18 +559,12 @@ "Repository": "CRAN", "Hash": "6154ec2223172bce8162d4153cda21f7" }, - "inline": { - "Package": "inline", - "Version": "0.3.15", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "24fe9c7832cd19e60c04ffb46f2cbb64" - }, "installr": { "Package": "installr", "Version": "0.22.0", "Source": "Repository", - "Repository": "CRAN", + "Repository": "RSPM", + "OS_type": "windows", "Hash": "9c639b8d0d75fc2b5249c23886cf6ebe" }, "ipred": { @@ -670,23 +576,23 @@ }, "isoband": { "Package": "isoband", - "Version": "0.2.0", + "Version": "0.2.5", "Source": "Repository", - "Repository": "CRAN", - "Hash": "15f6d57a664cd953a31ae4ea61e5e60e" + "Repository": "RSPM", + "Hash": "7ab57a6de7f48a8dc84910d1eca42883" }, "iterators": { "Package": "iterators", - "Version": "1.0.12", + "Version": "1.0.13", "Source": "Repository", "Repository": "CRAN", - "Hash": "117128f48662573ff4c4e72608b9e202" + "Hash": "64778782a89480e9a644f69aad9a2877" }, "janeaustenr": { "Package": "janeaustenr", "Version": "0.1.5", "Source": "Repository", - "Repository": "CRAN", + "Repository": "RSPM", "Hash": "8b07a4b9d0a0d97d9fe12de8af6d219e" }, "jsonlite": { @@ -696,13 +602,6 @@ "Repository": "CRAN", "Hash": "84b0ee361e2f78d6b7d670db9471c0c5" }, - "kableExtra": { - "Package": "kableExtra", - "Version": "1.1.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "37e11c605b3c0b7207763beda52c8535" - }, "kernlab": { "Package": "kernlab", "Version": "0.9-29", @@ -712,24 +611,24 @@ }, "knitr": { "Package": "knitr", - "Version": "1.28", + "Version": "1.31", "Source": "Repository", - "Repository": "CRAN", - "Hash": "915a6f0134cdbdf016d7778bc80b2eda" + "Repository": "RSPM", + "Hash": "c3994c036d19fc22c5e2a209c8298bfb" }, "labeling": { "Package": "labeling", - "Version": "0.3", + "Version": "0.4.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "73832978c1de350df58108c745ed0e3e" + "Hash": "3d5108641f47470611a32d0bdf357a72" }, "later": { "Package": "later", - "Version": "1.0.0", + "Version": "1.2.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "6d927978fc658d24175ce37db635f9e5" + "Repository": "RSPM", + "Hash": "b61890ae77fea19fc8acadd25db70aa4" }, "lattice": { "Package": "lattice", @@ -740,38 +639,31 @@ }, "lava": { "Package": "lava", - "Version": "1.6.6", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "3aff2b3035122185382c4ec9c500382f" - }, - "lazyeval": { - "Package": "lazyeval", - "Version": "0.2.2", + "Version": "1.6.8.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "d908914ae53b04d4c0c0fd72ecc35370" + "Hash": "4f337c8dcd7fdf0df89ee74c4f5d94d5" }, "lgr": { "Package": "lgr", - "Version": "0.3.3", + "Version": "0.4.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "919a29725a197aea9a723b96db3e7975" + "Hash": "55545b597ebc09be71e7b886932ab327" }, "lhs": { "Package": "lhs", - "Version": "1.0.1", + "Version": "1.1.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "43b4afe7aed4a471ab164a5b764f7151" + "Hash": "e44ecdb78d9373a6a11515bf6ec00251" }, "lifecycle": { "Package": "lifecycle", - "Version": "0.1.0", + "Version": "1.0.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "dc0e9c03b3635ff433b045ce6bf0612d" + "Hash": "3471fb65971f1a7b2d4ae7848cf2db8d" }, "listenv": { "Package": "listenv", @@ -780,33 +672,19 @@ "Repository": "CRAN", "Hash": "0bde42ee282efb18c7c4e63822f5b4f7" }, - "lme4": { - "Package": "lme4", - "Version": "1.1-21", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "951b57d6afd25bebada4efee4f4c3478" - }, - "loo": { - "Package": "loo", - "Version": "2.2.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "5384cbaf43ee10191a52478f5e2bad48" - }, "lubridate": { "Package": "lubridate", - "Version": "1.7.4", + "Version": "1.7.9.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "796afeea047cda6bdb308d374a33eeb6" + "Hash": "5b5b02f621d39a499def7923a5aee746" }, "magrittr": { "Package": "magrittr", - "Version": "1.5", + "Version": "2.0.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "1bb58822a20301cee84a41678e25d9b7" + "Hash": "41287f1ac7d28a92f0a286ed507928d3" }, "markdown": { "Package": "markdown", @@ -815,89 +693,75 @@ "Repository": "CRAN", "Hash": "61e4a10781dd00d7d81dd06ca9b94e95" }, - "matrixStats": { - "Package": "matrixStats", - "Version": "0.55.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "1270d684b417c455fae69ad025153f90" - }, "mgcv": { "Package": "mgcv", - "Version": "1.8-31", + "Version": "1.8-33", "Source": "Repository", "Repository": "CRAN", - "Hash": "4bb7e0c4f3557583e1e8d3c9ffb8ba5c" + "Hash": "eb7b6439bc6d812eed2cddba5edc6be3" }, "mime": { "Package": "mime", - "Version": "0.9", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "e87a35ec73b157552814869f45a63aa3" - }, - "miniUI": { - "Package": "miniUI", - "Version": "0.1.1.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "fec5f52652d60615fdb3957b3d74324a" - }, - "minqa": { - "Package": "minqa", - "Version": "1.2.4", + "Version": "0.11", "Source": "Repository", - "Repository": "CRAN", - "Hash": "eaee7d2a6f3ed4491df868611cb064cc" + "Repository": "RSPM", + "Hash": "8974a907200fc9948d636fe7d85ca9fb" }, "mlbench": { "Package": "mlbench", - "Version": "2.1-1", + "Version": "2.1-3", "Source": "Repository", - "Repository": "CRAN", - "Hash": "978aa169a072e2a0ed2aa5d8a8f3a07c" + "Repository": "RSPM", + "Hash": "6bb7265771062ba4f059c53e1daed30b" }, "mlflow": { "Package": "mlflow", - "Version": "1.6.0", + "Version": "1.13.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "4ef9f64ceb0aee549eeb1de7e843f04b" + "Hash": "16cea4cbbcf1f76f604264056e18d4c8" }, "mlr3": { "Package": "mlr3", - "Version": "0.1.7", + "Version": "0.10.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "cc73299a03196f3a17c4f5507e341623" + "Repository": "RSPM", + "Hash": "dcfcdc072f7f0a814680d1a3be84c5b3" }, "mlr3learners": { "Package": "mlr3learners", - "Version": "0.1.6", + "Version": "0.4.3", "Source": "Repository", "Repository": "CRAN", - "Hash": "b8d91f849c24bb6d558d84ab953e8f2a" + "Hash": "863f0ec14c987b49cd56ebaedd303b31" }, "mlr3measures": { "Package": "mlr3measures", - "Version": "0.1.1", + "Version": "0.3.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "82c53ce4c51cfbe2fc9f372b926d6567" + "Hash": "668ab83e95837c0ab5f600ed233f6da8" }, "mlr3misc": { "Package": "mlr3misc", - "Version": "0.1.8", + "Version": "0.7.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "697dbe66a60fd599cee7f56d349dfd19" + "Hash": "ce4902bd98d8b2e75d8e9a77b74f591e" }, "mlr3tuning": { "Package": "mlr3tuning", - "Version": "0.1.2", + "Version": "0.6.0", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "d9fd81186105f02c7e3a3176d29335a7" + }, + "modeldata": { + "Package": "modeldata", + "Version": "0.1.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "b30a7449e9d1a71f38f0815ffded6f6e" + "Hash": "9ff91d86290b17774fdc7dc490e2298d" }, "munsell": { "Package": "munsell", @@ -908,24 +772,17 @@ }, "nlme": { "Package": "nlme", - "Version": "3.1-144", + "Version": "3.1-149", "Source": "Repository", "Repository": "CRAN", - "Hash": "e80d41932d3cc235ccbbbb9732ae162e" - }, - "nloptr": { - "Package": "nloptr", - "Version": "1.2.1", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "123f92613e59e1f5671f80f72363ae60" + "Hash": "7c24ab3a1e3afe50388eb2d893aab255" }, "nnet": { "Package": "nnet", - "Version": "7.3-12", + "Version": "7.3-14", "Source": "Repository", "Repository": "CRAN", - "Hash": "68287aec1f476c41d16ce1ace445800c" + "Hash": "0d87e50e11394a7151a28873637d799a" }, "numDeriv": { "Package": "numDeriv", @@ -936,24 +793,24 @@ }, "openssl": { "Package": "openssl", - "Version": "1.4.1", + "Version": "1.4.4", "Source": "Repository", - "Repository": "CRAN", - "Hash": "49f7258fd86ebeaea1df24d9ded00478" + "Repository": "RSPM", + "Hash": "f4dbc5a47fd93d3415249884d31d6791" }, "pROC": { "Package": "pROC", - "Version": "1.16.1", + "Version": "1.17.0.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "162b2306b3a9ff917c087361599012d5" + "Hash": "e25078f6e770b81121672874474f69c0" }, - "packrat": { - "Package": "packrat", - "Version": "0.5.0", + "pack": { + "Package": "pack", + "Version": "0.1-1", "Source": "Repository", "Repository": "CRAN", - "Hash": "2ebd34a38f4248281096cc723535b66d" + "Hash": "c4f814b30334e8bc6124647794781a09" }, "pacman": { "Package": "pacman", @@ -964,38 +821,38 @@ }, "paradox": { "Package": "paradox", - "Version": "0.1.0", + "Version": "0.7.0", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "d01214431e7055472c8d73488d035f85" + }, + "parallelly": { + "Package": "parallelly", + "Version": "1.23.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "4c72f7c6af1c4b9975f414011a680b7e" + "Hash": "2f841d7915986a8fd995ab4ec105dd1b" }, "parsnip": { "Package": "parsnip", - "Version": "0.0.5", + "Version": "0.1.6", "Source": "Repository", - "Repository": "CRAN", - "Hash": "ececc6518695f3390f5dd7b45558c0e7" + "Repository": "RSPM", + "Hash": "f3a52d34ee4a038ebe5ef8e29f46fb57" }, "patchwork": { "Package": "patchwork", - "Version": "1.0.0", + "Version": "1.1.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "16eee5b5edc41eec5af1149ccdc6b2c9" + "Repository": "RSPM", + "Hash": "c446b30cb33ec125ff02588b60660ccb" }, "pillar": { "Package": "pillar", - "Version": "1.4.3", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "fa3ed60396b6998d0427c57dab90fba4" - }, - "pkgbuild": { - "Package": "pkgbuild", - "Version": "1.0.6", + "Version": "1.6.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "899835dfe286963471cbdb9591f8f94f" + "Repository": "RSPM", + "Hash": "43f228eb4b49093d1c8a5c93cae9efe9" }, "pkgconfig": { "Package": "pkgconfig", @@ -1004,33 +861,12 @@ "Repository": "CRAN", "Hash": "01f28d4278f15c76cddbea05899c5d6f" }, - "pkgload": { - "Package": "pkgload", - "Version": "1.0.2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "5e655fb54cceead0f095f22d7be33da3" - }, - "plogr": { - "Package": "plogr", - "Version": "0.2.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "09eb987710984fc2905c7129c7d85e65" - }, "plyr": { "Package": "plyr", - "Version": "1.8.5", + "Version": "1.8.6", "Source": "Repository", "Repository": "CRAN", - "Hash": "3f1b0dbcc503320e6e7aae6c3ff87eaa" - }, - "praise": { - "Package": "praise", - "Version": "1.0.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "a555924add98c99d2f411e37e7d25e9f" + "Hash": "ec0e5ab4e5f851f6ef32cd1d1984957f" }, "prettyunits": { "Package": "prettyunits", @@ -1041,17 +877,17 @@ }, "prismatic": { "Package": "prismatic", - "Version": "0.2.0", + "Version": "1.0.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "1751eff9cd67384716c6e324ffdab339" + "Hash": "ff0fd99eeae6a4ec43be2be881588681" }, "processx": { "Package": "processx", - "Version": "3.4.2", + "Version": "3.5.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "20a082f2bde0ffcd8755779fd476a274" + "Repository": "RSPM", + "Hash": "0cbca2bc4d16525d009c4dbba156b37c" }, "prodlim": { "Package": "prodlim", @@ -1060,26 +896,33 @@ "Repository": "CRAN", "Hash": "c243bf70db3a6631a0c8783152fb7db9" }, + "progress": { + "Package": "progress", + "Version": "1.2.2", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "14dc9f7a3c91ebb14ec5bb9208a07061" + }, "promises": { "Package": "promises", - "Version": "1.1.0", + "Version": "1.2.0.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "efbbe62da4709f7040a380c702bc7103" + "Hash": "4ab2c43adb4d4699cf3690acd378d75d" }, "ps": { "Package": "ps", - "Version": "1.3.0", + "Version": "1.6.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "919a32c940a25bc95fd464df9998a6ba" + "Repository": "RSPM", + "Hash": "32620e2001c1dce1af49c49dccbb9420" }, "purrr": { "Package": "purrr", - "Version": "0.3.3", + "Version": "0.3.4", "Source": "Repository", "Repository": "CRAN", - "Hash": "22aca7d1181718e927d403a8c2d69d62" + "Hash": "97def703420c8ab10d8f0e6c72101e02" }, "ranger": { "Package": "ranger", @@ -1097,43 +940,43 @@ }, "readr": { "Package": "readr", - "Version": "1.3.1", + "Version": "2.0.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "af8ab99cd936773a148963905736907b" + "Repository": "RSPM", + "Hash": "849038f0839134ab35e719a9820005a6" }, "recipes": { "Package": "recipes", - "Version": "0.1.9", + "Version": "0.1.15", "Source": "Repository", "Repository": "CRAN", - "Hash": "605c30dae049a94180ca1ab5066120c8" + "Hash": "e53c0258e2d126419df25e5390082819" }, "remotes": { "Package": "remotes", - "Version": "2.1.1", + "Version": "2.2.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "57c3009534f805f0f6476ffee68483cc" + "Hash": "430a0908aee75b1fcba0e62857cab0ce" }, "renv": { "Package": "renv", - "Version": "0.9.3-30", + "Version": "0.14.0-3", "Source": "GitHub", "RemoteType": "github", "RemoteHost": "api.github.com", - "RemoteRepo": "renv", "RemoteUsername": "rstudio", + "RemoteRepo": "renv", "RemoteRef": "master", - "RemoteSha": "916923a009addb383a2c30826e6266a38da4327d", - "Hash": "291d049b7931ebd571be09c5fc82acbe" + "RemoteSha": "9344caf707ab31d7e6e64bc7c42215a052df19ad", + "Hash": "bfa3f4642a506c1fbc475ab3dc12b312" }, "reshape2": { "Package": "reshape2", - "Version": "1.4.3", + "Version": "1.4.4", "Source": "Repository", - "Repository": "CRAN", - "Hash": "15a23ad30f51789188e439599559815c" + "Repository": "RSPM", + "Hash": "bb5996d0bd962d214a11140d77589917" }, "reticulate": { "Package": "reticulate", @@ -1144,17 +987,17 @@ }, "rlang": { "Package": "rlang", - "Version": "0.4.6", + "Version": "0.4.11", "Source": "Repository", "Repository": "CRAN", - "Hash": "aa263e3ce17b177c49e0daade2ee3cdc" + "Hash": "515f341d3affe0de9e4a7f762efb0456" }, "rmarkdown": { "Package": "rmarkdown", - "Version": "2.1", + "Version": "2.9", "Source": "Repository", - "Repository": "CRAN", - "Hash": "9d1c61d476c448350c482d6664e1b28b" + "Repository": "RSPM", + "Hash": "912c09266d5470516df4df7a303cde92" }, "rpart": { "Package": "rpart", @@ -1163,110 +1006,47 @@ "Repository": "CRAN", "Hash": "9787c1fcb680e655d062e7611cadf78e" }, - "rprojroot": { - "Package": "rprojroot", - "Version": "1.3-2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "f6a407ae5dd21f6f80a6708bbb6eb3ae" - }, "rsample": { "Package": "rsample", - "Version": "0.0.5", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "48a16b22b5594526c3f24d79b7f8e645" - }, - "rsconnect": { - "Package": "rsconnect", - "Version": "0.8.16", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "3924a1c20ce2479e89a08b0ca4c936c6" - }, - "rstan": { - "Package": "rstan", - "Version": "2.19.2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "088a3674b7bb2688184beaa2907ddad4" - }, - "rstanarm": { - "Package": "rstanarm", - "Version": "2.19.2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "3efac7d6a136d27333505429b3fe6847" - }, - "rstantools": { - "Package": "rstantools", - "Version": "2.0.0", + "Version": "0.1.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "fd4a4bf7068df96dfdd8551827ccb969" + "Repository": "RSPM", + "Hash": "9e4f4f3b91998715bcc740f88ea328a3" }, "rstudioapi": { "Package": "rstudioapi", - "Version": "0.11", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "33a5b27a03da82ac4b1d43268f80088a" - }, - "rvest": { - "Package": "rvest", - "Version": "0.3.5", + "Version": "0.13", "Source": "Repository", "Repository": "CRAN", - "Hash": "6a20c2cdf133ebc7ac45888c9ccc052b" + "Hash": "06c85365a03fdaf699966cc1d3cf53ea" }, "sass": { "Package": "sass", - "Version": "0.1.2.1", + "Version": "0.4.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "bd7168e8f7710ee96b2d5bf94d9c1a38" + "Repository": "RSPM", + "Hash": "50cf822feb64bb3977bda0b7091be623" }, "scales": { "Package": "scales", - "Version": "1.1.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "a1c68369c629ea3188d0676e37069c65" - }, - "selectr": { - "Package": "selectr", - "Version": "0.4-2", + "Version": "1.1.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "3838071b66e0c566d55cc26bd6e27bf4" + "Hash": "6f76f71042411426ec8df6c54f34e6dd" }, - "shiny": { - "Package": "shiny", + "shades": { + "Package": "shades", "Version": "1.4.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "6ca23724bb2c804c1d0b3db4862a39c7" + "Repository": "RSPM", + "Hash": "3a2cf5eecb2cbca814d01d56bfb51494" }, - "shinyjs": { - "Package": "shinyjs", - "Version": "1.1", + "shape": { + "Package": "shape", + "Version": "1.4.5", "Source": "Repository", "Repository": "CRAN", - "Hash": "b40a5207b6624f6e2b8cdb50689cdb69" - }, - "shinystan": { - "Package": "shinystan", - "Version": "2.5.0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "85e1a9e77cdd1b8740e92b37f8fdce7b" - }, - "shinythemes": { - "Package": "shinythemes", - "Version": "1.1.2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "8f047210d7d68ea4860a3c0d8cced272" + "Hash": "58510f25472de6fd363d76698d29709e" }, "sitmo": { "Package": "sitmo", @@ -1275,33 +1055,26 @@ "Repository": "CRAN", "Hash": "0f9ba299f2385e686745b066c6d7a7c4" }, - "sourcetools": { - "Package": "sourcetools", - "Version": "0.1.7", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "947e4e02a79effa5d512473e10f41797" - }, - "stopwords": { - "Package": "stopwords", - "Version": "1.0", + "slider": { + "Package": "slider", + "Version": "0.1.5", "Source": "Repository", "Repository": "CRAN", - "Hash": "96dc2a146912716288a00619ba486823" + "Hash": "ac1eb08941cacd8ff16f0041ca60a3de" }, "storr": { "Package": "storr", - "Version": "1.2.1", + "Version": "1.2.5", "Source": "Repository", "Repository": "CRAN", - "Hash": "0a3635220b58f2c2faccd78e97b0cafd" + "Hash": "96034207276a46a44dc81b8d43397602" }, "stringi": { "Package": "stringi", - "Version": "1.4.6", + "Version": "1.7.3", "Source": "Repository", - "Repository": "CRAN", - "Hash": "e99d8d656980d2dd416a962ae55aec90" + "Repository": "RSPM", + "Hash": "7943cfae120c77a255025e5f63856532" }, "stringr": { "Package": "stringr", @@ -1312,45 +1085,38 @@ }, "survival": { "Package": "survival", - "Version": "3.1-8", + "Version": "3.2-7", "Source": "Repository", "Repository": "CRAN", - "Hash": "ad25122f95d04988f6f79d69aaadd53d" + "Hash": "39c4ac6d22dad33db0ee37b40810ea12" }, "swagger": { "Package": "swagger", - "Version": "3.9.2", + "Version": "3.33.1", "Source": "Repository", "Repository": "CRAN", - "Hash": "406eb0098dbd3413734895833961a21d" + "Hash": "f28d25ed70c903922254157c11b0081d" }, - "sys": { - "Package": "sys", - "Version": "3.3", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "507f3116a38d37ad330a038b3be07b66" - }, - "testthat": { - "Package": "testthat", - "Version": "2.3.1", + "swatches": { + "Package": "swatches", + "Version": "0.5.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "68dad590f6445cdcdaa5b7eec60e9686" + "Hash": "1040343007abd3dafe49b0bfd29199a3" }, - "threejs": { - "Package": "threejs", - "Version": "0.3.3", + "sys": { + "Package": "sys", + "Version": "3.4", "Source": "Repository", "Repository": "CRAN", - "Hash": "2ad32c3a8745e827977f394bc387e3b0" + "Hash": "b227d13e29222b4574486cfcbde077fa" }, "tibble": { "Package": "tibble", - "Version": "2.1.3", + "Version": "3.1.3", "Source": "Repository", - "Repository": "CRAN", - "Hash": "8248ee35d1e15d1e506f05f5a5d46a75" + "Repository": "RSPM", + "Hash": "038455513fde65e79c25e724e0c84ca2" }, "tictoc": { "Package": "tictoc", @@ -1361,45 +1127,31 @@ }, "tidymodels": { "Package": "tidymodels", - "Version": "0.0.3", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "dcf217ad7b364b18193e7f7410161387" - }, - "tidyposterior": { - "Package": "tidyposterior", - "Version": "0.0.2", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "35353b78b2e01ee8d6495b0a9681d7fd" - }, - "tidypredict": { - "Package": "tidypredict", - "Version": "0.4.4", + "Version": "0.1.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "c9e7c1f9e9a14ff66e2677e3b78a96d6" + "Hash": "dc3ec48f9e05a955f132b0e075ce6c34" }, "tidyr": { "Package": "tidyr", - "Version": "1.0.2", + "Version": "1.1.3", "Source": "Repository", - "Repository": "CRAN", - "Hash": "fb73a010ace00d6c584c2b53a21b969c" + "Repository": "RSPM", + "Hash": "450d7dfaedde58e28586b854eeece4fa" }, "tidyselect": { "Package": "tidyselect", - "Version": "1.0.0", + "Version": "1.1.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "7d4b0f1ab542d8cb7a40c593a4de2f36" + "Repository": "RSPM", + "Hash": "7243004a708d06d4716717fa1ff5b2fe" }, "tidytext": { "Package": "tidytext", - "Version": "0.2.2", + "Version": "0.3.1", "Source": "Repository", - "Repository": "CRAN", - "Hash": "ca16fda85d4abb418323ca4e533db085" + "Repository": "RSPM", + "Hash": "0debc5a59ccbe48dfd8e9c98db184b95" }, "timeDate": { "Package": "timeDate", @@ -1410,38 +1162,45 @@ }, "tinytex": { "Package": "tinytex", - "Version": "0.19", + "Version": "0.32", "Source": "Repository", - "Repository": "CRAN", - "Hash": "b1c0c22cab714ec9ce9472a8073b6922" + "Repository": "RSPM", + "Hash": "db9a6f2cf147751322d22c9f6647c7bd" }, "tokenizers": { "Package": "tokenizers", "Version": "0.2.1", "Source": "Repository", - "Repository": "CRAN", + "Repository": "RSPM", "Hash": "a064f646b3a692e62dfb5d9ea690a4ea" }, "tune": { "Package": "tune", - "Version": "0.0.1", + "Version": "0.1.5", "Source": "Repository", - "Repository": "CRAN", - "Hash": "5dc1092c4121af932cd59e657d6f91cd" + "Repository": "RSPM", + "Hash": "ee53286ff0fe213dbc5264c399767572" }, "txtq": { "Package": "txtq", - "Version": "0.2.0", + "Version": "0.2.3", "Source": "Repository", "Repository": "CRAN", - "Hash": "92e5a36ad5b445e895a0f9b98db4d810" + "Hash": "38a421b8003ba704b5b8471e6c3762a7" + }, + "tzdb": { + "Package": "tzdb", + "Version": "0.1.2", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "fb2b801053decce71295bb8cb04d438b" }, "utf8": { "Package": "utf8", - "Version": "1.1.4", + "Version": "1.2.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "4a5081acfb7b81a572e4384a7aaf2af1" + "Repository": "RSPM", + "Hash": "c9c462b759a5cc844ae25b5942654d13" }, "uuid": { "Package": "uuid", @@ -1452,66 +1211,59 @@ }, "vctrs": { "Package": "vctrs", - "Version": "0.2.4", + "Version": "0.3.8", "Source": "Repository", - "Repository": "CRAN", - "Hash": "6c839a149a30cb4ffc70443efa74c197" + "Repository": "RSPM", + "Hash": "ecf749a1b39ea72bd9b51b76292261f1" }, "viridisLite": { "Package": "viridisLite", - "Version": "0.3.0", + "Version": "0.4.0", "Source": "Repository", - "Repository": "CRAN", - "Hash": "ce4f6271baa94776db692f1cb2055bee" + "Repository": "RSPM", + "Hash": "55e157e2aa88161bdb0754218470d204" }, - "webshot": { - "Package": "webshot", - "Version": "0.5.2", + "vroom": { + "Package": "vroom", + "Version": "1.5.4", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "1a23013f39e67bb57cbda6f4ddde5470" + }, + "warp": { + "Package": "warp", + "Version": "0.2.0", "Source": "Repository", "Repository": "CRAN", - "Hash": "e99d80ad34457a4853674e89d5e806de" + "Hash": "2982481615756e24e79fee95bdc95daa" }, "withr": { "Package": "withr", - "Version": "2.1.2", + "Version": "2.4.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "aa57ed55ff2df4bea697a07df528993d" + "Hash": "ad03909b44677f930fa156d47d7a3aeb" }, "workflows": { "Package": "workflows", - "Version": "0.1.0", + "Version": "0.2.2", "Source": "Repository", - "Repository": "CRAN", - "Hash": "302e039172a133df812f778f47cfa84c" + "Repository": "RSPM", + "Hash": "160db58e0fd753a5a9afb86dfbe7b007" }, "xfun": { "Package": "xfun", - "Version": "0.12", + "Version": "0.25", "Source": "Repository", - "Repository": "CRAN", - "Hash": "ccd8453a7b9e380628f6cd2862e46cad" + "Repository": "RSPM", + "Hash": "853d45ffff0a9af1e0af017cd359f75e" }, "xml2": { "Package": "xml2", - "Version": "1.2.2", + "Version": "1.3.2", "Source": "Repository", "Repository": "CRAN", - "Hash": "63ad35854e01d8b59ca18095b5145bb7" - }, - "xtable": { - "Package": "xtable", - "Version": "1.8-4", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "b8acdf8af494d9ec19ccb2481a9b11c2" - }, - "xts": { - "Package": "xts", - "Version": "0.12-0", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "cae1f4b14c523f62b61276fb3962d4fb" + "Hash": "d4d71a75dd3ea9eb5fa28cc21f9585e2" }, "yaml": { "Package": "yaml", @@ -1522,10 +1274,10 @@ }, "yardstick": { "Package": "yardstick", - "Version": "0.0.5", + "Version": "0.0.8", "Source": "Repository", - "Repository": "CRAN", - "Hash": "fdeb9dd70a68058fca68e94af04cfc62" + "Repository": "RSPM", + "Hash": "b49509a72e4c99ef95036c061885ab07" }, "zeallot": { "Package": "zeallot", @@ -1533,13 +1285,6 @@ "Source": "Repository", "Repository": "CRAN", "Hash": "ee9b643aa8331c45d8d82eb3a137c9bc" - }, - "zoo": { - "Package": "zoo", - "Version": "1.8-7", - "Source": "Repository", - "Repository": "CRAN", - "Hash": "157e0e442de69a5b00ee5c7066d6184d" } } } diff --git a/renv/.gitignore b/renv/.gitignore index 82740ba..993aebf 100644 --- a/renv/.gitignore +++ b/renv/.gitignore @@ -1,3 +1,5 @@ +local/ +lock/ library/ python/ staging/ diff --git a/renv/activate.R b/renv/activate.R index 0aae679..5538c27 100644 --- a/renv/activate.R +++ b/renv/activate.R @@ -2,13 +2,27 @@ local({ # the requested version of renv - version <- "0.9.3-30" + version <- "0.14.0-3" # the project directory project <- getwd() + # allow environment variable to control activation + activate <- Sys.getenv("RENV_ACTIVATE_PROJECT") + if (!nzchar(activate)) { + + # don't auto-activate when R CMD INSTALL is running + if (nzchar(Sys.getenv("R_INSTALL_PKG"))) + return(FALSE) + + } + + # bail if activation was explicitly disabled + if (tolower(activate) %in% c("false", "f", "0")) + return(FALSE) + # avoid recursion - if (!is.na(Sys.getenv("RENV_R_INITIALIZING", unset = NA))) + if (nzchar(Sys.getenv("RENV_R_INITIALIZING"))) return(invisible(TRUE)) # signal that we're loading renv during R startup @@ -36,80 +50,9 @@ local({ } - # construct path to library root - root <- local({ - - path <- Sys.getenv("RENV_PATHS_LIBRARY", unset = NA) - if (!is.na(path)) - return(path) - - path <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT", unset = NA) - if (!is.na(path)) - return(file.path(path, basename(project))) - - file.path(project, "renv/library") - - }) - - # construct path to renv in library - libpath <- local({ - - prefix <- paste("R", getRversion()[1, 1:2], sep = "-") - - # include SVN revision for development versions of R - # (to avoid sharing platform-specific artefacts with released versions of R) - devel <- - identical(R.version[["status"]], "Under development (unstable)") || - identical(R.version[["nickname"]], "Unsuffered Consequences") - - if (devel) - prefix <- paste(prefix, R.version[["svn rev"]], sep = "-r") - - file.path(root, prefix, R.version$platform) - - }) - - # try to load renv from the project library - if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { - - # warn if the version of renv loaded does not match - loadedversion <- utils::packageDescription("renv", fields = "Version") - if (version != loadedversion) { - - # assume four-component versions are from GitHub; three-component - # versions are from CRAN - components <- strsplit(loadedversion, "[.-]")[[1]] - remote <- if (length(components) == 4L) - paste("rstudio/renv", loadedversion, sep = "@") - else - paste("renv", loadedversion, sep = "@") - - fmt <- paste( - "renv %1$s was loaded from project library, but renv %2$s is recorded in lockfile.", - "Use `renv::record(\"%3$s\")` to record this version in the lockfile.", - "Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library.", - sep = "\n" - ) - - msg <- sprintf(fmt, loadedversion, version, remote) - warning(msg, call. = FALSE) - - } - - # load the project - return(renv::load()) - - } - - # try to bootstrap an renv installation + # load bootstrap tools bootstrap <- function(version, library) { - # fix up repos - repos <- getOption("repos") - on.exit(options(repos = repos), add = TRUE) - repos[repos == "@CRAN@"] <- "https://cloud.r-project.org" - options(repos = repos) - # attempt to download renv tarball <- tryCatch(renv_bootstrap_download(version), error = identity) if (inherits(tarball, "error")) @@ -122,35 +65,58 @@ local({ } - renv_bootstrap_download_impl <- function(url, destfile) { + renv_bootstrap_tests_running <- function() { + getOption("renv.tests.running", default = FALSE) + } - mode <- "wb" + renv_bootstrap_repos <- function() { - # https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17715 - fixup <- - Sys.info()[["sysname"]] == "Windows" && - identical(getOption("download.file.method"), "wininet") && - substring(url, 1, 5) == "file:" + # check for repos override + repos <- Sys.getenv("RENV_CONFIG_REPOS_OVERRIDE", unset = NA) + if (!is.na(repos)) + return(repos) - if (fixup) - mode <- "w+b" + # if we're testing, re-use the test repositories + if (renv_bootstrap_tests_running()) + return(getOption("renv.tests.repos")) - download.file( - url = url, - destfile = destfile, - mode = mode, - quiet = TRUE + # retrieve current repos + repos <- getOption("repos") + + # ensure @CRAN@ entries are resolved + repos[repos == "@CRAN@"] <- getOption( + "renv.repos.cran", + "https://cloud.r-project.org" ) + # add in renv.bootstrap.repos if set + default <- c(FALLBACK = "https://cloud.r-project.org") + extra <- getOption("renv.bootstrap.repos", default = default) + repos <- c(repos, extra) + + # remove duplicates that might've snuck in + dupes <- duplicated(repos) | duplicated(names(repos)) + repos[!dupes] + } renv_bootstrap_download <- function(version) { - methods <- list( - renv_bootstrap_download_cran_latest, - renv_bootstrap_download_cran_archive, - renv_bootstrap_download_github - ) + # if the renv version number has 4 components, assume it must + # be retrieved via github + nv <- numeric_version(version) + components <- unclass(nv)[[1]] + + methods <- if (length(components) == 4L) { + list( + renv_bootstrap_download_github + ) + } else { + list( + renv_bootstrap_download_cran_latest, + renv_bootstrap_download_cran_archive + ) + } for (method in methods) { path <- tryCatch(method(version), error = identity) @@ -162,21 +128,44 @@ local({ } + renv_bootstrap_download_impl <- function(url, destfile) { + + mode <- "wb" + + # https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17715 + fixup <- + Sys.info()[["sysname"]] == "Windows" && + substring(url, 1L, 5L) == "file:" + + if (fixup) + mode <- "w+b" + + utils::download.file( + url = url, + destfile = destfile, + mode = mode, + quiet = TRUE + ) + + } + renv_bootstrap_download_cran_latest <- function(version) { - # check for renv on CRAN matching this version - db <- as.data.frame(available.packages(), stringsAsFactors = FALSE) - if (!"renv" %in% rownames(db)) - stop("renv is not available on your declared package repositories") + spec <- renv_bootstrap_download_cran_latest_find(version) - entry <- db["renv", ] - if (!identical(entry$Version, version)) - stop("renv is not available on your declared package repositories") + message("* Downloading renv ", version, " ... ", appendLF = FALSE) - message("* Downloading renv ", version, " from CRAN ... ", appendLF = FALSE) + type <- spec$type + repos <- spec$repos info <- tryCatch( - download.packages("renv", destdir = tempdir()), + utils::download.packages( + pkgs = "renv", + destdir = tempdir(), + repos = repos, + type = type, + quiet = TRUE + ), condition = identity ) @@ -185,19 +174,65 @@ local({ return(FALSE) } - message("OK") + # report success and return + message("OK (downloaded ", type, ")") info[1, 2] } + renv_bootstrap_download_cran_latest_find <- function(version) { + + # check whether binaries are supported on this system + binary <- + getOption("renv.bootstrap.binary", default = TRUE) && + !identical(.Platform$pkgType, "source") && + !identical(getOption("pkgType"), "source") && + Sys.info()[["sysname"]] %in% c("Darwin", "Windows") + + types <- c(if (binary) "binary", "source") + + # iterate over types + repositories + for (type in types) { + for (repos in renv_bootstrap_repos()) { + + # retrieve package database + db <- tryCatch( + as.data.frame( + utils::available.packages(type = type, repos = repos), + stringsAsFactors = FALSE + ), + error = identity + ) + + if (inherits(db, "error")) + next + + # check for compatible entry + entry <- db[db$Package %in% "renv" & db$Version %in% version, ] + if (nrow(entry) == 0) + next + + # found it; return spec to caller + spec <- list(entry = entry, type = type, repos = repos) + return(spec) + + } + } + + # if we got here, we failed to find renv + fmt <- "renv %s is not available from your declared package repositories" + stop(sprintf(fmt, version)) + + } + renv_bootstrap_download_cran_archive <- function(version) { name <- sprintf("renv_%s.tar.gz", version) - repos <- getOption("repos") + repos <- renv_bootstrap_repos() urls <- file.path(repos, "src/contrib/Archive/renv", name) destfile <- file.path(tempdir(), name) - message("* Downloading renv ", version, " from CRAN archive ... ", appendLF = FALSE) + message("* Downloading renv ", version, " ... ", appendLF = FALSE) for (url in urls) { @@ -256,7 +291,7 @@ local({ return(FALSE) } - message("Done!") + message("OK") return(destfile) } @@ -288,11 +323,337 @@ local({ } + renv_bootstrap_platform_prefix <- function() { + + # construct version prefix + version <- paste(R.version$major, R.version$minor, sep = ".") + prefix <- paste("R", numeric_version(version)[1, 1:2], sep = "-") + + # include SVN revision for development versions of R + # (to avoid sharing platform-specific artefacts with released versions of R) + devel <- + identical(R.version[["status"]], "Under development (unstable)") || + identical(R.version[["nickname"]], "Unsuffered Consequences") + + if (devel) + prefix <- paste(prefix, R.version[["svn rev"]], sep = "-r") + + # build list of path components + components <- c(prefix, R.version$platform) + + # include prefix if provided by user + prefix <- renv_bootstrap_platform_prefix_impl() + if (!is.na(prefix) && nzchar(prefix)) + components <- c(prefix, components) + + # build prefix + paste(components, collapse = "/") + + } + + renv_bootstrap_platform_prefix_impl <- function() { + + # if an explicit prefix has been supplied, use it + prefix <- Sys.getenv("RENV_PATHS_PREFIX", unset = NA) + if (!is.na(prefix)) + return(prefix) + + # if the user has requested an automatic prefix, generate it + auto <- Sys.getenv("RENV_PATHS_PREFIX_AUTO", unset = NA) + if (auto %in% c("TRUE", "True", "true", "1")) + return(renv_bootstrap_platform_prefix_auto()) + + # empty string on failure + "" + + } + + renv_bootstrap_platform_prefix_auto <- function() { + + prefix <- tryCatch(renv_bootstrap_platform_os(), error = identity) + if (inherits(prefix, "error") || prefix %in% "unknown") { + + msg <- paste( + "failed to infer current operating system", + "please file a bug report at https://github.com/rstudio/renv/issues", + sep = "; " + ) + + warning(msg) + + } + + prefix + + } + + renv_bootstrap_platform_os <- function() { + + sysinfo <- Sys.info() + sysname <- sysinfo[["sysname"]] + + # handle Windows + macOS up front + if (sysname == "Windows") + return("windows") + else if (sysname == "Darwin") + return("macos") + + # check for os-release files + for (file in c("/etc/os-release", "/usr/lib/os-release")) + if (file.exists(file)) + return(renv_bootstrap_platform_os_via_os_release(file, sysinfo)) + + # check for redhat-release files + if (file.exists("/etc/redhat-release")) + return(renv_bootstrap_platform_os_via_redhat_release()) + + "unknown" + + } + + renv_bootstrap_platform_os_via_os_release <- function(file, sysinfo) { + + # read /etc/os-release + release <- utils::read.table( + file = file, + sep = "=", + quote = c("\"", "'"), + col.names = c("Key", "Value"), + comment.char = "#", + stringsAsFactors = FALSE + ) + + vars <- as.list(release$Value) + names(vars) <- release$Key + + # get os name + os <- tolower(sysinfo[["sysname"]]) + + # read id + id <- "unknown" + for (field in c("ID", "ID_LIKE")) { + if (field %in% names(vars) && nzchar(vars[[field]])) { + id <- vars[[field]] + break + } + } + + # read version + version <- "unknown" + for (field in c("UBUNTU_CODENAME", "VERSION_CODENAME", "VERSION_ID", "BUILD_ID")) { + if (field %in% names(vars) && nzchar(vars[[field]])) { + version <- vars[[field]] + break + } + } + + # join together + paste(c(os, id, version), collapse = "-") + + } + + renv_bootstrap_platform_os_via_redhat_release <- function() { + + # read /etc/redhat-release + contents <- readLines("/etc/redhat-release", warn = FALSE) + + # infer id + id <- if (grepl("centos", contents, ignore.case = TRUE)) + "centos" + else if (grepl("redhat", contents, ignore.case = TRUE)) + "redhat" + else + "unknown" + + # try to find a version component (very hacky) + version <- "unknown" + + parts <- strsplit(contents, "[[:space:]]")[[1L]] + for (part in parts) { + + nv <- tryCatch(numeric_version(part), error = identity) + if (inherits(nv, "error")) + next + + version <- nv[1, 1] + break + + } + + paste(c("linux", id, version), collapse = "-") + + } + + renv_bootstrap_library_root_name <- function(project) { + + # use project name as-is if requested + asis <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT_ASIS", unset = "FALSE") + if (asis) + return(basename(project)) + + # otherwise, disambiguate based on project's path + id <- substring(renv_bootstrap_hash_text(project), 1L, 8L) + paste(basename(project), id, sep = "-") + + } + + renv_bootstrap_library_root <- function(project) { + + path <- Sys.getenv("RENV_PATHS_LIBRARY", unset = NA) + if (!is.na(path)) + return(path) + + path <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT", unset = NA) + if (!is.na(path)) { + name <- renv_bootstrap_library_root_name(project) + return(file.path(path, name)) + } + + prefix <- renv_bootstrap_profile_prefix() + paste(c(project, prefix, "renv/library"), collapse = "/") + + } + + renv_bootstrap_validate_version <- function(version) { + + loadedversion <- utils::packageDescription("renv", fields = "Version") + if (version == loadedversion) + return(TRUE) + + # assume four-component versions are from GitHub; three-component + # versions are from CRAN + components <- strsplit(loadedversion, "[.-]")[[1]] + remote <- if (length(components) == 4L) + paste("rstudio/renv", loadedversion, sep = "@") + else + paste("renv", loadedversion, sep = "@") + + fmt <- paste( + "renv %1$s was loaded from project library, but this project is configured to use renv %2$s.", + "Use `renv::record(\"%3$s\")` to record renv %1$s in the lockfile.", + "Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library.", + sep = "\n" + ) + + msg <- sprintf(fmt, loadedversion, version, remote) + warning(msg, call. = FALSE) + + FALSE + + } + + renv_bootstrap_hash_text <- function(text) { + + hashfile <- tempfile("renv-hash-") + on.exit(unlink(hashfile), add = TRUE) + + writeLines(text, con = hashfile) + tools::md5sum(hashfile) + + } + + renv_bootstrap_load <- function(project, libpath, version) { + + # try to load renv from the project library + if (!requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) + return(FALSE) + + # warn if the version of renv loaded does not match + renv_bootstrap_validate_version(version) + + # load the project + renv::load(project) + + TRUE + + } + + renv_bootstrap_profile_load <- function(project) { + + # if RENV_PROFILE is already set, just use that + profile <- Sys.getenv("RENV_PROFILE", unset = NA) + if (!is.na(profile) && nzchar(profile)) + return(profile) + + # check for a profile file (nothing to do if it doesn't exist) + path <- file.path(project, "renv/local/profile") + if (!file.exists(path)) + return(NULL) + + # read the profile, and set it if it exists + contents <- readLines(path, warn = FALSE) + if (length(contents) == 0L) + return(NULL) + + # set RENV_PROFILE + profile <- contents[[1L]] + if (nzchar(profile)) + Sys.setenv(RENV_PROFILE = profile) + + profile + + } + + renv_bootstrap_profile_prefix <- function() { + profile <- renv_bootstrap_profile_get() + if (!is.null(profile)) + return(file.path("renv/profiles", profile)) + } + + renv_bootstrap_profile_get <- function() { + profile <- Sys.getenv("RENV_PROFILE", unset = "") + renv_bootstrap_profile_normalize(profile) + } + + renv_bootstrap_profile_set <- function(profile) { + profile <- renv_bootstrap_profile_normalize(profile) + if (is.null(profile)) + Sys.unsetenv("RENV_PROFILE") + else + Sys.setenv(RENV_PROFILE = profile) + } + + renv_bootstrap_profile_normalize <- function(profile) { + + if (is.null(profile) || profile %in% c("", "default")) + return(NULL) + + profile + + } + + # load the renv profile, if any + renv_bootstrap_profile_load(project) + + # construct path to library root + root <- renv_bootstrap_library_root(project) + + # construct library prefix for platform + prefix <- renv_bootstrap_platform_prefix() + + # construct full libpath + libpath <- file.path(root, prefix) + + # attempt to load + if (renv_bootstrap_load(project, libpath, version)) + return(TRUE) + + # load failed; inform user we're about to bootstrap + prefix <- paste("# Bootstrapping renv", version) + postfix <- paste(rep.int("-", 77L - nchar(prefix)), collapse = "") + header <- paste(prefix, postfix) + message(header) + + # perform bootstrap bootstrap(version, libpath) + # exit early if we're just testing bootstrap + if (!is.na(Sys.getenv("RENV_BOOTSTRAP_INSTALL_ONLY", unset = NA))) + return(TRUE) + # try again to load if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { - message("Successfully installed and loaded renv ", version, ".") + message("* Successfully installed and loaded renv ", version, ".") return(renv::load()) } diff --git a/renv/settings.dcf b/renv/settings.dcf index 11a53ea..fc4e479 100644 --- a/renv/settings.dcf +++ b/renv/settings.dcf @@ -1,6 +1,8 @@ external.libraries: ignored.packages: package.dependency.fields: Imports, Depends, LinkingTo +r.version: snapshot.type: implicit use.cache: TRUE vcs.ignore.library: TRUE +vcs.ignore.local: TRUE From 83453eda93f1595268f77128cfd671a6089ed4a4 Mon Sep 17 00:00:00 2001 From: ercbk Date: Tue, 10 Aug 2021 21:59:48 -0400 Subject: [PATCH 06/17] readme: minor edits --- README.Rmd | 4 ++-- README.md | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.Rmd b/README.Rmd index fc4647a..3c52d72 100644 --- a/README.Rmd +++ b/README.Rmd @@ -7,7 +7,7 @@ output: github_document ![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) -Experiments conducted in May 2020 +Experiments conducted in May 2020. Packages had to be updated in August 2021, but the scripts haven't been re-ran to make sure everything still works. Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. @@ -339,7 +339,7 @@ b_r + e_r + plot_layout(guides = "auto") + ## Summary * Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error if your data size is a few thousand rows - + Kuhn-Johnson's runtime starts to really balloon once you get into dataset with over a thousand rows. + + Kuhn-Johnson's runtime starts to really balloon once you get into datasets with over a thousand rows. + The extra folds made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka's, it was minutes. * For smaller datasets, you should have at least 3 repeats when running Rashka's method. * This is just one dataset, but I still found it surprising how little a difference repeats made in reducing generalizaion error. The benefit only kicked in with the dataset that had hundred rows. diff --git a/README.md b/README.md index a211fe8..64863e5 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,12 @@ # Nested Cross-Validation: Comparing Methods and Implementations - ![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) -Experiments conducted in May 2020 +Experiments conducted in May 2020. Packages had to be updated in August +2021, but the scripts haven’t been re-ran to make sure everything still +works. Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to @@ -191,7 +192,7 @@ implementation and method used. a similar amount of generalization error if your data size is a few thousand rows - Kuhn-Johnson’s runtime starts to really balloon once you get - into dataset with over a thousand rows. + into datasets with over a thousand rows. - The extra folds made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka’s, it was minutes. - For smaller datasets, you should have at least 3 repeats when From 72d68e0afc31e1591cf94844776c54825c2e8118 Mon Sep 17 00:00:00 2001 From: ercbk Date: Tue, 10 Aug 2021 22:03:47 -0400 Subject: [PATCH 07/17] readme: minor edit --- README.Rmd | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.Rmd b/README.Rmd index 3c52d72..b993235 100644 --- a/README.Rmd +++ b/README.Rmd @@ -337,7 +337,7 @@ b_r + e_r + plot_layout(guides = "auto") + * n = 800 remains under 2.5% percent error for all repeat values, but also shows considerable volatility. -## Summary +## Discussion * Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error if your data size is a few thousand rows + Kuhn-Johnson's runtime starts to really balloon once you get into datasets with over a thousand rows. + The extra folds made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka's, it was minutes. diff --git a/README.md b/README.md index 64863e5..4665aad 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ implementation and method used. - n = 800 remains under 2.5% percent error for all repeat values, but also shows considerable volatility. -## Summary +## Discussion - Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error if your data size is a few From bfb7af6d74b32bdeb80775f40fb532efafe524ab Mon Sep 17 00:00:00 2001 From: ercbk Date: Wed, 11 Aug 2021 10:54:12 -0400 Subject: [PATCH 08/17] readme: added notes to discussion section --- README.Rmd | 20 ++++++++++----- README.md | 74 ++++++++++++++++++++++++++++++++++++++---------------- 2 files changed, 67 insertions(+), 27 deletions(-) diff --git a/README.Rmd b/README.Rmd index b993235..392cfa1 100644 --- a/README.Rmd +++ b/README.Rmd @@ -9,9 +9,9 @@ output: github_document Experiments conducted in May 2020. Packages had to be updated in August 2021, but the scripts haven't been re-ran to make sure everything still works. -Nested cross-validation has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. +Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. -The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. +While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. The other diffferences are just the number of folds/resamples used in the outer and inner loops which are essentially just tuning parameters. Various elements of the technique affect the run times and performance. These include: @@ -132,7 +132,7 @@ durations * The chosen algorithm with hyperparameters is fit on the entire training set, and the resulting final model predicts on a 100K row Friedman dataset. * The percent error between the the average mean absolute error (MAE) across the outer-loop folds and the MAE of the predictions on this 100K dataset is calculated for each combination of repeat, data size, and method. * To make this experiment manageable in terms of runtimes, I am using AWS instances: a r5.2xlarge for the Elastic Net and a r5.24xlarge for Random Forest. - + Also see the Other Notes section + + Also see the Discussion section * Iterating through different numbers of repeats, sample sizes, and methods makes a functional approach more appropriate than running imperative scripts. Also, given the long runtimes and impermanent nature of my internet connection, it would also be nice to cache each iteration as it finishes. The [{drake}](https://github.com/ropensci/drake) package is superb on both counts, so I'm using it to orchestrate. ```{r perf_build_times_kj, echo=FALSE, message=FALSE} @@ -338,11 +338,19 @@ b_r + e_r + plot_layout(guides = "auto") + ## Discussion - * Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error if your data size is a few thousand rows + * The performance experimental framework used here could be useful as a way to gain insight into the amounts and types of resources that a project's first steps might require. For example, testing simiulated data before collection of actual data begins. + * The elasticnet model was slower to train than the random forest for the 100 row dataset. Compute resources should be optimized for each algorithm. For example, the number of vCPUs capable of being utilized by a random forest algorithm is much higher than number for an elasticnet algorithm. The elasticnet only used the number of vCPUs that matched the number of training folds while the random forest used all available vCPUs. Using a sparse matrix or another package (e.g. biglasso) might help to lower training times for elasticnet. + * Adjusting the inner-loop strategy seems to have the most effect on the volatility of the results. + * For data sizes of a few thousand rows, Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error as compared to the Raschka method. The similar results in generalization error might be specific to this dataset though. + Kuhn-Johnson's runtime starts to really balloon once you get into datasets with over a thousand rows. - + The extra folds made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka's, it was minutes. + + The extra folds in the outer loop made a huge difference. With Kuhn-Johnson, the runtimes were hours, and with Raschka's, it was minutes. * For smaller datasets, you should have at least 3 repeats when running Rashka's method. - * This is just one dataset, but I still found it surprising how little a difference repeats made in reducing generalizaion error. The benefit only kicked in with the dataset that had hundred rows. + * This is just one dataset, but I still found it surprising how little a difference repeats made in reducing generalizaion error. The benefit only kicked in with the dataset that had hundred rows. + + +## Next Steps + * Raschka's method using the majority vote method from Kuhn-Johnson for the final hyperparameter settings might be an additional optimization step. If the final k-fold cv can be discarded without much loss in generalization error, then maybe training times can be shortened further. + * There should be a version of this technique that's capable of working for time series. I have ideas, so it might be something I'll work on for a future project. diff --git a/README.md b/README.md index 4665aad..4d49b48 100644 --- a/README.md +++ b/README.md @@ -8,22 +8,23 @@ Experiments conducted in May 2020. Packages had to be updated in August 2021, but the scripts haven’t been re-ran to make sure everything still works. -Nested cross-validation has become a recommended technique for -situations in which the size of our dataset is insufficient to -simultaneously handle hyperparameter tuning and algorithm comparison. -Using standard methods such as k-fold cross-validation in these cases -may result in substantial increases in optimization bias where the more -models that are trained on a fold means there’s a greater opportunity -for a model to achieve a low score by chance. Nested cross-validation -has been shown to produce less biased, out-of-sample error estimates -even using datasets with only hundreds of rows and therefore gives a -better estimation of generalized performance. - +Nested cross-validation (or double cross-validaation) has become a +recommended technique for situations in which the size of our dataset is +insufficient to simultaneously handle hyperparameter tuning and +algorithm comparison. Using standard methods such as k-fold +cross-validation in these cases may result in substantial increases in +optimization bias where the more models that are trained on a fold means +there’s a greater opportunity for a model to achieve a low score by +chance. Nested cross-validation has been shown to produce less biased, +out-of-sample error estimates even using datasets with only hundreds of +rows and therefore gives a better estimation of generalized performance. The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being -trained during the process. While researching this technique, I found -two slightly different variations of performing nested cross-validation -— one authored by [Sabastian +trained during the process. + +While researching this technique, I found two slightly different +variations of performing nested cross-validation — one authored by +[Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). @@ -34,7 +35,9 @@ Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning -procedure the most often is the set used to fit the final model. +procedure the most often is the set used to fit the final model. The +other diffferences are just the number of folds/resamples used in the +outer and inner loops which are essentially just tuning parameters. Various elements of the technique affect the run times and performance. These include: @@ -146,7 +149,7 @@ implementation and method used. - To make this experiment manageable in terms of runtimes, I am using AWS instances: a r5.2xlarge for the Elastic Net and a r5.24xlarge for Random Forest. - - Also see the Other Notes section + - Also see the Discussion section - Iterating through different numbers of repeats, sample sizes, and methods makes a functional approach more appropriate than running imperative scripts. Also, given the long runtimes and impermanent @@ -188,19 +191,48 @@ implementation and method used. ## Discussion -- Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for - a similar amount of generalization error if your data size is a few - thousand rows +- The performance experimental framework used here could be useful as + a way to gain insight into the amounts and types of resources that a + project’s first steps might require. For example, testing simiulated + data before collection of actual data begins. +- The elasticnet model was slower to train than the random forest for + the 100 row dataset. Compute resources should be optimized for each + algorithm. For example, the number of vCPUs capable of being + utilized by a random forest algorithm is much higher than number for + an elasticnet algorithm. The elasticnet only used the number of + vCPUs that matched the number of training folds while the random + forest used all available vCPUs. Using a sparse matrix or another + package (e.g. biglasso) might help to lower training times for + elasticnet. +- Adjusting the inner-loop strategy seems to have the most effect on + the volatility of the results. +- For data sizes of a few thousand rows, Kuhn-Johnson trains 50x as + many models; takes 8x longer to run; for a similar amount of + generalization error as compared to the Raschka method. The similar + results in generalization error might be specific to this dataset + though. - Kuhn-Johnson’s runtime starts to really balloon once you get into datasets with over a thousand rows. - - The extra folds made a huge difference. With Kuhn-Johnson, the - runtimes were hours, and with Raschka’s, it was minutes. + - The extra folds in the outer loop made a huge difference. With + Kuhn-Johnson, the runtimes were hours, and with Raschka’s, it + was minutes. - For smaller datasets, you should have at least 3 repeats when running Rashka’s method. - This is just one dataset, but I still found it surprising how little a difference repeats made in reducing generalizaion error. The benefit only kicked in with the dataset that had hundred rows. +## Next Steps + +- Raschka’s method using the majority vote method from Kuhn-Johnson + for the final hyperparameter settings might be an additional + optimization step. If the final k-fold cv can be discarded without + much loss in generalization error, then maybe training times can be + shortened further. +- There should be a version of this technique that’s capable of + working for time series. I have ideas, so it might be something I’ll + work on for a future project. + ## References Boulesteix, AL, and C Strobl. 2009. “Optimal Classifier Selection and From d7a91752cde4ff55a2961c3b8b740ad6dc72ae37 Mon Sep 17 00:00:00 2001 From: ercbk Date: Wed, 11 Aug 2021 14:24:56 -0400 Subject: [PATCH 09/17] readme: minor edit --- README.Rmd | 2 +- README.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/README.Rmd b/README.Rmd index 392cfa1..9af24f8 100644 --- a/README.Rmd +++ b/README.Rmd @@ -7,7 +7,7 @@ output: github_document ![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) -Experiments conducted in May 2020. Packages had to be updated in August 2021, but the scripts haven't been re-ran to make sure everything still works. +Experiments conducted in May 2020. Packages in renv.lock had to be updated in August 2021, but all scripts haven't been re-ran to make sure everything still works. So, some refactoring of code may be necessary in order to reproduce results. Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. diff --git a/README.md b/README.md index 4d49b48..4eb97f6 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,10 @@ ![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) -Experiments conducted in May 2020. Packages had to be updated in August -2021, but the scripts haven’t been re-ran to make sure everything still -works. +Experiments conducted in May 2020. Packages in renv.lock had to be +updated in August 2021, but all scripts haven’t been re-ran to make sure +everything still works. So, some refactoring of code may be necessary in +order to reproduce results. Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is From 3e4241899e2c5e99035320a96e9f5aa0602d8c19 Mon Sep 17 00:00:00 2001 From: ercbk Date: Fri, 13 Aug 2021 07:52:17 -0400 Subject: [PATCH 10/17] readme: minor edits --- README.Rmd | 10 +++++----- README.md | 25 ++++++++++++++++--------- 2 files changed, 21 insertions(+), 14 deletions(-) diff --git a/README.Rmd b/README.Rmd index 9af24f8..724b53c 100644 --- a/README.Rmd +++ b/README.Rmd @@ -11,7 +11,7 @@ Experiments conducted in May 2020. Packages in renv.lock had to be updated in Au Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. -While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. The other diffferences are just the number of folds/resamples used in the outer and inner loops which are essentially just tuning parameters. +While researching this technique, I found two slightly different variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell Johnson](https://www.tidymodels.org/learn/work/nested-resampling/). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. Therefore, the hyperparameter tuning that takes place in the inner-loop during nested cross-validation is only in service of algorithm selection. Kuhn-Johnson uses majority vote. Whichever set of hyperparameter values has been chosen during the inner-loop tuning procedure the most often is the set used to fit the final model. The other diffferences are just the number of folds/resamples used in the outer and inner loops which are essentially just tuning parameters. Various elements of the technique affect the run times and performance. These include: @@ -29,10 +29,10 @@ The results from these experiments should give us an idea about which methodolog ## Recommendations: - * Use {mlr3} or other R model packages outside of the {tidymodels} ecosystem and code the nested cross-validation loops manually. - * For data sizes in the low thousands, Raschka's method performs just as well as Kuhn-Johnson's but is substantially faster. - * For data sizes in the hundreds, Raschka's method with at least 3 repeats performs just as well as Kuhn-Johnson's but is still substantially faster even with the repeats. - * Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the r5.#xlarge instances ran fastest. + * For faster training times, use {mlr3} or other R model packages outside of the {tidymodels} ecosystem and code the nested cross-validation loops manually (Code: [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). + * Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the r5.#xlarge instances ran fastest. The most efficient number of vCPUs may vary according to the algorithm. + * For the data in this experiment with row numbers in the low thousands, Raschka's method performed just as well as Kuhn-Johnson's but was substantially faster. + * For the data in this experiment with row numbers in the hundreds, Raschka's method with at least 3 repeats performed just as well as Kuhn-Johnson's but was still substantially faster even with the repeats. diff --git a/README.md b/README.md index 4eb97f6..ea87f5f 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ variations of performing nested cross-validation — one authored by [Sabastian Raschka](https://github.com/rasbt/stat479-machine-learning-fs19/blob/master/11_eval4-algo/code/11-eval4-algo__nested-cv_verbose1.ipynb) and the other by [Max Kuhn and Kjell -Johnson](https://tidymodels.github.io/rsample/articles/Applications/Nested_Resampling.html). +Johnson](https://www.tidymodels.org/learn/work/nested-resampling/). After the nested cross-validation procedure and an algorithm is chosen, Raschka performs an extra k-fold cross-validation using the inner-loop cv strategy on the entire training set in order to tune his final model. @@ -64,16 +64,23 @@ lower training times, lower costs, and lower generalization error. ## Recommendations: -- Use {mlr3} or other R model packages outside of the {tidymodels} - ecosystem and code the nested cross-validation loops manually. -- For data sizes in the low thousands, Raschka’s method performs just - as well as Kuhn-Johnson’s but is substantially faster. -- For data sizes in the hundreds, Raschka’s method with at least 3 - repeats performs just as well as Kuhn-Johnson’s but is still - substantially faster even with the repeats. +- For faster training times, use {mlr3} or other R model packages + outside of the {tidymodels} ecosystem and code the nested + cross-validation loops manually (Code: + [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), + [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), + [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). - Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the - r5.\#xlarge instances ran fastest. + r5.\#xlarge instances ran fastest. The most efficient number of + vCPUs may vary according to the algorithm. +- For the data in this experiment with row numbers in the low + thousands, Raschka’s method performed just as well as Kuhn-Johnson’s + but was substantially faster. +- For the data in this experiment with row numbers in the hundreds, + Raschka’s method with at least 3 repeats performed just as well as + Kuhn-Johnson’s but was still substantially faster even with the + repeats. ## Duration From 75ba163d52bc414e6ceb6fecda47034d6d908f4f Mon Sep 17 00:00:00 2001 From: ercbk Date: Fri, 13 Aug 2021 07:55:18 -0400 Subject: [PATCH 11/17] readme: minor edits --- README.Rmd | 2 +- README.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/README.Rmd b/README.Rmd index 724b53c..97380e9 100644 --- a/README.Rmd +++ b/README.Rmd @@ -7,7 +7,7 @@ output: github_document ![](images/ncv.png) [![DOI](https://zenodo.org/badge/242267104.svg)](https://zenodo.org/badge/latestdoi/242267104) -Experiments conducted in May 2020. Packages in renv.lock had to be updated in August 2021, but all scripts haven't been re-ran to make sure everything still works. So, some refactoring of code may be necessary in order to reproduce results. +Experiments conducted in May 2020. Packages in renv.lock had to be updated in August 2021, but all scripts haven't been re-ran to make sure everything still works. So, some refactoring of code may be necessary in order to reproduce results (e.g. {future} progress bar implementation and plan options have changed). Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is insufficient to simultaneously handle hyperparameter tuning and algorithm comparison. Using standard methods such as k-fold cross-validation in these cases may result in substantial increases in optimization bias where the more models that are trained on a fold means there's a greater opportunity for a model to achieve a low score by chance. Nested cross-validation has been shown to produce less biased, out-of-sample error estimates even using datasets with only hundreds of rows and therefore gives a better estimation of generalized performance. The primary issue with this technique is that it is usually computationally expensive with potentially tens of 1000s of models being trained during the process. diff --git a/README.md b/README.md index ea87f5f..9f65a31 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,8 @@ Experiments conducted in May 2020. Packages in renv.lock had to be updated in August 2021, but all scripts haven’t been re-ran to make sure everything still works. So, some refactoring of code may be necessary in -order to reproduce results. +order to reproduce results (e.g. {future} progress bar implementation +and plan options have changed). Nested cross-validation (or double cross-validaation) has become a recommended technique for situations in which the size of our dataset is From d037c7740994a442557b100cffb2373a2b1d8885 Mon Sep 17 00:00:00 2001 From: ercbk Date: Fri, 13 Aug 2021 08:10:15 -0400 Subject: [PATCH 12/17] readme: minor edits --- README.Rmd | 3 ++- README.md | 10 +++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/README.Rmd b/README.Rmd index 97380e9..86b1ec4 100644 --- a/README.Rmd +++ b/README.Rmd @@ -29,7 +29,7 @@ The results from these experiments should give us an idea about which methodolog ## Recommendations: - * For faster training times, use {mlr3} or other R model packages outside of the {tidymodels} ecosystem and code the nested cross-validation loops manually (Code: [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). + * For faster training times, use {mlr3} (caveat: see Discussion) or other R model packages outside of the {tidymodels} ecosystem and code the nested cross-validation loops manually (Code: [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). * Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the r5.#xlarge instances ran fastest. The most efficient number of vCPUs may vary according to the algorithm. * For the data in this experiment with row numbers in the low thousands, Raschka's method performed just as well as Kuhn-Johnson's but was substantially faster. * For the data in this experiment with row numbers in the hundreds, Raschka's method with at least 3 repeats performed just as well as Kuhn-Johnson's but was still substantially faster even with the repeats. @@ -339,6 +339,7 @@ b_r + e_r + plot_layout(guides = "auto") + ## Discussion * The performance experimental framework used here could be useful as a way to gain insight into the amounts and types of resources that a project's first steps might require. For example, testing simiulated data before collection of actual data begins. + * {mlr3} wasn't included in the Kuhn-Johnson section of the duration experiment, because with the extra folds/resamples, the RAM usage rapidly increases to the maximum and either locks-up or slows the training time tremendously. I haven't explored this further. * The elasticnet model was slower to train than the random forest for the 100 row dataset. Compute resources should be optimized for each algorithm. For example, the number of vCPUs capable of being utilized by a random forest algorithm is much higher than number for an elasticnet algorithm. The elasticnet only used the number of vCPUs that matched the number of training folds while the random forest used all available vCPUs. Using a sparse matrix or another package (e.g. biglasso) might help to lower training times for elasticnet. * Adjusting the inner-loop strategy seems to have the most effect on the volatility of the results. * For data sizes of a few thousand rows, Kuhn-Johnson trains 50x as many models; takes 8x longer to run; for a similar amount of generalization error as compared to the Raschka method. The similar results in generalization error might be specific to this dataset though. diff --git a/README.md b/README.md index 9f65a31..dc86012 100644 --- a/README.md +++ b/README.md @@ -65,9 +65,9 @@ lower training times, lower costs, and lower generalization error. ## Recommendations: -- For faster training times, use {mlr3} or other R model packages - outside of the {tidymodels} ecosystem and code the nested - cross-validation loops manually (Code: +- For faster training times, use {mlr3} (caveat: see Discussion) or + other R model packages outside of the {tidymodels} ecosystem and + code the nested cross-validation loops manually (Code: [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). @@ -204,6 +204,10 @@ implementation and method used. a way to gain insight into the amounts and types of resources that a project’s first steps might require. For example, testing simiulated data before collection of actual data begins. +- {mlr3} wasn’t included in the Kuhn-Johnson section of the duration + experiment, because with the extra folds/resamples, the RAM usage + rapidly increases to the maximum and either locks-up or slows the + training time tremendously. I haven’t explored this further. - The elasticnet model was slower to train than the random forest for the 100 row dataset. Compute resources should be optimized for each algorithm. For example, the number of vCPUs capable of being From 3f21283b40200e36e63048ebf411360c7ddf7354 Mon Sep 17 00:00:00 2001 From: ercbk Date: Fri, 13 Aug 2021 08:17:05 -0400 Subject: [PATCH 13/17] readme: minor edits --- README.Rmd | 1 + README.md | 3 +++ 2 files changed, 4 insertions(+) diff --git a/README.Rmd b/README.Rmd index 86b1ec4..bfa1027 100644 --- a/README.Rmd +++ b/README.Rmd @@ -351,6 +351,7 @@ b_r + e_r + plot_layout(guides = "auto") + ## Next Steps * Raschka's method using the majority vote method from Kuhn-Johnson for the final hyperparameter settings might be an additional optimization step. If the final k-fold cv can be discarded without much loss in generalization error, then maybe training times can be shortened further. + * There's probably room to increase the number of folds in the inner-loop of Raschka's method in order to gain more stable results while keeping the training time comparitively low. * There should be a version of this technique that's capable of working for time series. I have ideas, so it might be something I'll work on for a future project. diff --git a/README.md b/README.md index dc86012..e8137d3 100644 --- a/README.md +++ b/README.md @@ -242,6 +242,9 @@ implementation and method used. optimization step. If the final k-fold cv can be discarded without much loss in generalization error, then maybe training times can be shortened further. +- There’s probably room to increase the number of folds in the + inner-loop of Raschka’s method in order to gain more stable results + while keeping the training time comparitively low. - There should be a version of this technique that’s capable of working for time series. I have ideas, so it might be something I’ll work on for a future project. From bfc048c704ec79d412e2658ffdab4d897b55a650 Mon Sep 17 00:00:00 2001 From: ercbk Date: Fri, 13 Aug 2021 15:56:06 -0400 Subject: [PATCH 14/17] readme: minor edits --- README.Rmd | 7 ++++--- README.md | 27 +++++++++++++++------------ 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/README.Rmd b/README.Rmd index bfa1027..dd6bc19 100644 --- a/README.Rmd +++ b/README.Rmd @@ -338,7 +338,6 @@ b_r + e_r + plot_layout(guides = "auto") + ## Discussion - * The performance experimental framework used here could be useful as a way to gain insight into the amounts and types of resources that a project's first steps might require. For example, testing simiulated data before collection of actual data begins. * {mlr3} wasn't included in the Kuhn-Johnson section of the duration experiment, because with the extra folds/resamples, the RAM usage rapidly increases to the maximum and either locks-up or slows the training time tremendously. I haven't explored this further. * The elasticnet model was slower to train than the random forest for the 100 row dataset. Compute resources should be optimized for each algorithm. For example, the number of vCPUs capable of being utilized by a random forest algorithm is much higher than number for an elasticnet algorithm. The elasticnet only used the number of vCPUs that matched the number of training folds while the random forest used all available vCPUs. Using a sparse matrix or another package (e.g. biglasso) might help to lower training times for elasticnet. * Adjusting the inner-loop strategy seems to have the most effect on the volatility of the results. @@ -350,8 +349,10 @@ b_r + e_r + plot_layout(guides = "auto") + ## Next Steps - * Raschka's method using the majority vote method from Kuhn-Johnson for the final hyperparameter settings might be an additional optimization step. If the final k-fold cv can be discarded without much loss in generalization error, then maybe training times can be shortened further. - * There's probably room to increase the number of folds in the inner-loop of Raschka's method in order to gain more stable results while keeping the training time comparitively low. + * The performance experimental framework used here could be useful as a way to gain insight into the amounts and types of resources that a project's first steps might require. For example, testing simiulated data before collection of actual data begins. I might try to see if there's much difficulty extending it to {mlr3} (assuming I can figure out the RAM issue) and Python. + * Experiment with Raschka's method more. + + Raschka's method using the majority vote method from Kuhn-Johnson for the final hyperparameter settings might be an additional optimization step. If the final k-fold cv can be discarded without much loss in generalization error, then maybe training times can be shortened further. + + There's probably room to increase the number of folds in the inner-loop of Raschka's method in order to gain more stable results while keeping the training time comparitively low. * There should be a version of this technique that's capable of working for time series. I have ideas, so it might be something I'll work on for a future project. diff --git a/README.md b/README.md index e8137d3..42becbc 100644 --- a/README.md +++ b/README.md @@ -200,10 +200,6 @@ implementation and method used. ## Discussion -- The performance experimental framework used here could be useful as - a way to gain insight into the amounts and types of resources that a - project’s first steps might require. For example, testing simiulated - data before collection of actual data begins. - {mlr3} wasn’t included in the Kuhn-Johnson section of the duration experiment, because with the extra folds/resamples, the RAM usage rapidly increases to the maximum and either locks-up or slows the @@ -237,14 +233,21 @@ implementation and method used. ## Next Steps -- Raschka’s method using the majority vote method from Kuhn-Johnson - for the final hyperparameter settings might be an additional - optimization step. If the final k-fold cv can be discarded without - much loss in generalization error, then maybe training times can be - shortened further. -- There’s probably room to increase the number of folds in the - inner-loop of Raschka’s method in order to gain more stable results - while keeping the training time comparitively low. +- The performance experimental framework used here could be useful as + a way to gain insight into the amounts and types of resources that a + project’s first steps might require. For example, testing simiulated + data before collection of actual data begins. I might try to see if + there’s much difficulty extending it to {mlr3} (assuming I can + figure out the RAM issue) and Python. +- Experiment with Raschka’s method more. + - Raschka’s method using the majority vote method from + Kuhn-Johnson for the final hyperparameter settings might be an + additional optimization step. If the final k-fold cv can be + discarded without much loss in generalization error, then maybe + training times can be shortened further. + - There’s probably room to increase the number of folds in the + inner-loop of Raschka’s method in order to gain more stable + results while keeping the training time comparitively low. - There should be a version of this technique that’s capable of working for time series. I have ideas, so it might be something I’ll work on for a future project. From 5f5b8503a3f74565491df39d668b8c6485035071 Mon Sep 17 00:00:00 2001 From: Eric Book Date: Mon, 12 Dec 2022 11:22:43 -0500 Subject: [PATCH 15/17] Update README.md updated recommendations with {parsnip} update --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 42becbc..b3f3516 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,7 @@ lower training times, lower costs, and lower generalization error. [mlr3](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/raschka/nested-cv-mlr3-raschka.R), [ranger-kj](https://github.com/ercbk/nested-cross-validation-comparison/blob/master/duration-experiment/kuhn-johnson/nested-cv-ranger-kj.R), [Kuhn-Johnson](https://www.tidymodels.org/learn/work/nested-resampling/)). + - 2022-11-11 {parsnip} has undergone changes potentially leading to a 3-fold speed-up ([link](https://parsnip.tidymodels.org/news/index.html#parsnip-103)) - Choose compute resources with large amounts of RAM instead of opting for powerful processors. From the AWS cpu product line, I found the r5.\#xlarge instances ran fastest. The most efficient number of @@ -133,6 +134,7 @@ implementation and method used. larger, I think {h2o} would be more competitive. - The {tidymodels} packages, {parsnip} and {tune}, add substantial overhead. + - 2022-11-11 {parsnip} has undergone changes potentially leading to a 3-fold speed-up ([link](https://parsnip.tidymodels.org/news/index.html#parsnip-103)) ## Performance From 233ef15d3d034a5ca310bd08fb463b448adf3581 Mon Sep 17 00:00:00 2001 From: Jordi Rosell Date: Fri, 11 Oct 2024 17:28:36 +0200 Subject: [PATCH 16/17] Update plan-raschka.R --- performance-experiment/Raschka/plan-raschka.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/performance-experiment/Raschka/plan-raschka.R b/performance-experiment/Raschka/plan-raschka.R index e439e82..ffb37c6 100644 --- a/performance-experiment/Raschka/plan-raschka.R +++ b/performance-experiment/Raschka/plan-raschka.R @@ -1,4 +1,4 @@ -# Kuhn-Johnson drake plan +# Raschka drake plan # Notes: From d0016e1cb08640749cadad2addf1ce824c7802da Mon Sep 17 00:00:00 2001 From: Eric Book Date: Sun, 13 Oct 2024 09:17:56 -0400 Subject: [PATCH 17/17] Update _drake-raschka.R Corrected typo in comments --- _drake-raschka.R | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_drake-raschka.R b/_drake-raschka.R index aa6ccc8..a66c8d1 100644 --- a/_drake-raschka.R +++ b/_drake-raschka.R @@ -1,8 +1,8 @@ -# drake make file for Kuhn-Johnson performance experiment +# drake make file for Raschka performance experiment # Notes: -# 1. see plan-kj.R for more details on how this thing works +# 1. see plan-raschka.R for more details on how this thing works # 2. link to {future} issue with instructions on special PuTTY settings, https://github.com/HenrikBengtsson/future/issues/370