Iterated F-Racing for mixed spaces and dependencies
Source:vignettes/advanced_tune.Rmd
advanced_tune.RmdThe package supports a larger number of tuning algorithms, which can all be looked up and selected via TuneControl(). One of the cooler algorithms is iterated F-racing from the irace::irace() package (technical description here). This not only works for arbitrary parameter types (numeric, integer, discrete, logical), but also for so-called dependent / hierarchical parameters:
ps = makeParamSet(
makeNumericParam("C", lower = -12, upper = 12, trafo = function(x) 2^x),
makeDiscreteParam("kernel", values = c("vanilladot", "polydot", "rbfdot")),
makeNumericParam("sigma", lower = -12, upper = 12, trafo = function(x) 2^x,
requires = quote(kernel == "rbfdot")),
makeIntegerParam("degree", lower = 2L, upper = 5L,
requires = quote(kernel == "polydot"))
)
ctrl = makeTuneControlIrace(maxExperiments = 200L)
rdesc = makeResampleDesc("Holdout")
res = tuneParams("classif.ksvm", iris.task, rdesc, par.set = ps, control = ctrl, show.info = FALSE)
print(head(as.data.frame(res$opt.path)))
## C kernel sigma degree mmce.test.mean dob eol
## 1 -5.9282731 vanilladot NA NA 0.08 1 NA
## 2 -2.3600395 vanilladot NA NA 0.04 1 NA
## 3 7.2604604 vanilladot NA NA 0.02 1 NA
## 4 -5.3486126 vanilladot NA NA 0.12 1 NA
## 5 -5.7052889 rbfdot -11.885123 NA 0.70 1 NA
## 6 0.7742423 rbfdot 7.386183 NA 0.68 1 NA
## error.message exec.time
## 1 <NA> 0.069
## 2 <NA> 0.812
## 3 <NA> 0.011
## 4 <NA> 0.011
## 5 <NA> 0.021
## 6 <NA> 0.013See how we made the kernel parameters like sigma and degree dependent on the kernel selection parameters? This approach allows you to tune parameters of multiple kernels at once, efficiently concentrating on the ones which work best for your given data set.
Tuning across whole model spaces with ModelMultiplexer
We can now take the following example even one step further. If we use the makeModelMultiplexer() we can tune over different model classes at once, just as we did with the SVM kernels above.
base.learners = list(
makeLearner("classif.ksvm"),
makeLearner("classif.randomForest")
)
lrn = makeModelMultiplexer(base.learners)Function makeModelMultiplexerParamSet() offers a simple way to construct a parameter set for tuning: The parameter names are prefixed automatically and the requires element is set, too, to make all parameters subordinate to selected.learner.
ps = makeModelMultiplexerParamSet(lrn,
makeNumericParam("sigma", lower = -12, upper = 12, trafo = function(x) 2^x),
makeIntegerParam("ntree", lower = 1L, upper = 500L)
)
print(ps)
## Type len Def
## selected.learner discrete - -
## classif.ksvm.sigma numeric - -
## classif.randomForest.ntree integer - -
## Constr Req Tunable
## selected.learner classif.ksvm,classif.randomForest - TRUE
## classif.ksvm.sigma -12 to 12 Y TRUE
## classif.randomForest.ntree 1 to 500 Y TRUE
## Trafo
## selected.learner -
## classif.ksvm.sigma Y
## classif.randomForest.ntree -
rdesc = makeResampleDesc("CV", iters = 2L)
ctrl = makeTuneControlIrace(maxExperiments = 200L)
res = tuneParams(lrn, iris.task, rdesc, par.set = ps,
control = ctrl, show.info = FALSE)
print(head(as.data.frame(res$opt.path)))
## selected.learner classif.ksvm.sigma classif.randomForest.ntree
## 1 classif.ksvm -10.178141 NA
## 2 classif.randomForest NA 26
## 3 classif.randomForest NA 415
## 4 classif.randomForest NA 182
## 5 classif.ksvm 4.035821 NA
## 6 classif.randomForest NA 295
## mmce.test.mean dob eol error.message exec.time
## 1 0.71333333 1 NA <NA> 0.026
## 2 0.08000000 1 NA <NA> 0.023
## 3 0.07333333 1 NA <NA> 0.040
## 4 0.08000000 1 NA <NA> 0.038
## 5 0.31333333 1 NA <NA> 0.033
## 6 0.07333333 1 NA <NA> 0.049Multi-criteria evaluation and optimization
During tuning you might want to optimize multiple, potentially conflicting, performance measures simultaneously.
In the following example we aim to minimize both, the false positive and the false negative rates (fpr and fnr). We again tune the hyperparameters of an SVM (function kernlab::ksvm()) with a radial basis kernel and use sonar.task() for illustration. As search strategy we choose a random search.
For all available multi-criteria tuning algorithms see TuneMultiCritControl().
ps = makeParamSet(
makeNumericParam("C", lower = -12, upper = 12, trafo = function(x) 2^x),
makeNumericParam("sigma", lower = -12, upper = 12, trafo = function(x) 2^x)
)
ctrl = makeTuneMultiCritControlRandom(maxit = 30L)
rdesc = makeResampleDesc("Holdout")
res = tuneParamsMultiCrit("classif.ksvm", task = sonar.task,
resampling = rdesc, par.set = ps,
measures = list(fpr, fnr), control = ctrl, show.info = FALSE)
res
## Tune multicrit result:
## Points on front: 3
head(as.data.frame(trafoOptPath(res$opt.path)))
## C sigma fpr.test.mean fnr.test.mean dob eol
## 1 5.828127e-04 1.117236e+03 1.0000000 0.000 1 NA
## 2 4.765984e-01 4.673627e-04 1.0000000 0.000 2 NA
## 3 5.589422e+02 1.792674e+01 1.0000000 0.000 3 NA
## 4 9.022222e+01 8.238328e-01 1.0000000 0.000 4 NA
## 5 1.019504e+01 2.220979e-03 0.1666667 0.075 5 NA
## 6 4.499292e-03 2.643150e-01 1.0000000 0.000 6 NA
## error.message exec.time
## 1 <NA> 0.031
## 2 <NA> 0.021
## 3 <NA> 0.028
## 4 <NA> 0.029
## 5 <NA> 0.030
## 6 <NA> 0.022The results can be visualized with function plotTuneMultiCritResult(). The plot shows the false positive and false negative rates for all parameter settings evaluated during tuning. Points on the Pareto front are slightly increased.
