mice 3.19.0

Major changes

• Added predict_mi() to generate predictions from models fitted on
  multiply imputed datasets. The function pools predictions across
  imputations using Rubin’s rules, and can return point predictions
  or prediction intervals at a specified confidence level.

  Typical workflow:

  1. Fit a model separately on each completed dataset.
  2. Call predict_mi() with the list of models and the corresponding
     new data (per imputation).
  3. Obtain either pooled predictions (pool = TRUE) or per-imputation
     predictions (pool = FALSE).

  This functionality makes it easier to evaluate predictive performance
  on test sets while correctly accounting for imputation uncertainty.

  Contributed: @Fdvanleeuwen, @thomvolker (#720)

• Adds a correction for the Barnard-Rubin degrees of freedom calculation
  that provides stabler results for small samples and zero within-imputation
  variance. Contributed: @frederikfabriciusbjerre (#726)

Minor changes

• Adds fallback for lmer objects in pool() without requiring broom.mixed.
  Contributed: @anya-decarlo (#728)

• Explicitly load toenail data from the mice package to avoid lme4 conflict. Contributed: @bbolker (#730)

Major changes

• Fixed a long-standing issue in the internal augment() function that affected ordered factors (#713).

  Previously, augment() would:

  1. Convert ordered factors into unordered ones, and
  2. Reorder their levels alphabetically, ignoring the user-specified order.

  The old behavior could degrade imputation quality for ordinal outcomes when using the "polr" method, potentially causing model convergence issues or increased noise in imputations.

  The issue did not affect methods for unordered factors ("logreg", "polyreg", "mnar.logreg"), where level order is inconsequential.

  Thanks to @mmansolf for identifying the problem and suggesting a fix. The updated augment() now
  correctly preserves the ordered class and level order of factor variables.

• mice will now automatically move all passive variables to the end of the visitSequence for passive methods used without a user-specified visitSequence.
  This change in behavior ensures greater consistency at the end of each iteration.

  The new behavior works well for simple cases. However, for more complex situations — especially when passive variables depend on other passive variables — it is recommended to manually specify a visitSequence that updates each passive variable immediately after one of its right-hand side predictors changes. (#699)

• Adds the calltype argument to mice() for mixing predictorMatrix and formulas specifications per variable-block. The calltype argument allows the user to specify some variables (or blocks of variables) by the formulas argument, and other variables by predictorMatrix argument. (Note: This argument was called modeltype in version 3.17.1).

  calltype is a character vector of length(blocks) elements that indicates how the imputation model is specified. Entries can one of two values: "pred" or "formula". If calltype = "pred", the predictors of the imputation model for the block are specified by the corresponding row of the predictorMatrix. If calltype = "formula" the imputation model is specified by relevant entry in formulas. The default depends on the presence of the formulas argument. If formulas is present, then mice() sets calltype = "formula" for any block for which a formula is specified. Otherwise, calltype = "pred".

• Introduces an optimized matchindex C++ function to improve speed of predictive mean matching (#695)

Minor changes

• Updates security dependabot to dawidd6/action-download-artifact@v9
• Allow for negative adjusted R2 in pool.r.squared() (#700)
• Combines and updates tests for lasso.select.norm() and lasso.norm() into one file test-mice.impute.lasso.norm.R
• Combines and updates tests for lasso.select.logreg() and lasso.logreg() into one file test-mice.impute.lasso.logreg.R
• Adds support for roxygen markdown documentation

Bug fixes

• Adds a patch to resolve a problem - introduced in mice 3.17.0 - with the dfcom argument of pool(..., dfcom = .., ) (#689, #706, #707)
• Fix sequence mismatch between method and formulas (#698)

Major changes

• Imputing categorical data by predictive mean matching. Predictive mean matching (PMM) is the default method of mice() for imputing numerical variables, but it has long been possible to impute factors. This enhancement introduces better support to work with categorical variables in PMM. The former system translated factors into integers by ynum <- as.integer(f). However, the order of integers in ynum may have no sensible interpretation for an unordered factor. The new system quantifies ynum and could yield better results because of higher $R^2$. The method calculates the canonical correlation between y (as dummy matrix) and a linear combination of imputation model predictors x. The algorithm then replaces each category of y by a single number taken from the first canonical variate. After this step, the imputation model is fitted, and the predicted values from that model are extracted to function as the similarity measure for the matching step.

• The method works for both ordered and unordered factors. No special precautions are taken to ensure monotonicity between the category numbers and the quantifications, so the method should be able to preserve quadratic and other non-monotone relations of the predicted metric. It may be beneficial to remove very sparsely filled categories, for which there is a new trim argument. All you have to use the new technique is specify to mice(..., method = "pmm", ...). Both numerical and categorical variables will then be imputed by PMM.

• Potential advantages are:

  • Simpler and faster than fitting a generalised linear model, e.g., logistic regression or the proportional odds model;
  • Should be insensitive to the order of categories;
  • No need to solve problems with perfect prediction;
  • Should inherit the good statistical properties of predictive mean matching.

• Note that we still lack solid evidence for these claims. (#576). Contributed @stefvanbuuren

• New system-independent method for pooling: This version introduces a new function pool.table() that takes a tidy table of parameter estimates stemming from m repeated analyses. The input data must consist of three columns (parameter name, estimate, standard error) and a specification of the degrees of freedom of the model fitted to the complete data. The pool.table() function outputs 14 pooled statistics in a tidy form. The primary use of pool.table() is to support parameter pooling for techiques that have no tidy() or glance() methods, either within R or outside R. The pool.table() function also allows for a novel workflows that 1) break apart the traditional pool() function into a data-wrangling part and a parameters-reducing part, and 2) does not necessarily depend on classed R objects. (#574). Contributed @stefvanbuuren

• literanger: Adds support for the literanger package for rf imputation that is about twice as fast as ranger (#648). Thanks @stephematician for the contribution.

Breaking changes

• The complete(..., action = "long", ...) command puts the columns named ".imp" and ".id" in the last two positions of the long data (instead of first two positions). In this way, the columns of the imputed data will have the same positions as in the original data, which is more user-friendly and easier to work with. Note that any existing code that assumes that variables ".imp" and ".id" are in columns 1 and 2 will need to be modified. The advice is to modify the code using the variable names ".imp" and ".id". If you want the old behaviour, specify the argument order = "first". (#569). Contributed @stefvanbuuren

• Drops support for S4. Convert S4-related code to S3. Syntax as(df, "mids") is deprecated. Use as.mids(df) instead.

• Adopts the broom-convention for naming lower and upper bounds of the confidence interval as "conf.low" and "conf.high". Do not use non-syntactic names anymore, like "2.5 %".

Minor changes

• Adds support for the dots argument to ranger::ranger(...) in mice.impute.rf() (#563). Contributed @edbonneville
• Prepares for the deprecation of the blocks argument at various places
• Removes the need for blocks in initialize_chain()
• In rbind(), when formulas are concatenated and duplicate names are found, also rename the duplicated variables in formulas by their new name
• Solves problem with the package documentation link
• Simplifies NEWS.md formatting to get correct version sequence on CRAN and in-package NEWS
• Initialize single-variables blocks in make.method() in a more efficient way (resolves #672)
• Prevent as.mids() from filling the imp object for complete variables
• Defines S3 class constructors for mids, mads, mira and mipo objects

Bug fixes

• Fixes the "large logo" problem. (#574). Contributed @hanneoberman
• Patches a bug in complete() that auto-repeated imputed values into cells that should NOT be imputed (occurred as a special case of rbind(), where the first set of rows was imputed and the second was not).
• Replaces the internal variable type by the more informative pred (currently active row of predictorMatrix)
• Fixes a bug in filter.mids() that incorrectly removed empty components in the imp object
• Fixes a bug in ibind() that incorrectly used length(blocks) as the first dimension of the chainMean and chainVar objects
• Corrects the description visitSequence, chainMean and chainVar components of the mids object
• Fixes problems with zero predictors (#588)
• Fixes a problem with the minpuc argument in quickpred() (#634)
• Fixes coef() not available on S4 object when using with lavaan (#615, #616)
• Adds .github/dependabot.yml configuration to automate daily check (#598)
• Update documentation tags to roxygen2 7.3.1 requirements
• Repairs lost braces in the documentation
• Fixes an installation problem when Rprofile prints to stdout on Fedora, R version 4.1.3 (#646, #647). Thanks @brookslogan for the fix.
• Fixes a bug during initialization of factor values
• Removes methods and rlang from Depends
• Removes export of non-user facing ampute() helpers
• Clears \link statements that do not pass CRAN checks

Major changes

• Expands futuremice() functionality by allowing for external packages and user-written functions (#550). Contributed @thomvolker

• Adds GH issue templates bug_report, feature_request and help_wanted (#560). Contributed @hanneoberman

Minor changes

• Removes documentation files for rbind.mids() and cbind.mids() to conform to CRAN policy
• Adds mitml and glmnet to imports so that test code conforms to _R_CHECK_DEPENDS_ONLY=true flag in R CMD check
• Initializes random number generator in futuremice() if there is no .Random.seed yet.
• Updates GitHub actions for package checking and site building
• Preserves user settings in predictorMatrix for case F by adding a predictorMatrix argument to make.predictorMatrix()
• Polishes mice.impute.mpmm() example code

Bug fixes

• Adds proper support for factors to mice.impute.2lonly.pmm() (#555)
• Solves function naming problems for S3 generic functions tidy(), update(), format() and sum()
• Out-comments and weeds example&test code to silence R CMD check with _R_CHECK_DEPENDS_ONLY=true
• Fixes small bug in futuremice() that throws an error when the number of cores is not specified, but the number of available cores is greater than the number of imputations.
• Solves a bug in mice.impute.mpmm() that changed the column order of the data

mice 3.15.0

Major changes

• Adds a function futuremice() with support for parallel imputation using the future package (#504). Contributed @thomvolker, @gerkovink

• Adds multivariate predictive mean matching mice.impute.mpmm(). (#460). Contributed @Mingyang-Cai

• Adds convergence() for convergence evaluation (#484). Contributed @hanneoberman

• Reverts the internal seed behaviour back to mice 3.13.10 (#515). #432 introduced new local seed in response to #426. However, various issues arose with this facility (#459, #492, #502, #505). This version restores the old behaviour using global .Random.seed. Contributed @gerkovink

• Adds a custom.t argument to pool() that allows the advanced user to specify a custom rule for calculating the total variance $T$. Contributed @gerkovink

• Adds new argument exclude to mice.impute.pmm() that excludes a user-specified vector of values from matching. Excluded values will not appear in the imputations. Since the observed values are not imputed, the user-specified values are still being used to fit the imputation model (#392, #519). Contributed @gerkovink

Minor changes

• Styles all .R and .Rmd files
• Makes post-processing assignment consistent with lines 85/86 in sampler.R (#511)
• Edit test broken on R<4 (#501). Contributed @MichaelChirico
• Adds support for models reporting contrasts rather than terms (#498). Contributed @LukasWallrich
• Applies edits to autocorrelation function (#491). Contributed @hanneoberman
• Changes p-value calculation to more robust alternative (#494). Contributed @AndrewLawrence
• Uses inherits() to check on class membership
• Adds decprecation notices to parlmice()
• Adapt prop, patterns and weights matrices for pattern with only 1's
• Adds warning when patterns cannot be generated (#449, #317, #451)
• Adds warning on the order of model terms in D1() and D2() (#420)
• Adds example code to fit model on train data and apply to test data to mice()
• Adds example code on synthetic data generation and analysis in make.where()
• Adds testfile test-mice.impute.rf.R(#448)

Bug fixes

• Replaces .Random.seed reads from the .GlobalEnv by get(".Random.seed", envir = globalenv(), mode = "integer", inherits = FALSE)
• Repairs capitalisation problems with lastSeedValue variable name
• Solves x$lastSeedValue problem in cbind.mids() (#502)
• Fixes problems with ampute()
• Preserves stochastic nature of mice() by smarter random seed initialisation (#459)
• Repairs a drop = FALSE buglet in mice.impute.rf() (#447, #448)
• @str-amg reported that the new dependency on withr package should have version 2.4.0 (published in January 2021) or higher. Versions withr 2.3.0 and before may give Error: object 'local_seed' is not exported by 'namespace:withr'. Either update manually, or install the patched version mice 3.14.1 from GitHub. (#445). NOTE: withr is no longer needed in mice 3.15.0

Major changes

• Adds four new univariate functions using the lasso for automatic variable selection:

Contributed by @EdoardoCostantini (#438).

• Adds Jamshidian && Jalal's non-parametric MCAR test, mice::MCAR() and associated plot method. Contributed by @cjvanlissa (#423).

• Adds two new functions pool.syn() and pool.scalar.syn() that specialise pooling estimates from synthetic data. The "reiter2003" pooling rule assumes that synthetic data were created from complete data. Thanks Thom Volker (#436).

• Avoids changing the global .Random.seed (#426, #432) by implementing withr::local_preserve_seed() and withr::local_seed(). This change provides stabler behavior in complex scripts. The change does not appear to break reproducibility when mice() was run with a seed. Nevertheless, if you run into a reproducibility problem, install mice 3.13.12 or before.

• Improves the imputation of parabolic data in mice.impute.quadratic(), adds a parameter quad.outcome containing the name of the outcome variable in the complete-data model. Contributed @Mingyang-Cai, @gerkovink (#408)

• By default, mice.impute.rf() now uses the faster ranger package as back-end instead of randomForest package. If you want the old behaviour specify the rfPackage = "randomForest" argument to the mice(...) call. Contributed @prockenschaub (#431).

• Generalises pool() so that it processes the parameters from all gamlss sub-models. Thanks Marcio Augusto Diniz (#406, #405)

• Uses the robust standard error estimate for pooling when pool() can extract robust.se from the object returned by broom::tidy() (#310)

Bug fixes

• Contains an emergency solution as install.on.demand() broke the standard CRAN workflow. mice 3.14.0 does not call install.on.demand() anymore for recommended packages. Also, install.on.demand() will not run anymore in non-interactive mode.
• Repairs an error in the mice:::barnard.rubin() function for infinite dfcom. Thanks @huftis (#441).
• Solves problem with Xi <- as.matrix(...) in mice.impute.2l.lmer() that occurred when a cluster contains only one observation (#384)
• Edits the predictorMatrix to a monotone pattern if visitSequence = "monotone" and maxit = 1 (#316)
• Solves a problem with the plot produced by md.pattern() (#318, #323)
• Fixes the intercept in make.formulas() (#305, #324)
• Fixes seed when using newdata in mice.mids() (#313, #325)
• Solves a problem with row names of the where element created in rbind() (#319)
• Solves a bug in mnar imputation routine. Contributed by Margarita Moreno Betancur.

Minor changes

• Replaces URL to jstatsoft with DOI
• Update reference to literature (#442)
• Informs the user that pool() cannot take a mids object (#433)
• Updates documentation for post-processing functionality (#387)
• Adds Rcpp necessities
• Solves a problem with "last resort" initialisation of factors (#410)
• Documents the "flat-line behaviour" of mice.impute.2l.lmer() to indicate a problem in fitting the imputation model (#385)
• Add reprex to test (#326)
• Documents that multivariate imputation methods do not support the post parameter (#326)

Major changes

• Updated mids2spss() replaces the foreign by haven package. Contributed Gerko Vink (#291)

Minor changes

• Repairs an error in tests\testhat\test-D1.R that failed on mitml 0.4-0
• Reverts with.mids() function to old version because the change in commit 4634094 broke downstream package metafor (#292)
• Solves a glitch in mice.impute.rf() in finding candidate donors (#288, #289)

Much faster predictive mean matching

• The new matchindex C function makes predictive mean matching 50 to 600 times faster.
  The speed of pmm is now on par with normal imputation (mice.impute.norm())
  and with the miceFast package, without compromising on the statistical quality of
  the imputations. Thanks to Polkas Polkas/miceFast#10 and
  suggestions by Alexander Robitzsch. See #236 for more details.

New ignore argument to mice

• New ignore argument to mice(). This argument is a logical vector
  of nrow(data) elements indicating which rows are ignored when creating
  the imputation model. We may use the ignore argument to split the data
  into a training set (on which the imputation model is built) and a test
  set (that does not influence the imputation model estimates). The argument
  is based on the suggestion in
  #32 (comment). See #32 for
  more background and techniques. Crafted by Patrick Rockenschaub

New filter() function for mids objects

• New filter() method that subsets a mids object (multiply-imputed data set).
  The method accepts a logical vector of length nrow(data), or an expression
  to construct such a vector from the incomplete data. (#269).
  Crafted by Patrick Rockenschaub.

Changes affecting reproducibility

• Breaking change: The matcher algorithm in pmm has changed to matchindex
  for speed improvements. If you want the old behavior, specify mice(..., use.matcher = TRUE).

Minor changes

• Corrected installation problem related to cpp11 package (#286)
• Simplifies with.mids() by calling eval_tidy() on a quosure. Does not yet solve #265.
• Improve documentation for pool() and pool.scalar() (#142, #106, #190 and others)
• Makes tidy.mipo more flexible (#276)
• Solves a problem if nelsonaalen() gets a tibble (#272)
• Add explanation to how NAs can appear in the imputed data (#267)
• Add warning to quickpred() documentation (#268)
• Styles all sources files with styler
• Improves consistency in code and documentation
• Moves internally defined functions to global namespace
• Solves bug in internal sum.scores()
• Adds deprecated messages to lm.mids(), glm.mids(), pool.compare()
• Removes expandcov()
• Strips out all return() calls placed just before end-of-function
• Remove all trailing spaces
• Repairs a bug in the routine for finding the printFlag value (#258)
• Update URL's after transfer to organisation amices

Major changes

• The Cox model does not return df.residual, which caused problematic behavior in the D1(), D2(), D3(), anova() and pool(). mice now extracts the relevant information from other parts of the objects returned by survival::coxph(), which solves long-standing issues with the integration of the Cox model (#246).
• Adds missing Rccp dependency to work with tidyr 1.1.1 (#248).

Minor changes

• Addresses warnings: Non-file package-anchored link(s) in documentation object.
• Updates on ampute documentation (#251).
• Ask user permission before installing a package from suggests.

Major changes

• New functions tidy.mipo() and glance.mipo() return standardized output that conforms to broom specifications. Kindly contributed by Vincent Arel Bundock (#240).

Minor changes

• Solves a problem with the D3 testing script that produced an error on CRAN (#244).