Memory error. _targets/meta/meta grows too big #186
Description
Prework
- Read and agree to the code of conduct and contributing guidelines.
- If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
- Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
- Runnable: post enough R code and data so any onlooker can create the error on their own computer.
- Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
- Readable: format your code according to the tidyverse style guide.
Description
Sometimes, when working in a project. Suddenly I start receiving the error you can see below. Poking around, I can see _targets/meta/meta has grown to > 1GB.
Specifically, the error column of the meta DF seems to contain the problematic content.
Reproducible example
I am attaching here a reduced version of the project where it happens: memory_error.zip
If you delete the _targets folder (or tar_destroy() it), the project runs again.
- Post a minimal reproducible example so the maintainer can troubleshoot the problems you identify. A reproducible example is:
- Runnable: post enough R code and data so any onlooker can create the error on their own computer.
- Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
- Readable: format your code according to the tidyverse style guide.
Expected result
meta should not grow so much as to cause an error. Alternatively, if meta grows that much, a specific warning about it would be nice.
Diagnostic information
targets::tar_make()
Error in data.table::fwrite(x = data, file = tmp, sep = database_sep_outer, :
Unable to allocate -138142212 MB * 8 thread buffers; '12: Cannot allocate memory'. Please read ?fwrite for nThread, buffMB and verbose options.
Error: callr subprocess failed: Unable to allocate -138142212 MB * 8 thread buffers; '12: Cannot allocate memory'. Please read ?fwrite for nThread, buffMB and verbose options.
Type .Last.error.trace to see where the error occured
> .Last.error.trace
Stack trace:
Process 25034:
1. targets::tar_make()
2. targets:::callr_outer(targets_function = tar_make_inner, targets_arguments = targets_arguments, ...
3. targets:::trn(is.null(callr_function), callr_inner(target_script_path(), ...
4. base:::do.call(callr_function, prepare_callr_arguments(callr_function, ...
5. (function (func, args = list(), libpath = .libPaths(), repos = default_repos(), ...
6. callr:::get_result(output = out, options)
7. throw(newerr, parent = remerr[[2]])
x callr subprocess failed: Unable to allocate -138142212 MB * 8 thread buffers; '12: Cannot allocate memory'. Please read ?fwrite for nThread, buffMB and verbose options.
Process 25091:
19. (function (targets_script, targets_function, targets_arguments) ...
20. base:::source(targets_script)
21. base:::withVisible(eval(ei, envir))
22. base:::eval(ei, envir)
23. base:::eval(ei, envir)
24. targets::tar_invalidate(matches("TESTS"))
25. meta$database$overwrite_storage(data)
26. data.table::fwrite(x = data, file = tmp, sep = database_sep_outer, ...
27. base:::.handleSimpleError(function (e) ...
28. h(simpleError(msg, call))
x Unable to allocate -138142212 MB * 8 thread buffers; '12: Cannot allocate memory'. Please read ?fwrite for nThread, buffMB and verbose options.