Prework

• Read and agree to the code of conduct and contributing guidelines.
• Confirm that your issue is most likely a genuine bug in the targets package itself and not a user error, a known limitation, or a bug in another package that targets depends on. For miscellaneous troubleshooting, please post to the discussions instead.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
  • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • Readable: format your code according to the tidyverse style guide.

Description

On my company's SGE cluster, tar_config_set() does not work because it calls file.rename() to transfer a file between different physical drives.

tar_config_set(store = "_targets_1.0.1")
#> Warning message:
#> In file.rename(from = tmp, to = self$path) :
#>   cannot rename file '/tmp/RtmpZDh8JW/file18894eba0de' to '_targets.yaml', reason 'Invalid cross-device link'

I can isolate this error using a temp file.

tmp <- tempfile()
file.rename(tmp, "test.out")
#> [1] FALSE
#> Warning message:
#> In file.rename(tmp, "test.out") :
#>   cannot rename file '/tmp/RtmpZDh8JW/file188977bcad56' to 'test.out', reason 'Invalid cross-device link'

To do

Stage temporary files at the project root instead of at a tempfile(). Applies to:

1. In tar_option_set(), just write directly to _targets.yaml. Only tar_option_set() actually writes anything there, so there should not be a race condition.
2. store_produce_stage.tar_aws(): store staged files inside scratch/ in the local data store.
3. store_produce_stage.tar_aws_file(): store a target's staged file at scratch/<target_name> in the local data store for downloads. Upload staging should stay as is.