Woohoo!

(Please advise if this is the appropiate place to have this conversation, or you'd prefer to discuss in an issue)

Hi @wlandau - I've loosely following targets's development since you first brought it to my attention, and I think I may have been the first user to request this feature #196 .

I'm curious about the configuration artefact, _targets.yaml -
Do you anticipate _targets.yaml to be versioned, or is _targets.yaml more in line with a .env file in a Ruby / docker-compose - ie- a local override?

Presently, given that _targets.yaml only configures a storage path, I'd imagine that it would work just fine as a local override, however, I'm curious as to your vision of _targets.yaml's scope/role, and whether it aligns with this.

I'm curious about the configuration artefact, _targets.yaml -
Do you anticipate _targets.yaml to be versioned, or is _targets.yaml more in line with a .env file in a Ruby / docker-compose - ie- a local override?

targets does not specifically watch _targets.yaml for changes if that is what you mean. _targets.yaml is more like a special environment file like .Rprofile or .Renviron.

Presently, given that _targets.yaml only configures a storage path, I'd imagine that it would work just fine as a local override, however, I'm curious as to your vision of _targets.yaml's scope/role, and whether it aligns with this.

I am not sure which specific features will go into _targets.yaml going forward, but I have a general plan. Some settings affect the pipeline and the external callr process that runs it, while other settings affect more interactive components like menus and warnings. The former are controlled with tar_option_set(), and the latter are controlled with environment variables like TAR_ASK and TAR_WARN. _targets.yaml will manage settings that affect both the behavior of the pipeline and the local/interactive R session simultaneously. For example, both processes need to know about the data store path because tar_read() uses it as well as tar_make(). Neither a tar_option_set() option nor an environment variable would alone be sufficient.

Thank you @wlandau , your answer clearly implies that _target.yaml is going to be more in line with a .env file, which is very helpful

thanks for developing this, & drake, I've recently reviewed targetss docs & been very excited for what I've seen on the HPC side of things, so am looking forward to transitioning into it

tar_config_set(store = "custom/path") writes store: custom/path

In my hands, there are a two issues with this:

1. if custom/path doesn't exist, the tar_config_set silently fails and does not write the_targets.yaml file.
2. if you've set an alternative store, then with_dir calls, like withr::with_dir("/some/external/targets/folder", tar_meta()) will also expect the targets to live in the alternative store. This is not the behavior I expected; I would expect that it would only alter the storage location for the targets in the active directory where the _targets.yaml file is specified. In other words, the _targets.yaml file of the folder should be the overriding setting, not the current state of the store environment variable.

1. if custom/path doesn't exist, the tar_config_set silently fails and does not write the_targets.yaml file.

That's surprising. Doesn't seem to reproduce on my end.

file.exists("_targets.yaml")
#> [1] FALSE
targets::tar_config_set(store = "custom/path")
file.exists("_targets.yaml")
#> [1] TRUE
readLines("_targets.yaml")
#> [1] "store: custom/path"

^{Created on 2021-05-21 by the reprex package (v2.0.0)}

2. if you've set an alternative store, then with_dir calls, like withr::with_dir("/some/external/targets/folder", tar_meta()) will also expect the targets to live in the alternative store. This is not the behavior I expected; I would expect that it would only alter the storage location for the targets in the active directory where the _targets.yaml file is specified. In other words, the _targets.yaml file of the folder should be the overriding setting, not the current state of the store environment variable.

Again, that's surprising, and the current behavior should align with your expectations. That's certainly what i see on my end. targets does not use an environment variable for the store, it uses the YAML file.

tmp <- tempfile()
dir.create(tmp)
store <- file.path(tmp, "store")
library(targets)
library(withr)
with_dir(tmp, tar_config_set(store = store))
with_dir(tmp, tar_script())
with_dir(tmp, tar_make())
#> • start target data
#> • built target data
#> • start target summary
#> • built target summary
#> • end pipeline
with_dir(tmp, tar_meta())
#> # A tibble: 3 x 17
#>   name  type  data  command depend    seed path  time                size  bytes
#>   <chr> <chr> <chr> <chr>   <chr>    <int> <lis> <dttm>              <chr> <int>
#> 1 summ  func… 725b… <NA>    <NA>   NA      <chr… NA                  <NA>     NA
#> 2 data  stem  593b… 4d28b3… 15669…  1.59e9 <chr… 2021-05-21 14:38:06 7f6e…   456
#> 3 summ… stem  8460… 504e74… aaa01…  7.60e8 <chr… 2021-05-21 14:38:06 24d5…   123
#> # … with 7 more variables: format <chr>, iteration <chr>, parent <lgl>,
#> #   children <list>, seconds <dbl>, warnings <lgl>, error <lgl>
tar_meta() # Should error out.
#> Error: utility functions like tar_read() and tar_progress() require a _targets/ data store produced by tar_make() or similar.

^{Created on 2021-05-21 by the reprex package (v2.0.0)}

1. You're right. It was a latency issue (I was using a remote filesystem and checking for the file too quickly).
2. Here, the issue only seems to arise when I'm inside a tar_make() call. SO, with with_dir works fine in just an interactive R session. But the problem is that for unitar, I'm using with_dir(..., tar_meta() inside a target factory, so I need to be able to run that in an external project when running a _targets.R file. And here, in my hands, it fails.

In my hands, I can do this:

#_targets.R
print("Test normal")
print(tar_meta())

print("Test with_dir")
print(withr::with_dir("external_targets", tar_meta()))

When I run tar_make() on this project, it works... the second tar_meta call using external_targets succeeds and correctly lists the targets in the external folder. I rely on this to build paths with unitar_path, which I need to call inside a _targets.R file for building targets that refer to external projects.

But if you do

tar_config_set(store="custom/path")
tar_make()

it fails for me, erroring out on the tar_meta call with withr... All I have to do is delete the _targets.yaml file and it works again.

I see what you mean now. In the general case, it would be dangerous to allow a target to modify these config settings because then it could corrupt the pipeline/run for the other targets. Many functions, including tar_make() and its siblings, deliberately lock the configuration settings on purpose. When that locking happens, config settings are read-only in a special R6 class and _targets.yaml is totally ignored while the lock is in effect for the current R session.

https://github.com/ropensci/targets/blob/main/R/utils_callr.R#L63-L64

One workaround is to call with_dir() from inside a callr::r() process. That should create a new R process that avoids the config lock of the parent R process. If you don't have too many of those targets, there should not be much overhead due to creating new processes. But if it gets to be too much, I may consider a withr-like wrapper for lockless operations. Currently on the fence about that.

Related: 1f8cfb5 fixes a bug/oversight where locked config settings were incorrectly not passed to parallel workers.