How to get targets to watch a local package development and update pipeline if changes made [help] #1200

njtierney · 2023-12-13T23:20:17Z

njtierney
Dec 13, 2023

Help

I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi there,

We are developing code in a package locally that gets run in a targets pipeline. Sometimes we change how a function works in this package, and we would like the targets pipeline to rerun if that contains a function that is related. I guess similar behaviour to how things change if you change a function in the R directory in a targets pipeline.

So my question is, is there a way to get targets to watch a specific directory of an R package, to see if there are changes there and update the pipeline if necessary?

My initial thoughts were something like:

tar_target(<rpkg>, "path/to/rpkg", format = "file")

Or something like that, but I'm not sure if that's going to be too strict?

Answered by joelnitta

Dec 14, 2023

@njtierney I think you are taking the example in the manual too literally. There is no need to call the function without arguments. It is just being shown to indicate that you are using the function from the package that you want to track.

Please consider the first reprex I showed:

library(targets)
tar_option_set(
  packages = c("tibble", "dplyr", "tidyr"),
  imports = c("tibble", "tidyr")
)
list(
  tar_target(
    name = data,
    command = tibble(x = rnorm(100), y = rnorm(100))
  ),
  tar_target(
    name = data_processed,
    command = select(data, x)
  ),
  tar_target(
    name = data_no_na,
    command = drop_na(data)
  )
)

In this case, if you were to install a new version of {tibb…

View full answer

joelnitta · 2023-12-13T23:56:23Z

joelnitta
Dec 13, 2023

Have you read Package-based invalidation in the targets handbook? I think that has a solution for pretty much what you describe:

# _targets.R
library(targets)
tar_option_set(
  packages = c("package1", "package2", ...), # `...` is for other packages.
  imports = c("package1", "package2")
)
list(
  tar_target(name = output, command = function_from_package1())
)

5 replies

njtierney Dec 14, 2023
Author

That looks pretty great to me! We will just check that functionality out before I mark this as an answer. Thanks so much, @joelnitta :)

smwindecker Dec 14, 2023

Hi @joelnitta, @njtierney and I tried this approach in our repo, and we could not get it to work. Could be we were too literal in our replication of the code in the manual...

We found that it was trying to execute the code in function_from_package1(), and so errored where it was looking for required arguments that weren't specified. You can see this if you run tar_make() on the targets below. The error it spits out "use of NULL environment is defunct" is the same error as if you tried to run select() with no arguments.

# Load packages required to define the pipeline:
library(targets)

# Set target options:
tar_option_set(
  packages = c("tibble", 'dplyr'),
  imports = c('tibble', 'dplyr') # packages that your targets need to run
)

list(
  tar_target(
    name = output,
    command = select()
  ),
  tar_target(
    name = data,
    command = tibble(x = rnorm(100), y = rnorm(100))
    # format = "feather" # efficient storage for large data frames
  ),
  tar_target(
    name = data_processed,
    command = select(data, x)
  )
)

joelnitta Dec 14, 2023

@smwindecker I think you've found a bug that appears to be triggered when "dplyr" is included in imports.

For example, compare these two reprexes (the only difference is imports: c("tibble", "tidyr") in the first, and c("tibble", "dplyr") in the second.

This works:

targets::tar_dir({
  targets::tar_script({
    # Load packages required to define the pipeline:
    library(targets)

    # Set target options:
    tar_option_set(
      packages = c("tibble", "dplyr", "tidyr"),
      imports = c("tibble", "tidyr")
    )

    list(
      tar_target(
        name = data,
        command = tibble(x = rnorm(100), y = rnorm(100))
      ),
      tar_target(
        name = data_processed,
        command = select(data, x)
      ),
      tar_target(
        name = data_no_na,
        command = drop_na(data)
      )
    )
  })
  targets::tar_make()
})
#> ▶ dispatched target data
#> ● completed target data [0.026 seconds]
#> ▶ dispatched target data_no_na
#> ● completed target data_no_na [0.001 seconds]
#> ▶ dispatched target data_processed
#> ● completed target data_processed [0.011 seconds]
#> ▶ completed pipeline [1.427 seconds]

^{Created on 2023-12-14 with reprex v2.0.2}

But this doesn't:

targets::tar_dir({
  targets::tar_script({
    # Load packages required to define the pipeline:
    library(targets)

    # Set target options:
    tar_option_set(
      packages = c("tibble", "dplyr", "tidyr"),
      imports = c("tibble", "dplyr")
    )

    list(
      tar_target(
        name = data,
        command = tibble(x = rnorm(100), y = rnorm(100))
      ),
      tar_target(
        name = data_processed,
        command = select(data, x)
      ),
      tar_target(
        name = data_no_na,
        command = drop_na(data)
      )
    )
  })
  targets::tar_make()
})
#> Error:
#> ! Error running targets::tar_make()
#> Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
#> Debugging guide: https://books.ropensci.org/targets/debugging.html
#> How to ask for help: https://books.ropensci.org/targets/help.html
#> Last error message:
#>     use of NULL environment is defunct
#> Last error traceback:
#>     tryCatch(withCallingHandlers({ NULL saveRDS(do.call(do.call, c(readRDS("...
#>     tryCatchList(expr, classes, parentenv, handlers)
#>     tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
#>     doTryCatch(return(expr), name, parentenv, handler)
#>     tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
#>     tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>     doTryCatch(return(expr), name, parentenv, handler)
#>     withCallingHandlers({ NULL saveRDS(do.call(do.call, c(readRDS("/var/fold...
#>     saveRDS(do.call(do.call, c(readRDS("/var/folders/gp/h66562zx3ps986y5z2jz...
#>     do.call(do.call, c(readRDS("/var/folders/gp/h66562zx3ps986y5z2jzrc0h0000...
#>     (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
#>     (function (targets_function, targets_arguments, options, envir = NULL, s...
#>     tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
#>     tryCatchList(expr, classes, parentenv, handlers)
#>     tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>     doTryCatch(return(expr), name, parentenv, handler)
#>     withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
#>     targets::tar_callr_inner_try(targets_function = targets_function, target...
#>     do.call(targets_function, targets_arguments)
#>     (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
#>     local_init(pipeline = pipeline, meta = meta_init(path_store = path_store...
#>     self$start()
#>     self$ensure_meta()
#>     self$meta$record_imports(self$pipeline$imports, self$pipeline)
#>     self$data_imports(envir, pipeline)
#>     hash_imports(envir)
#>     hash_imports_graph(envir, graph_envir(envir))
#>     topo_sort_igraph(graph)
#>     igraph::topo_sort(igraph)
#>     ensure_igraph(graph)
#>     graph_envir(envir)
#>     graph_edges(edges_envir(envir))
#>     igraph::graph_from_data_frame(edges)
#>     as.data.frame(d)
#>     edges_envir(envir)
#>     lapply(names, envir_deps, envir = envir, names = names)
#>     FUN(X[[i]], ...)
#>     if_any(is.function(value), deps_function(value), character(0))
#>     deps_function(value)
#>     `body<-`(`*tmp*`, value = eval(call("substitute", body(fun), env), envir...
#>     as.function(c(as.list(formals(fun)), list(value)), envir)
#>     as.function.default(c(as.list(formals(fun)), list(value)), envir)
#> Backtrace:
#>     ▆
#>  1. └─targets::tar_make()
#>  2.   └─targets:::callr_outer(...)
#>  3.     ├─targets:::if_any(...)
#>  4.     └─targets:::callr_error(traced_condition = out, fun = fun)
#>  5.       └─targets::tar_throw_run(message, class = class(traced_condition$condition))
#>  6.         └─targets::tar_error(...)
#>  7.           └─rlang::abort(message = message, class = class, call = tar_empty_envir)

^{Created on 2023-12-14 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.2 (2023-10-31)
#>  os       macOS Sonoma 14.1.2
#>  system   x86_64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Asia/Tokyo
#>  date     2023-12-14
#>  pandoc   2.19.2 @ /usr/local/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  backports     1.4.1   2021-12-13 [1] CRAN (R 4.3.0)
#>  base64url     1.4     2018-05-14 [1] CRAN (R 4.3.0)
#>  callr         3.7.3   2022-11-02 [1] CRAN (R 4.3.0)
#>  cli           3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
#>  codetools     0.2-19  2023-02-01 [1] CRAN (R 4.3.2)
#>  data.table    1.14.8  2023-02-17 [1] CRAN (R 4.3.0)
#>  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.0)
#>  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.0)
#>  fansi         1.0.5   2023-10-08 [1] CRAN (R 4.3.0)
#>  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs            1.6.3   2023-07-20 [1] CRAN (R 4.3.0)
#>  glue          1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.0)
#>  igraph        1.5.0   2023-06-16 [1] CRAN (R 4.3.0)
#>  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.0)
#>  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.0)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.0)
#>  processx      3.8.2   2023-06-30 [1] CRAN (R 4.3.0)
#>  ps            1.7.5   2023-04-18 [1] CRAN (R 4.3.0)
#>  purrr         1.0.2   2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo          1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils       2.12.2  2022-11-11 [1] CRAN (R 4.3.0)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  reprex        2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang         1.1.2   2023-11-04 [1] CRAN (R 4.3.0)
#>  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler        1.9.1   2023-03-04 [1] CRAN (R 4.3.0)
#>  targets       1.4.0   2023-12-11 [1] CRAN (R 4.3.0)
#>  tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.0)
#>  utf8          1.2.4   2023-10-22 [1] CRAN (R 4.3.0)
#>  vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.0)
#>  withr         2.5.2   2023-10-30 [1] CRAN (R 4.3.0)
#>  xfun          0.41    2023-11-01 [1] CRAN (R 4.3.0)
#>  yaml          2.3.7   2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

@wlandau should this be filed as an issue?

wlandau Dec 14, 2023
Maintainer

Turns out this is an extremely rare edge case. The error comes from trying to inspect dplyr:::.standalone_types_check_dot_call for dependencies, which is a custom primitive function and thus has no closure (very odd to find in a non-base package). 3fd04b9 fixes that immediate error on my end, but the example still doesn't work because the dependency graph has cycles. Also, dplyr defines a ton of S3 methods, but targets misses all of them because static code analysis can only detect generics. As I think you know, imports is really only for simple local packages, and even then you might hit other static code analysis issues beyond our control.

Also, thank you @joelnitta for helping me respond to GitHub discussions!

njtierney Dec 14, 2023
Author

Hi everyone,

Thanks @joelnitta and @wlandau your help! And again @wlandau you are probably the most prompt maintainer I know of.

How do you anticipate users to use the target local package functions in their workflow?

For example below, we import function_from_package1(), into output.

Are we supposed to call it as output() or as function_from_package1().

We don't have an example we can share on hand as we have a private repository, but we can whip something up after an upcoming deadline.

The issue we were running into was that we were getting errors where function_from_package1() was erroring because it was called without arguments, although @smwindecker might be able to provide a bit more context around this.

# _targets.R
library(targets)
tar_option_set(
  packages = c("package1", "package2", ...), # `...` is for other packages.
  imports = c("package1", "package2")
)
list(
  tar_target(name = output, command = function_from_package1())
)

joelnitta · 2023-12-14T23:51:21Z

joelnitta
Dec 14, 2023

@njtierney I think you are taking the example in the manual too literally. There is no need to call the function without arguments. It is just being shown to indicate that you are using the function from the package that you want to track.

Please consider the first reprex I showed:

library(targets)
tar_option_set(
  packages = c("tibble", "dplyr", "tidyr"),
  imports = c("tibble", "tidyr")
)
list(
  tar_target(
    name = data,
    command = tibble(x = rnorm(100), y = rnorm(100))
  ),
  tar_target(
    name = data_processed,
    command = select(data, x)
  ),
  tar_target(
    name = data_no_na,
    command = drop_na(data)
  )
)

In this case, if you were to install a new version of {tibble} or {tidyr} that had changes to functions used in your pipeline (here, tibble() and drop_na()), any targets built using those functions (here, data and data_no_na) would be re-run.

2 replies

njtierney Dec 15, 2023
Author

Ah perfect, thank you @joelnitta ! Saras can't seem to answer this thread or comment for some reason, but we pass on our thanks!

wlandau Jan 3, 2024
Maintainer

Saras can't seem to answer this thread or comment for some reason

Sorry about the locked thread. I enable temporary interaction limits so I can truly unplug when I go on vacation. Unfortunately I don't know of a way to make this more obvious to users, other than #1007. I posted https://github.com/orgs/community/discussions/84052 to request a more obvious way to let users know why interactions may be limited.

How to get targets to watch a local package development and update pipeline if changes made [help] #1200

Uh oh!

njtierney Dec 13, 2023

Help

Description

Replies: 2 comments · 7 replies

Uh oh!

joelnitta Dec 13, 2023

Uh oh!

njtierney Dec 14, 2023 Author

Uh oh!

smwindecker Dec 14, 2023

Uh oh!

Uh oh!

joelnitta Dec 14, 2023

Uh oh!

wlandau Dec 14, 2023 Maintainer

Uh oh!

njtierney Dec 14, 2023 Author

Uh oh!

joelnitta Dec 14, 2023

Uh oh!

njtierney Dec 15, 2023 Author

Uh oh!

wlandau Jan 3, 2024 Maintainer

njtierney
Dec 13, 2023

Replies: 2 comments 7 replies

joelnitta
Dec 13, 2023

njtierney Dec 14, 2023
Author

wlandau Dec 14, 2023
Maintainer

njtierney Dec 14, 2023
Author

joelnitta
Dec 14, 2023

njtierney Dec 15, 2023
Author

wlandau Jan 3, 2024
Maintainer