[help] mirai dispatcher timeout using targets with crew #1112

alexpiper · 2023-08-18T06:12:26Z

alexpiper
Aug 18, 2023

Help

I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi, im experiencing an issue when attempting to run a targets pipeline with both a local crew controller and renv activated, i get a "no targets found" error. The pipeline runs fine when the crew controller is omitted and renv activated, or when the crew controller is active and renv is deactivated. It just does not work together (see reprex below).

crew + renv (fails)

# _targets.R file
library(targets)  
library(crew)  
tar_option_set(controller = crew_controller_local())
list(
      tar_target(data, data.frame(x = sample.int(1e3))),
      tar_target(summary, mean(data$x, na.rm = TRUE))
)

> tar_make()
✖ no targets found [7.406 seconds]
Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: connection to local process timed out after 5s

renv with no crew (works)

# _targets.R file
library(targets)
library(crew)
list(
      tar_target(data, data.frame(x = sample.int(1e3))),
      tar_target(summary, mean(data$x, na.rm = TRUE))
)

> tar_make()
▶ start target data
● built target data [0 seconds]
▶ start target summary
● built target summary [0 seconds]
▶ end pipeline [0.125 seconds]

crew with no renv (works)

# _targets.R file
library(targets)  
library(crew)  
tar_option_set(controller = crew_controller_local())
list(
      tar_target(data, data.frame(x = sample.int(1e3))),
      tar_target(summary, mean(data$x, na.rm = TRUE))
)

> renv::deactivate()
Restarting R session...
> library(targets)
> library(crew)
> tar_make()
▶ start target data
● built target data [0.016 seconds]
▶ start target summary
● built target summary [0 seconds]
▶ end pipeline [11.937 seconds]

Session info

> sessionInfo()
R version 4.2.0 (2022-04-22 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_Australia.utf8  LC_CTYPE=English_Australia.utf8    LC_MONETARY=English_Australia.utf8
[4] LC_NUMERIC=C                       LC_TIME=English_Australia.utf8    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] crew_0.4.0    targets_1.2.2

loaded via a namespace (and not attached):
 [1] igraph_1.4.1      rstudioapi_0.14   knitr_1.42        magrittr_2.0.3    nanonext_0.9.2   
 [6] tidyselect_1.2.0  R6_2.5.1          rlang_1.1.1       fansi_1.0.4       tools_4.2.0      
[11] data.table_1.14.8 xfun_0.37         sessioninfo_1.2.2 utf8_1.2.3        cli_3.6.1        
[16] yaml_2.3.7        base64url_1.4     digest_0.6.31     tibble_3.2.1      lifecycle_1.0.3  
[21] processx_3.8.2    mirai_0.9.1       callr_3.7.3       vctrs_0.6.3       ps_1.7.5         
[26] codetools_0.2-19  glue_1.6.2        getip_0.1-3       compiler_4.2.0    pillar_1.9.0     
[31] backports_1.4.1   renv_0.17.3-88    pkgconfig_2.0.3

Any help with this would be greatly appreciated.

wlandau · 2023-08-18T13:07:56Z

wlandau
Aug 18, 2023
Maintainer

I tried your example on Windows and renv with the versions of package you have, and I could not reproduce the error.

There is another error that reads:

connection to local process timed out after 5s

which seems related to #1101 and wlandau/crew#76. If you upgrade to development mirai with the following, it may solve the issue:

install.packages(c("nanonext", "mirai"), repos = "https://shikokuchuo.r-universe.dev")

If not, then it could be something renv-related or callr-related, as was the case with #506 and rstudio/renv#773. You could try the pipeline without callr using tar_make(callr_function = NULL). In general, it is helpful to see how much you can isolate the issue by peeling back more layers. If you can reproduce the error without targets, it will be a lot simpler to understand. If you try to run the example from https://wlandau.github.io/crew/ in your renv project and it hits the same error, then you will have narrowed it down.

0 replies

alexpiper · 2023-08-21T03:03:44Z

alexpiper
Aug 21, 2023
Author

Thanks for your help with this Will,

Installing the development versions of mirai and nanonext didnt help, neither did running it without callr, but i think ive narrowed it down to the daemons function in mirai, where the dispatcher times out when loading larger renv lockfiles (i.e., fails when lockfile contains 260 packages), but runs successfully with a smaller renv lockfile (110 packages).

Interestingly daemons runs successfully when dispatcher = FALSE, so it seems to be related to the dispatcher in particular. The time limit for launching the dispatcher seems to be hardcoded as 5000ms in mirai:::.timelimit, so perhaps if this timelimit can be edited it may fix the issue. I will open an issue over the mirai github.

See further examples below of the issue occuring with crew and mirai

Crew

controller <- crew_controller_local(
  name = "example",
  workers = 2,
  seconds_idle = 10,
  
)
controller$start()
Error in launch_and_sync_daemon(sock = sock, urld, parse_dots(...), url,  : 
 sync with dispatcher or daemon failed after 5s

Mirai

# Running daemons with dispatcher (Fails)
daemons(1, dispatcher = TRUE)
Error in launch_and_sync_daemon(sock = sock, urld, dots, n, urlc) : 
  sync with dispatcher or daemon failed after 5s

# Running daemons without dispatcher (works)
daemons(1, dispatcher = FALSE)
[1] 1

19 replies

wlandau Aug 23, 2023
Maintainer

Yup, works great! I removed those envvar hacks from crew, and the dispatcher starts in about 0.6s even when the renv library takes around 8 seconds to load on the host.

shikokuchuo Aug 23, 2023

FYI in r-lib/mirai@97ebf51 I find the libpath more robustly through the nanonext library which is loaded even if mirai is not attached. This is not as efficient vs. if mirai is already attached but consistent in the case it is not. Trying to find the path of an unattached package is just slow... see the hoops we have to jump through...

shikokuchuo Aug 23, 2023

And this should be the final iteration: 6e5175b. I have essentially re-implemented find.package() in a one-liner that takes 3 microseconds to run vs. 80.

wlandau Aug 24, 2023
Maintainer

Thanks, @shikokuchuo! Still works on my end (and quickly).

alexpiper Aug 24, 2023
Author

Piping in here to thank you both for the quick response to this issue, the issue is resolved with the latest development version of mirai.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[help] mirai dispatcher timeout using targets with crew #1112

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 19 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[help] mirai dispatcher timeout using targets with crew #1112

Uh oh!

alexpiper Aug 18, 2023

Help

Description

Replies: 2 comments · 19 replies

Uh oh!

wlandau Aug 18, 2023 Maintainer

Uh oh!

alexpiper Aug 21, 2023 Author

Uh oh!

wlandau Aug 23, 2023 Maintainer

Uh oh!

shikokuchuo Aug 23, 2023

Uh oh!

Uh oh!

shikokuchuo Aug 23, 2023

Uh oh!

wlandau Aug 24, 2023 Maintainer

Uh oh!

alexpiper Aug 24, 2023 Author

alexpiper
Aug 18, 2023

Replies: 2 comments 19 replies

wlandau
Aug 18, 2023
Maintainer

alexpiper
Aug 21, 2023
Author

wlandau Aug 23, 2023
Maintainer

wlandau Aug 24, 2023
Maintainer

alexpiper Aug 24, 2023
Author