Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm embarking on my first heavy use of dynamic branching and want to make sure I'm not committing to an incorrect or unwise path before I get too deep. I've read through the book and discussions and there's been some useful related points but I haven't seen anything that exactly answers my question. (Apologies if I've missed something.)

Basically, if I have a chain of dynamic branches, can I guarantee when I combine some later on in the pipeline that the "right" branches will always be mapped together?

Consider this simplified example:

library(targets)
library(dplyr)

f_a_b <- function(a) {
  a |> rename(x_b = x_a)
}
f_b_c <- function(b) {
  b |> rename(x_c = x_b)
}
f_b_d <- function(b) {
  b |> rename(x_d = x_b)
}
f_c_e <- function(c) {
  c |> rename(x_e = x_c)
}
f_a_d_e <- function(a, d, e) {
  a |> cross_join(d) |> cross_join(e)
}

list(
  tar_target(
    name = aa,
    tibble(x_a = 1:5),
    iteration = "vector"
  ),

  tar_target(
    name = bb,
    f_a_b(aa),
    pattern = map(aa)
  ),
  tar_target(
    name = cc,
    f_b_c(bb),
    pattern = map(bb)
  ),
  tar_target(
    name = dd,
    f_b_d(bb),
    pattern = map(bb)
  ),
  tar_target(
    name = ee,
    f_c_e(cc),
    pattern = map(cc)
  ),
  tar_target(
    name = zz,
    f_a_d_e(aa, dd, ee),
    pattern = map(aa, dd, ee)
  )
)

tar_make()
tar_load(zz)
zz
#     x_a   x_d   x_e
#   <int> <int> <int>
# 1     1     1     1
# 2     2     2     2
# 3     3     3     3
# 4     4     4     4
# 5     5     5     5

Can I guarantee that the branches of aa, dd and ee will always be joined in the right "order", ensuring that every row will always have a single value, or is it possible that they could get mismatched? I understand that it's not based on the actual ordering but on the hashes - if the hashes don't change then the same branches should be matched up, but if multiple branches change their hash could that change how they get matched?
I've played around a bit with this toy example - inserting, deleting, modifying and rearranging values in aa - and everything has worked exactly as I would expect. But of course there is often a difference in how things can work between toy examples and real world behaviour at scale. It's not clear to me whether the implementation guarantees that the "order" will be respected, or whether it's just luck at the small scale and everything could blow up when we're looking at complications of 1-2k branches, varying processing lengths and parallel processing with crew.

For a bit more context, one example of this pattern is that aa defines custom aggregations of detailed industry breakdowns for a data set, bb is the calculation of the aggregations (joining another non-branching target, which I simplified out for the example), cc is the calculation of some derived data, dd and ee load the aggregated and derived data to a database, and zz tells the database to refresh a materialized view.
I want to branch on each row of aa because there are relatively frequent changes, we want to have any changes processed as quickly as possible, and it would be very time consuming to reprocess and load all of the unchanged ones.
In the final target zz, it's basically branching on aa and using dd and ee just to register dependencies - if they were "mismatched" then it could be possible run a branch of zz (refreshing a materialized view) before the relevant branch of dd or ee (loading data used by the materialized view).