Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi, apparently using iteration = "group" is ignored when the pattern is downstream from another pattern? Here's a simple example with some commentary explaining why it would make sense to do things this way:

library(dplyr)
library(targets)
library(tidyr)

list(
  tar_target(
    nested_df,
    tibble(
      id = 1:3,
      nest = rep(list(letters[1:3]), 3)
    )
  ),
  tar_target(
    nested_row,
    # do something expensive per nested row
    nested_df %>% mutate(res = paste0("res", id)),
    pattern = map(nested_df)
  ),
  tar_target(
    unnested_row,
    # expensive result gets replicated
    unnest(nested_row, nest),
    pattern = map(nested_row)
  ),
  tar_target(
    grouped_row,
    # introduce groups
    unnested_row %>%
      group_by(id,nest) %>%
      tar_group(),
    pattern = map(unnested_row),
    iteration = "group"
  ),
  tar_target(
    over_rows,
    # try to do further expensive things per separate row
    # (not actually separate -> risk of e.g. memory overflow)
    rep(
      numeric(10^9),
      length(grouped_row$tar_group)
    ),
    pattern = map(grouped_row),
    iteration = "list"
  )
)

I don't necessarily expect this to work the way I want it to. Another sensible outcome (though undesirable for me) would be to get multiple rows all with the same tar_group value, but as it currently stands, it just returns all the rows as if no grouping had been requested, which feels like it's just silently failing to do anything.

I had a similar situation in #1113, where Will suggested that cheap operations could be performed before the actual pipeline and arranged as desired into a dataframe. That worked at the time, but no longer applies here as some of the steps I want to unnest over are rather complicated.

I've also looked over #348, #554, #971. It seems like his recommendation for these situations is to use static branching, but I'd like to know if there is some other alternative as this results in a network viz with redundant clutter. I've considered saving the actual big object as a side effect and passing along the filename so I can perform the splits I want on a pattern-free target but it feels a bit hacky (and doing it directly is not an option since it would involve loading all the large objects in memory at once).