Body
provide the functionality that a Dag depends on an asset, but waits for more than one partition to be ready. (e.g., a Dag only fires when all the 31 partitions of the month arrive)
Introduce RollupMapper and the initial window types (Month, Week, Day), plus the scheduler plumbing that holds the downstream until all expected upstream keys are present, then fires once.
Committer
Body
provide the functionality that a Dag depends on an asset, but waits for more than one partition to be ready. (e.g., a Dag only fires when all the 31 partitions of the month arrive)
Introduce
RollupMapperand the initial window types (Month, Week, Day), plus the scheduler plumbing that holds the downstream until all expected upstream keys are present, then fires once.Committer