-
Help
DescriptionHi! Is there a way to conclude a pipeline if a condition is met? I decided to use the files as ground truth since I needed something easy to inspect; also, the mere existence of the file is not enough to skip the reprocessing since its content is also relevant to this decision. But if all combinations have already been processed, the pipeline does error: map() cannot be run on an empty set. I'd like to have a more graceful stopping if this condition is met, but I could not find a graceful way of stopping the pipelines without marking all downstream targets as errored. It's better than a full-fledged error or a bunch of errored branches, but still, it's reported as an errored pipeline, which is not truthful. Is there any more elegant solution that marks the pipeline as successfully completed? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I had the same issue, and ended up with a solution that I think works quite nicely (although a bit hacky). You simply wrap your target, or a list of any number of targets, inside an See example below: Basically, all the targets that are wrapped by this command just return a dummy character (and creates a dummy file as well) which is thus a valid return value (i.e. no erroring) for both tar_target and tarchetypes::tar_file-targets (i.e. tar_target() with format = "file"). And since these targets don't care about the upstream targets either during the skip_condition, you can chain many of them. You can even use this method to dynamically end the pipeline early, or even (effectively) select a specific path through the target DAG during run-time, while ending up at the same downstream targets - which I think is very powerful! For this, you'd just check for which target did not return NULL.txt in the "joining target" in a similar if-else block, and then going forward with the correct dependecy. |
Beta Was this translation helpful? Give feedback.
targetsisn't designed for this, so there isn't a perfect solution. The error sounds like it gets the job done. Another thing you could try is to have each target check the condition and then runtar_cancel()if it is already met.