Thanks to visit codestin.com
Credit goes to Github.com

Skip to content

Comments

ENH: Add pre- and post-processing with NestedPipeFunc classmethod decorator#595

Draft
LennartGevers wants to merge 2 commits intopipefunc:mainfrom
LennartGevers:NestedPipeFunc-wrap-method
Draft

ENH: Add pre- and post-processing with NestedPipeFunc classmethod decorator#595
LennartGevers wants to merge 2 commits intopipefunc:mainfrom
LennartGevers:NestedPipeFunc-wrap-method

Conversation

@LennartGevers
Copy link
Contributor

@LennartGevers LennartGevers commented Jan 24, 2025

I wanted to share some of my progress on #565 Feature Idea/Request: Macro for Pre- and Post-process steps of a PipeFunc. for some feedback.
It's still WIP ATM I justed wanted to outline my current idea of the feature and see wether it works. (So don't mind the process_func function defined within the classmethod. Also, all of the names are not final ofc).

I actually had some iterations on the design of this feature but now came up with the @NestedPipeFunc.inject() decorator.
It's a bit more generalized and flexible than @NestedPipeFunc.wrap(pre=my_pre, post=my_post) but also more compact and can actually serve several purposes.

The decorator can be used to add pre- and post process up/downstream nodes to a PipeFunc like below:

import math
from pipefunc import PipeFunc, pipefunc, NestedPipeFunc, Pipeline


def with_factor(x: float, factor: float) -> int:
    return math.floor(x* factor)


@pipefunc("post_a")
def post_a(number: int) -> str:
    return str(number) + "1"


@NestedPipeFunc.inject(
    [with_factor, post_a],
    input_overwrites=dict(x="with_factor", number="a"),
    output_overwrites=dict(a="post_a"),
)
@pipefunc("a")
def a(x: int, y: int) -> int:
    return x + 1 - y


test_pipeline = Pipeline([a])
print(test_pipeline.run("a", kwargs={"x": 1.1, "y": 3.1, "factor": 3}))
# returns 0.89999999999999991

But the catch is that because NestedPipeFunc.inject just accepts a list of Pipefuncs/Callables, one can actually use it to manage reuseable subroutines/subpipelines. This is quite simmilar to Hamiltons subdag decorator but also includes the pipe.input and pipe.output feature.

Maybe I'd also implement that one can pass default values to NestedPipeFunc.inject in order to set default values of those subroutines, like

@NestedPipeFunc.inject(
    [with_factor, post_a],
    input_overwrites=dict(x="with_factor", number="a"),
    output_overwrites=dict(a="post_a"),
    default_values=dict(factor=3)
)
@pipefunc("a")
def a(x: int, y: int) -> int:
    return x + 1 - y

to make the usage of subroutines more flexible.
I'm not settled with the names input_overwrites and output_overwrites, I think there might be terms which are suited better and would convey the idea of the feature better.

@LennartGevers LennartGevers changed the title Nested pipe func wrap method ENH: Add pre- and post-processing with NestedPipeFunc classmethod decorator Jan 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 24, 2025

✅ PR Title Formatted Correctly

The title of this PR has been updated to match the correct format. Thank you!

@codspeed-hq
Copy link

codspeed-hq bot commented Jan 24, 2025

CodSpeed Performance Report

Merging #595 will not alter performance

Comparing LennartGevers:NestedPipeFunc-wrap-method (31705e2) with main (441e5d2)

Summary

✅ 4 untouched benchmarks

@basnijholt
Copy link
Collaborator

basnijholt commented Jan 24, 2025

Good idea to make a PR early to get some eyes on it! 👀

I like the idea of making NestedPipeFunc.inject 😄

I have starred at the example for a while now but I need some help understanding it.

here for example

input_overwrites=dict(x="with_factor", number="a"),

re: x="with_factor"

  • key x is in input to the current @pipefunc (a) (where this line is written),
  • its value ("with_factor") refers to the function that is injected

re: number="a":

  • key number appears in a different @pipefunc (post_a) which is injected
  • its value is the output_name of the current @pipefunc (a)

Is this example you wrote correct? If so, I do not understand what is going on or the logic 😅

Perhaps I need some fresh eyes before I understand it 😂

@LennartGevers
Copy link
Contributor Author

LennartGevers commented Jan 24, 2025

Alright, no actually I think this confirms my suspicion that this logic is yet a bit to cumbersome.
So the dictionary args should define "rearrangements" of func dependencies but this clearly needs some work to be done. I'll try to come up with a more straightforward solution that only needs one dict

@LennartGevers
Copy link
Contributor Author

LennartGevers commented Jan 24, 2025 •
edited
Loading

I actually think it's better to drop this overwrites feature and only rely on renames defined by the user to achieve the same results. I think that is where I make it too complicated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LennartGevers @basnijholt