-
-
Notifications
You must be signed in to change notification settings - Fork 32k
[functools] Chaining callables #114284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I have a package that does this, it might be helpful as an example. It is fully typed using mypy plugin: |
The given example:
seems pretty easy to handle with a regular Python function:
This is not only easier, but it also lets a type checker do its job and confirm that the intermediate output signatures match the corresponding input signatures. Also the example hides the main weaknesses of function composers:
Consider this more interesting example:
Here we have dependency injection for the prepocessor, a keyword-only argument, a docstring, logging of intermediate results, a fallback from a fast path, a hyperparameter for the training step, and named intermediate results that can be viewed at the breakpoint. We also get complete type checking. While it is unlikely that user would need all of these at the same time, the proposed The core problem with function chainers is that regular Python functions are already simple to implement but offer vastly more flexibility to meet common programming needs. |
Uh oh!
There was an error while loading. Please reload this page.
Feature or enhancement
Proposal:
This has been proposed before, multiple times, usually under the name
functools.compose
(see the links).My inspiration for this was
torch.nn.Sequential
: Examples ofnn.Sequential
usage in the wild on GitHubWhat I am proposing is functionally equivalent to
torch.nn.Sequential
, but accepts arbitrary Python callables instead of only instances oftorch.nn.Module
(a subset of Python callables).I am really pleased with the type hinting. I will cover the cases:
no arguments (an error)
sequential()
requires at least one argument callable, and type checkers raise a corresponding erroroverload 1
a single callable, the sequential has the parameters and return type of that callable
overload 2
multiple callables, the sequential has the parameters of the first callable and the return type of the last callable
overload 2, gone awry and fixed
if the last callable is overloaded, pyright seems to pick the return type of the first overload; the user can cast the last callable to fix this
The typing of this solution does nothing to validate that each callable is compatible with the next in the sequence, but I don't think that is a problem.
My use case
I am processing inputs to an LLM using huggingface datasets.
datasets.Dataset
objects contain data points that can be modeled asTypedDict
. To process the dataset, I chain calls todatasets.Dataset.map()
, passing in callables that map oneTypedDict
to anotherTypedDict
.When I need to operate outside of the
datasets.Dataset.map()
method chaining paradigm,functools.sequential()
allows me to write this beautifully:Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
bugs.python.org
https://bugs.python.org/issue1660179
https://bugs.python.org/issue35853
github.com/python/cpython/issues
#44584 (comment)
#80034 (comment)
github.com/python/cpython/pull
#11699
stackoverflow.com:
https://stackoverflow.com/questions/51385909/python-passing-arguments-through-a-chain-of-function-calls
https://stackoverflow.com/questions/20454118/better-way-to-call-a-chain-of-functions-in-python
https://stackoverflow.com/questions/72498798/python-chain-several-functions-into-one
https://stackoverflow.com/questions/73996773/most-beautiful-way-of-chaining-a-list-of-functions-in-python-numpy
https://stackoverflow.com/questions/68480108/execute-chain-of-functions-in-python
https://stackoverflow.com/questions/54089072/how-to-consume-chain-functions-pythonically
https://stackoverflow.com/questions/46897852/how-do-you-chain-multiple-functions-and-return-multiple-tuples-from-each-one
https://stackoverflow.com/questions/34613543/is-there-a-chain-calling-method-in-python
https://stackoverflow.com/questions/49112201/python-class-methods-chaining
https://stackoverflow.com/questions/68908195/chaining-functions-calls-on-some-list
https://stackoverflow.com/questions/58473775/chaining-output-between-diffrent-functions
https://stackoverflow.com/questions/17122644/compute-a-chain-of-functions-in-python
https://stackoverflow.com/questions/4021731/execute-functions-in-a-list-as-a-chain
I chose to stop here, but I could probably go on... I just googled
chain functions python site:stackoverflow.com
My question
From @Yhg1s, I read recently #96145 (comment):
A variation of this implementation using
functools.reduce()
was suggested as answers to 8 out of the 13 stackoverflow.com questions I linked above. Practically the same solution was previously proposed in #11699.My question is, does Python endorse this solution? Because in my opinion, a solution is warranted even if this is not it.
The text was updated successfully, but these errors were encountered: