Thanks to visit codestin.com
Credit goes to github.com

Skip to content

FEA SLEP006: debugging tool for sample props #18936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #22893
adrinjalali opened this issue Nov 28, 2020 · 7 comments
Open
Tracked by #22893

FEA SLEP006: debugging tool for sample props #18936

adrinjalali opened this issue Nov 28, 2020 · 7 comments
Milestone

Comments

@adrinjalali
Copy link
Member

As stated by @jnothman in #16079 (comment), we should have a debugging tool to show users how metadata is routed around.

Having this as a separate issue since it requires a rather detailed discussion of its own.

@adrinjalali adrinjalali added this to the 1.0 milestone Nov 28, 2020
@adrinjalali
Copy link
Member Author

There are a few approaches which come to my mind, thinking out loud.

  1. introduce introspection methods and ask estimators and meta-estimators where they send each parameter.
  • this has the benefit of not having to fit before inspecting the routing
  • it has the downside of having to implement the introspection separately, and may not stay in sync with the other parts of the implementation and may be buggy
  1. introduce a mechanism to tell build_method_metadata_params and build_router_metadata_request to record what they get and what they do with it.
  • less implementation, and almost reliable as long as every estimator use these methods
  • requires a call to fit, and probably a config param in a context, which has its own issue with parallelism
  1. introspect using get_metadata_request's code
  • doesn't require extra implementation by each estimator, and doesn't require a call to fit
  • it may be tricky to get right, and it probably won't be clean and maintainable code
  1. hook into methods such as fit and score
  • would probably require some config param and it again would have issues with parallelism
  • may introduce non-trivial overhead if implemented as a decorator for instance
  • keeping track of variables after slicing (in GridSearchCV for instance) is not trivial

@jnothman
Copy link
Member

Thanks for this @adrinjalali... your comments here seem to be more about what internal APIs will facilitate this inspection, rather than how the user will see the results in a way that is interpretable to them.

I don't think it solves it very well, but I'd wondered about having some kinds of annotations on diagram displays... I'd also been thinking about generating descriptions of composite estimators in natural language!

Sigh. This is a hard part of the puzzle, but for now it's a reason why I'd continue arguing for explicit declarations in code rather than implicit/batteries-included routing.

@adrinjalali adrinjalali modified the milestones: 1.0, 1.1 Aug 22, 2021
@jeremiedbb
Copy link
Member

Since we to move the slep006 to 1.2, I guess it makes sense to move this as well.

@jeremiedbb jeremiedbb modified the milestones: 1.1, 1.2 Mar 28, 2022
@adrinjalali adrinjalali changed the title FEA debugging tool for sample props FEA SLEP006: debugging tool for sample props Jul 16, 2022
@glemaitre glemaitre modified the milestones: 1.2, 1.3 Nov 16, 2022
@jeremiedbb
Copy link
Member

moving to 2.0 (#25851)

@adrinjalali
Copy link
Member Author

xref: #24027 (comment)

@haiatn
Copy link
Contributor

haiatn commented Jul 29, 2023

What is needed for this to complete?

@adrinjalali
Copy link
Member Author

a better __repr__ and a tool to visualize where metadata goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

5 participants