Add a utility to detect LLM usage in a config #3869
Conversation
| expectation: The expected result | ||
| request: pytest `request` fixture | ||
| """ | ||
| config = request.getfixturevalue(config) |
There was a problem hiding this comment.
@jeffkinnison Just curious - is there a reason you decided to create all of the tests as individual (and nested) fixtures as opposed to just creating each of them directly as parameterized tests?
There was a problem hiding this comment.
It was mostly for readability on my part, though it does also allow us to reuse the configs for both the dict and object cases.
There was a problem hiding this comment.
Got it! Thanks for the changes with making them module scoped. I think that because they're relatively light weight fixtures, this is okay to do and shouldn't slow things down or cause memory related issues.
There was a problem hiding this comment.
Last callout is that because some of the fixtures are dependent on each other, I think it may prevent pytest from parallelizing some of the tests. Again, it seems okay in this case given that the test itself is mostly an in place config or dict check and nothing too heavy, so each of the tests should run quickly.
The LLM encoder in ECD needs to share large parts of the code base with the LLM model type, but branching currently inspects configs for model type only. This adds a utility that detects LLM model type or the usage of LLM encoder in text features for ECD models (LLM encoder is not currently valid for GBM). This can be used as a replacement for expressions like
if config.model_type =="llm":.