feat: update GetModelResponse transform to support multiple model invocations on the same input #220
Conversation
There was a problem hiding this comment.
Since we're making changes to this part of the code I would strongly suggest to decouple generation and input probability computation (i.e. making two different functions).
This slipped in the first version of the library but I see no reason to compute them together in the same function. No eval algo except stereotyping needs the probability. And stereotyping only need the probability, not the output. So basically in all evals we're wasting a lot of compute.
Let me know if I'm missing something :)
|
I see that actually we would need to change another part of the code. but still I think it's worthwhile at least having two different transforms if we don't want to change the predict method now |
|
I see your point about having a separate transform for getting log probabilities, but note that |
|
The implementation of input log probability and generation are model/framework dependent. For example see this https://github.com/aws/fmeval/blob/main/examples/custom_model_runner_hf.ipynb for the HF model runner. There, the model is called twice. So are you referring to Jumpstart models? Also many models like those in Bedrock don't return the log prob so I think it would be much cleaner to have a NotImplemented or some similar error rather than returning a tuple with a None. Finally, it's pretty standard for other eval frameworks to separate the two. See the base class I'm fine to change this later on, but let's do it at a certain time |
Description of changes:
This PR updates
GetModelResponseso that it supports the semantic robustness use case where a model is invoked on the same input multiple times.Currently, the instance attribute
input_to_output_keysmaps a model input key to a list representing the model response keys (where a response is of the form(model_output, log_probability)where bothmodel_outputandlog_probabilityare optional).This PR changes
input_to_output_keyssuch that a model input key is mapped to a list of tuples, where each tuple is of the form(model_output, log_probability).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.