Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@danielezhu
Copy link
Contributor

Description of changes:
This PR ports over a copy of the helper models from fmeval.eval_algorithms.helper_models.helper_model into fmeval.helper_models. The reason behind keeping the old code is that I don't to make half-baked changes that will break the existing evaluation algorithm implementations. Eventually, when all evaluation algorithms use the new Transform-based design, I will obviously get rid of fmeval.eval_algorithms.helper_models.helper_model.

Note that the new helper model code isn't exactly the same as the original code, but the changes are mainly minor cleanup.

This PR also implements the create_shared_resource utility function, which is used to create a Ray actor out of a "shared resource" (which is typically an object that consumes lots of memory, like a helper model). This utility function allows us to decouple the helper models from Ray, which is one problem that the current implementation of BertscoreHelperModel faces (notice how there is the @ray.remote decorator on top of the class in our current implementation).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@danielezhu danielezhu merged commit 8ed98ab into aws:main Mar 8, 2024
@danielezhu danielezhu deleted the helper_models branch March 8, 2024 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants