feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs #281
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
4de8c13 to
605814c
Compare
danielezhu
suggested changes
May 28, 2024
| }, | ||
| ) | ||
| ] * 3 | ||
| with patch.object(s3_client, "upload_part", return_value={"ETag": 1}) as upload_part, patch.object( |
There was a problem hiding this comment.
Can we use a side effect for upload_part where it returns {"ETag": "etag_1"}, {"ETag": "etag_2"}, {"ETag": "etag_3"}? Later, we can validate that self._parts_info contains these values.
| for _ in range(num_of_save_times): | ||
| save_strategy.save(records) | ||
| assert upload_part.call_count == 3 | ||
| assert complete_multipart_upload.call_count == 1 |
There was a problem hiding this comment.
Following up from the comment above, at this point, self._parts_info[PARTS] should look like
[
{PART_NUMBER: 1, E_TAG: "etag_1"},
{PART_NUMBER: 1, E_TAG: "etag_2"},
{PART_NUMBER: 1, E_TAG: "etag_3"}
]
so we can use
complete_multipart_upload.assert_called_once_with(
Bucket=# mocked value,
Key=# mocked value,
UploadId="1234",
MultipartUpload={PARTS: # the list above},
)
danielezhu
approved these changes
May 28, 2024
oyangz
approved these changes
May 28, 2024
oyangz
added a commit
that referenced
this pull request
Jun 11, 2024
* feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs (#281) * feat: modify GeneratePrompt transform to take placeholder_dict (#288) * feat: modify GeneratePrompt transform to take placeholder_dict * fix: unit test * fix: requested changes --------- Co-authored-by: keerthanvasist <[email protected]> Co-authored-by: Xiaoyi Cheng <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
SaveStrategyto allow flexibility in saving localized evaluation outputsDescription of changes:
Introduces a new class
SaveStrategythat allows users define their own saving strategy for localized evaluation outputs. Due to the distributed nature of the computations. If the dataset is large, and all of the data is pulled to the head node, it might lead to OOM errors. In order to avoid that, the data is pulled in batches, andsavefunction is called on each batch at a time. In order to allow this mechanism, while allowing more flexbility in the way outputs are saved, this class works as aContextManager.This PR looks big, but is in essence a small change that reflects in every evaluation algorithm. The main change is in
save_strategy.pyandeval_algorithms/common.py(which are new files).Incidentally, this PR also updates ray version from
2.9.1to2.23.0By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.