feat: update implementation of SummarizationAccuracy to use Transform-based approach #214
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nathanng17
approved these changes
Mar 14, 2024
oyangz
approved these changes
Mar 14, 2024
lucfra
reviewed
Mar 19, 2024
| else: | ||
| dataset_configs = [DATASET_CONFIGS[dataset_name] for dataset_name in EVAL_DATASETS[self.eval_name]] | ||
|
|
||
| dataset_configs = get_dataset_configs(dataset_config, self.eval_name) |
There was a problem hiding this comment.
I'm not following here. what if dataset_config has been passed as argument?
Also. I think it would be much cleaner to have under the hood a generic implementation, with dataset taken as required argument and a built-in version that calls the generic version on the built-in datasets. Is this possible?
| dataset, [METEOR_SCORE, ROUGE_SCORE, BERT_SCORE], agg_method=MEAN | ||
| ) | ||
| dataset = pipeline.execute(dataset) | ||
| dataset_scores, category_scores = aggregate_evaluation_scores(dataset, METRIC_NAMES, agg_method=MEAN) |
There was a problem hiding this comment.
can we make this and also the save operation part of standard transforms?
|
|
||
| :param eval_algorithm_config: Summarization Accuracy eval algorithm config. | ||
| def __init__( | ||
| self, eval_algorithm_config: SummarizationAccuracyConfig = SummarizationAccuracyConfig(), use_ray: bool = True |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
SummarizationAccuracyevaluation algorithm so that it uses theTransform/TransformPipelineapproach.get_dataset_configsandexecute_recordalongside their corresponding unit tests.deepcopyfromTransform__init__, as it conflicts with Ray serialization and also didn't provide much value to begin with (and was potentially unexpected/unintuitive).The largest diff for this PR is in the unit tests for
SummarizationAccuracy. Because a lot of code was edited, deleted, moved, reading the diff may not be the best way to view the changes. I'd suggest just jumping to the file directly, and reading through the tests from a fresh perspective, since so much as been changed.The unit tests for
get_meteor_scoreandget_rouge_scorehave been moved totest_summarization_accuracy_metrics.pyand have been adapted to test theMeteorScoreandRougeScoretransforms. Note thatBertScoreisn't tested because unit tests that validate numerical values already exist for theBertscoreModelhelper model.Also note that there was actually some unintended behavior in the original unit tests for
get_rouge_score, where we were passing the same config for every parametrized test case (so therouge_typewas always"rouge2", instead of varying based on the test case). This has been fixed in my new unit tests, and as a result, some of the expected values have changed.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.