feat: add quasi_exact_inclusion metric to factual knowledge; change factual_knowledge score name to exact_inclusion
#302
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
polaschwoebel
previously approved these changes
Jul 5, 2024
There was a problem hiding this comment.
Nice!
Regarding naming: Should we rename the scores
FACTUAL_KNOWLEDGE-->EXACT_INCLUSIONandFACTUAL_KNOWLEDGE_FUZZY-->QUASI_EXACT_INCLUSION
to be consistent with the rest of the codebase and specifically QA accuracy?
This way, now that we have more than one score for the factual knowledge evaluation, we would not call both the evaluation itself and the score "factual knowledge".
| from fmeval.eval_algorithms.util import ( | ||
| validate_dataset, | ||
| get_dataset_configs, | ||
| normalize_text_quac_protocol, |
There was a problem hiding this comment.
Good idea to move shared functions out, let's do more of this as we continue to modularize the code base. I suggest we add a metrics.py in addition to utils.py for all metric-related computations going forward.
xiaoyi-cheng
requested changes
Jul 8, 2024
xiaoyi-cheng
approved these changes
Jul 9, 2024
quasi_exact_inclusion metric to factual knowledge; change factual_knowledge score name to exact_inclusion
xiaoyi-cheng
previously approved these changes
Jul 9, 2024
oyangz
previously approved these changes
Jul 9, 2024
|
Thanks for adding the score descriptions, could you update this list for reporting to use the new factual knowledge score names as well? |
xiaoyi-cheng
previously approved these changes
Jul 9, 2024
oyangz
previously approved these changes
Jul 10, 2024
polaschwoebel
previously approved these changes
Jul 10, 2024
1aee116
xiaoyi-cheng
approved these changes
Jul 10, 2024
oyangz
approved these changes
Jul 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes: I added a new metric to factual knowledge called "quasi_exact_inclusion" which tests checks whether the target output is included in the model output after both outputs are normalized. Since both qa_accuracy and factual_knowledge.py had similar logic in their normalization functions, I moved this to util and imported the function from there. I updated the EvalScores and outputs so they would output both metrics. I also updated the unit tests (to check for both metrics) and added more tests to check the second metrics output. Then, I added more unit tests to check the behavior of private functions in both qa_accuracy and factual_knowledge. Lastly, I updated the integration tests for factual_knowledge so that they would also check the second metric. I reviewed the code and tried to make sure that I updated documentation of functions/files so that they would incorporate both metrics.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.