Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: update the default prompt templates for the built-in datasets #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Mar 13, 2024
Merged

fix: update the default prompt templates for the built-in datasets #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
30a4ae6
fix: update the default prompt templates for the built-in datasets
jmikko Mar 11, 2024
22662da
Update unit test
jmikko Mar 11, 2024
162e69f
Update unit tests
jmikko Mar 11, 2024
53a6609
Update integration tests
jmikko Mar 13, 2024
912f863
Update integration tests (meteor)
jmikko Mar 13, 2024
d994a50
Update integration tests (delta rouge)
jmikko Mar 13, 2024
3c92f26
Update integration tests (delta bert)
jmikko Mar 13, 2024
e5dbeaf
Update integration tests (delta meteor)
jmikko Mar 13, 2024
7dcf156
Update integration tests (delta meteor)
jmikko Mar 13, 2024
e9a7fb9
Merge branch 'main' into new_default_prompt_template
jmikko Mar 13, 2024
be3c8f6
Merge branch 'main' into new_default_prompt_template
jmikko Mar 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions src/fmeval/eval_algorithms/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,14 +215,14 @@ class ModelTask(Enum):
DEFAULT_PROMPT_TEMPLATE = "$feature"

BUILT_IN_DATASET_DEFAULT_PROMPT_TEMPLATES = {
BOOLQ: 'Respond to the following question. Valid answers are "True" or "False". $feature Answer:',
TRIVIA_QA: "Respond to the following question with a short answer: $feature Answer:",
NATURAL_QUESTIONS: "Respond to the following question with a short answer: $feature Answer:",
GIGAWORD: "Summarise the following text in one sentence: $feature",
GOV_REPORT: "Summarise the following text in a few sentences: $feature",
BOOLQ: 'Respond to the following question. Valid answers are "True" or "False". $feature',
TRIVIA_QA: "Respond to the following question with a short answer: $feature",
NATURAL_QUESTIONS: "Respond to the following question with a short answer: $feature",
GIGAWORD: "Summarize the following text in one sentence: $feature",
GOV_REPORT: "Summarize the following text in a few sentences: $feature",
WOMENS_CLOTHING_ECOMMERCE_REVIEWS: (
"Classify the sentiment of the following review with 0 (negative sentiment)"
" or 1 (positive sentiment). Review: $feature. Classification:"
" or 1 (positive sentiment): $feature"
),
}

Expand Down
30 changes: 15 additions & 15 deletions test/integration/test_summarization_accuracy_semantic_robustness.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,11 @@ class TestSummarizationAccuracySemanticRobustness:
},
expected_evaluate_scores={
ROUGE_SCORE: 0.021908,
METEOR_SCORE: 0.107623,
BERT_SCORE: 0.559997,
DELTA_ROUGE_SCORE: 0.019394,
DELTA_METEOR_SCORE: 0.044310,
DELTA_BERT_SCORE: 0.033714,
METEOR_SCORE: 0.105540,
BERT_SCORE: 0.559893,
DELTA_ROUGE_SCORE: 0.021061,
DELTA_METEOR_SCORE: 0.046859,
DELTA_BERT_SCORE: 0.032417,
},
),
TestCaseEvaluate(
Expand All @@ -84,11 +84,11 @@ class TestSummarizationAccuracySemanticRobustness:
},
expected_evaluate_scores={
ROUGE_SCORE: 0.021908,
METEOR_SCORE: 0.107623,
BERT_SCORE: 0.559998,
DELTA_ROUGE_SCORE: 0.035696,
DELTA_METEOR_SCORE: 0.056931,
DELTA_BERT_SCORE: 0.027971,
METEOR_SCORE: 0.105540,
BERT_SCORE: 0.559893,
DELTA_ROUGE_SCORE: 0.037362,
DELTA_METEOR_SCORE: 0.056909,
DELTA_BERT_SCORE: 0.026363,
},
),
TestCaseEvaluate(
Expand All @@ -103,11 +103,11 @@ class TestSummarizationAccuracySemanticRobustness:
},
expected_evaluate_scores={
ROUGE_SCORE: 0.021908,
METEOR_SCORE: 0.107623,
BERT_SCORE: 0.559998,
DELTA_ROUGE_SCORE: 0.032187,
DELTA_METEOR_SCORE: 0.057705,
DELTA_BERT_SCORE: 0.027511,
METEOR_SCORE: 0.105540,
BERT_SCORE: 0.559893,
DELTA_ROUGE_SCORE: 0.030725,
DELTA_METEOR_SCORE: 0.054234,
DELTA_BERT_SCORE: 0.026511,
},
),
],
Expand Down
5 changes: 1 addition & 4 deletions test/unit/eval_algorithms/test_eval_algorithm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,5 @@ def test_get_default_prompt_template():
WHEN get_default_prompt_template() method is called
THEN expected default prompt template is returned
"""
assert (
get_default_prompt_template("trivia_qa")
== "Respond to the following question with a short answer: $feature Answer:"
)
assert get_default_prompt_template("trivia_qa") == "Respond to the following question with a short answer: $feature"
assert get_default_prompt_template("my_custom_dataset") == "$feature"