-
Notifications
You must be signed in to change notification settings - Fork 33
Add QuiltVQA dataset & FreeFormQuestionPromptTemplate template
#912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…d ExtractAnswerFromJson postprocess
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great addition, looks nice. Would be good to add a config file for QuiltVQA that exemplifies how the dataset is loaded (metric section could be commented out for now until we merge the LLMJudge metric), and to adapt the CoT tag restriction. Other than that only minor comments
Config file will be added in a follow PR once we have merged a first LLM Judge metric. |
Contributes to #908
FreeFormQuestionPromptTemplateprompt examplesQuiltVQA
With few-shot-examples
generates this prompt: