feat: add target_context to dataset columns #266
Conversation
There was a problem hiding this comment.
Non-blocking thought: is target context the only column type that we anticipate being a special case (ie requiring different logic for json parsing)? If we end up having other columns that are similar in behavior to target context, we will want to group these column types together.
Just to double check, is the different logic there because we expect the context to be a list? If so:
|
Interesting. Let's hold off this PR and discuss offline. Maybe we should keep |
Issue #, if available:
Description of changes:
Update: Initially we planned on the
target_contextdataset column taking list of strings. This does not work with ray operations such asmap_batchesdue to issues including ray-project/ray#39559 and other unsupported data type errors. Thus thetarget_contextdataset column has been modified to take a string, and we will use string concatenation when there are multiple target contexts, similar to the existingtarget_outputfield.Description (updated):
target_contextfor evaluation of RAG "ground truth" context provided in a dataset. Thetarget_contextfor each dataset sample is alist ofstring.Modifies.json_parserto accept lists of strings by updating JMESPath output validation and string castingBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.