feat: add configurable param logical_operator (OR/AND) to factual knowledge #307
Conversation
| :param output_key: The key corresponding to the factual knowledge score that | ||
| will be added to the input record. | ||
| :param target_output_delimiter: See the docstring in `FactualKnowledgeConfig`. | ||
| :param logical_operator: See the docstring in `FactualKnowledgeConfig`. |
There was a problem hiding this comment.
Please do not refer to other config param.
| raise EvalAlgorithmClientError( | ||
| "Empty target_output_delimiter is provided. Please either provide a non-empty string, or set it to None." | ||
| ) | ||
| if self.logical_operator != "OR" and self.logical_operator != "AND": |
There was a problem hiding this comment.
nit:
if self.logical_operator not in ["OR", "AND"]:
| os.environ["PARALLELIZATION_FACTOR"] = "2" | ||
|
|
||
| config = FactualKnowledgeConfig("<OR>") | ||
| config = FactualKnowledgeConfig("<OR>", "OR") |
There was a problem hiding this comment.
If you do not change this config, the exist tests should also pass, right?
There was a problem hiding this comment.
yup! I just wanted to add "OR" becuase "<OR>" was also there
There was a problem hiding this comment.
Looks good, one comment:
As discussed before, in my opinion the user experience would be better if target_output_delimiter and logical_operator where linked:
logical_operator = "OR"impliestarget_output_delimiter=<OR>andlogical_operator = "AND"impliestarget_output_delimiter=<AND>.
Is it possible to set the default target_output_delimiter dynamically based on the logical_operator? Alternatively we could raise a warning when the user modifies the logical_operator to "AND" but keeps the target_output_delimiter as <OR>.
| model_input="Who is Andrew R. Jassy?", | ||
| model_output="Chief Executive Officer of Amazon.com Inc.", | ||
| target_output="Chief Executive Officer of Amazon.com, Inc.", | ||
| delimiter="<OR", |
There was a problem hiding this comment.
Delimiter is different here (doesn't matter functionally but could be confusing?)
There was a problem hiding this comment.
@polaschwoebel We will work towards to change target_output (ground_truth) a list as the Ray issue is solved, then the delimiter should go away and there won't be conflict.
Similar to context(s): #305
There was a problem hiding this comment.
Makes sense. Here, I simply meant "" vs "<OR" (small typo in the code above).
Description of changes: I created a new configurable parameter called
logical_operatorthat accepts "OR" (as default) and "AND" in FactualKnowledgeConfig. I added checks so that when thelogical_operatoris "AND", all possible target outputs must be included in model generation in order for the answer to be correct. I also updated all of the current configurations, and created new unit tests to check for behavior given invalid configurations and scores for valid inputs.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.