docs: cas user dev guide for technique and intent#1874
Conversation
jmartin-tech
left a comment
There was a problem hiding this comment.
Targeted thoughts, there is likely more in the end to end use case to consider.
| buff selection (:doc:`_spec`, :doc:`_selection`) and handed to the intent service | ||
| (:doc:`services/intentservice`), which checks the codes against the typology and | ||
| works out the active intent set. A selected ``IntentProbe`` builds its prompts | ||
| from each active intent's stubs, tagging every attempt with its intent, and the |
There was a problem hiding this comment.
Given this should the prompt pruning process co-ordinate the stubs that are output? This relates to #1838 in that during a run it may be advantageous to ensure the same stubs are used across probes that can utilize them.
| * ``run.intent_spec`` -- the default intent scope used when no ``intent:`` | ||
| selector is given. | ||
| * ``cas.expand_intent_tree`` -- whether a category code expands to all of its | ||
| leaves. | ||
| * ``cas.serve_detectorless_intents`` -- whether to include intents that have no | ||
| mapped detector. | ||
|
|
||
| (``cas.trust_code_stubs`` enables code-generated stubs and is covered in the | ||
| developer guide.) |
There was a problem hiding this comment.
This is no longer the case, intents come from run.spec per #1866.
cas does not seem like a top level option to me, but more something that is part of the run configuration. Do we want to retain expand_intent_tree and serve_detectorless_intents? Both these options seem possible to eliminate, I don't see a reason to ever not expand a tree and removal of a leaf could be achived with negative(-) intent selection entries. Probes should come with a primary detector that acting as a fallback avoiding the reason to have serve_detectorless_intents.
As to trust_code_stubs I am not sold this should exist either. Currently these are all deployed code that is part of the install so the trust level can be implied. If future additions allow code based extension imported from the user XDG data path I could see adding this however again this could be a run level restriction to allow code from XDG paths more generally to support a user supplied plugin.
There was a problem hiding this comment.
-
expand_intent_tree: Agreed! In PR fix: drop name fallback for stub-less intents #1876, we're removing the intent name fallback. As a result, any non-leaf intent will log a warning that it has no stubs and will be skipped. Since today all non-leaf intents have emptydefault_stubs, settingexpand_intent_tree=Falseno longer makes sense. -
serve_detectorless_intents: I'm not entirely certain we should remove it. In fact, intents without a mapped detector fall back to the probe's detector. However, we only use this fallback ifserve_detectorless_intents=True. Withserve_detectorless_intents=False, these intents aren't served by IntentService. On the other hand, we need to consider thatprobe_detector_namesdepends on what the probe declares. ForGrandmaIntent, the default isalways.Fail, so serving detectorless intents with=Trueresults in unreliable detections. -
trust_code_stubs: Agreed.
There was a problem hiding this comment.
The reasoning to keep serve_detectorless_intents is unclear, I would hazard the current fallback to alwaysFail in the first IntentProbe implementation indicates to me that the primary detector may have simply been a placeholder that could be reconsidered. Though, I could also see it as logical to have the default detector when no intent specific detector is available be always.Fail since how to detect success is not defined for a specific random intent stub.
2ee2493 to
7d1f78b
Compare
7d1f78b to
642a611
Compare
Signed-off-by: Patricia Pampanelli <[email protected]>
Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Patricia Pampanelli <[email protected]>
Signed-off-by: Patricia Pampanelli <[email protected]>
Signed-off-by: Patricia Pampanelli <[email protected]>
Signed-off-by: Patricia Pampanelli <[email protected]>
Signed-off-by: Patricia Pampanelli <[email protected]>
642a611 to
29221fb
Compare
jmartin-tech
left a comment
There was a problem hiding this comment.
Some minor adjustments I will work on now.
Signed-off-by: Jeffrey Martin <[email protected]>
Signed-off-by: Jeffrey Martin <[email protected]>
de34fdb
into
NVIDIA:feature/technique_intent
Summary
docs/source/cas.rstinto a user-facing guide for garak's Context-AwareScanning (CAS) system: overview, the CTMS typology, an execution overview,
configuration, and reading per-intent results.
docs/source/extending.probe.rst.reporting.rst,payloads.rstand_selection.rst.Resolves #1709.
Dependencies
run.specselection grammar #1831 and Feature/intent run spec selector #1866. The guide documents theintent:selector,run.intent_specand thecas.*config introduced by those PRs; links to the updated_spec/configurable/clirefpages depend on them.