Thanks to visit codestin.com
Credit goes to github.com

Skip to content

docs: cas user dev guide for technique and intent#1874

Merged
jmartin-tech merged 8 commits into
NVIDIA:feature/technique_intentfrom
patriciapampanelli:docs/cas-user-dev-guide
Jun 30, 2026
Merged

docs: cas user dev guide for technique and intent#1874
jmartin-tech merged 8 commits into
NVIDIA:feature/technique_intentfrom
patriciapampanelli:docs/cas-user-dev-guide

Conversation

@patriciapampanelli

Copy link
Copy Markdown
Collaborator

Summary

  • Expand docs/source/cas.rst into a user-facing guide for garak's Context-Aware
    Scanning (CAS) system: overview, the CTMS typology, an execution overview,
    configuration, and reading per-intent results.
  • Add a "Writing an IntentProbe" section to docs/source/extending.probe.rst.
  • Add concise cross-links to the guide from reporting.rst, payloads.rst and _selection.rst.

Resolves #1709.

Dependencies

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Targeted thoughts, there is likely more in the end to end use case to consider.

Comment thread docs/source/cas.rst Outdated
Comment thread docs/source/cas.rst
buff selection (:doc:`_spec`, :doc:`_selection`) and handed to the intent service
(:doc:`services/intentservice`), which checks the codes against the typology and
works out the active intent set. A selected ``IntentProbe`` builds its prompts
from each active intent's stubs, tagging every attempt with its intent, and the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this should the prompt pruning process co-ordinate the stubs that are output? This relates to #1838 in that during a run it may be advantageous to ensure the same stubs are used across probes that can utilize them.

Comment thread docs/source/cas.rst Outdated
Comment on lines +98 to +106
* ``run.intent_spec`` -- the default intent scope used when no ``intent:``
selector is given.
* ``cas.expand_intent_tree`` -- whether a category code expands to all of its
leaves.
* ``cas.serve_detectorless_intents`` -- whether to include intents that have no
mapped detector.

(``cas.trust_code_stubs`` enables code-generated stubs and is covered in the
developer guide.)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer the case, intents come from run.spec per #1866.

cas does not seem like a top level option to me, but more something that is part of the run configuration. Do we want to retain expand_intent_tree and serve_detectorless_intents? Both these options seem possible to eliminate, I don't see a reason to ever not expand a tree and removal of a leaf could be achived with negative(-) intent selection entries. Probes should come with a primary detector that acting as a fallback avoiding the reason to have serve_detectorless_intents.

As to trust_code_stubs I am not sold this should exist either. Currently these are all deployed code that is part of the install so the trust level can be implied. If future additions allow code based extension imported from the user XDG data path I could see adding this however again this could be a run level restriction to allow code from XDG paths more generally to support a user supplied plugin.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • expand_intent_tree: Agreed! In PR fix: drop name fallback for stub-less intents #1876, we're removing the intent name fallback. As a result, any non-leaf intent will log a warning that it has no stubs and will be skipped. Since today all non-leaf intents have empty default_stubs, setting expand_intent_tree=False no longer makes sense.

  • serve_detectorless_intents: I'm not entirely certain we should remove it. In fact, intents without a mapped detector fall back to the probe's detector. However, we only use this fallback if serve_detectorless_intents=True. With serve_detectorless_intents=False, these intents aren't served by IntentService. On the other hand, we need to consider that probe_detector_names depends on what the probe declares. For GrandmaIntent, the default is always.Fail, so serving detectorless intents with =True results in unreliable detections.

  • trust_code_stubs: Agreed.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning to keep serve_detectorless_intents is unclear, I would hazard the current fallback to alwaysFail in the first IntentProbe implementation indicates to me that the primary detector may have simply been a placeholder that could be reconsidered. Though, I could also see it as logical to have the default detector when no intent specific detector is available be always.Fail since how to detect success is not defined for a specific random intent stub.

Comment thread docs/source/extending.probe.rst Outdated
Comment thread docs/source/extending.probe.rst Outdated
@patriciapampanelli patriciapampanelli moved this from In Progress to In Review in garak / Context Aware Scanning Jun 23, 2026
@patriciapampanelli patriciapampanelli force-pushed the docs/cas-user-dev-guide branch 3 times, most recently from 2ee2493 to 7d1f78b Compare June 23, 2026 23:47
@patriciapampanelli patriciapampanelli marked this pull request as ready for review June 24, 2026 01:40
@patriciapampanelli patriciapampanelli force-pushed the docs/cas-user-dev-guide branch from 7d1f78b to 642a611 Compare June 24, 2026 01:45
@patriciapampanelli patriciapampanelli force-pushed the docs/cas-user-dev-guide branch from 642a611 to 29221fb Compare June 26, 2026 18:13

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor adjustments I will work on now.

Comment thread docs/source/cas.rst Outdated
Comment thread docs/source/cas.rst
Signed-off-by: Jeffrey Martin <[email protected]>
Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech jmartin-tech changed the title Docs/cas user dev guide Docs: cas user dev guide for technique and intent Jun 30, 2026
@jmartin-tech jmartin-tech changed the title Docs: cas user dev guide for technique and intent docs: cas user dev guide for technique and intent Jun 30, 2026
@jmartin-tech jmartin-tech merged commit de34fdb into NVIDIA:feature/technique_intent Jun 30, 2026
16 checks passed
@github-project-automation github-project-automation Bot moved this from In Review to Done in garak / Context Aware Scanning Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants