What's Changed

New plugins

• probe: Atbash Encoding by @Nakul-Rajpal in #1343
• probe: ansi escape codes in tokenizer by @leondz in #1351
• dropbox repeated token attack by @dchiitmalla in #1244
• probe: DRA (Disguise and Reconstruction Attack) by @patriciapampanelli in #1345
• probes: package hallucination support for dart, perl, & raku by @dchiitmalla in #1243
• probe: Added token smuggling probe module by @mrowebot in #1192

New features

• feature: support detectors including None in output values by @leondz in #1280
• config: model_* to target_* by @leondz in #1383
• cli: add filtering support for --list_* options by @JosephDavisC in #1367

Improved plugins

• bugfix: future probes now use future phrasing by @leondz in #1388
• rename and extend Web injection probes by @erickgalinkin in #1335

Documentation

• fixing probable typo in cliref.rst by @cassiasamp in #1353
• docs(chore): Fix most build warnings by @mikemckiernan in #1359
• docs: reorganize table of concepts and getting started page by @mikemckiernan in #1360
• docs: theme recolor by @leondz in #1385
• docs: auto-include ASR in probe doc page if current calibration exists by @leondz in #1387
• bugfix: ASR figures in probe docs are now actual ASRs by @leondz in #1392

Tuning & fixes

• Specify flit build-system within pyproject.toml by @06kellyjac in #1317
• Update how license is defined in pyproject.toml by @06kellyjac in #1315
• reporting: add basic docs to scripts in analyze pkg by @leondz in #1369
• update: tektronix typo fix by @leondz in #1372
• cli: standardise reporting tool invocation patterns by @SubGlitch1 in #1370
• adding local fixture to setup wordnet environment by @dhruvmalik007 in #1342
• reporting: add aggregation of report jsonl digest objects, and tests for aggregation by @leondz in #1336
• update: rename data/misp_descriptions.tsv to data/tags.misp.tsv by @leondz in #1381

New Contributors

• @cassiasamp made their first contribution in #1353
• @Nakul-Rajpal made their first contribution in #1343
• @SubGlitch1 made their first contribution in #1370
• @dhruvmalik007 made their first contribution in #1342
• @patriciapampanelli made their first contribution in #1345
• @JosephDavisC made their first contribution in #1367

Full Changelog: v0.13.0...v0.13.1

What's Changed

New plugins

• Doctor attack + encoding/Leet by @leondz in #1180
• Simple Assistive Task Linkage Probe by @erickgalinkin in #1319
• Ascii Smuggling by @erickgalinkin in #1299
• Added more generalized version of Markdown exfil probe by @aishwaryap in #1259

New features

• Feature: rename failure -> attack success by @leondz in #1326
• Feature: conversation support by @jmartin-tech in #1254
• Feature: Configurable system prompt by @erickgalinkin in #1337

Improved plugins

• Expand python exploitation payloads by @erickgalinkin in #1300
• Bug fix in python regex in malwaregen.AnyCode detector by @aishwaryap in #1296
• Bug fix to malwaregen anycode detector in keyword using by @aishwaryap in #1302
• Adding wider detection of :: and // in malwaregen.AnyCode detector by @aishwaryap in #1307
• Update datasets and regex for package hallucination by @arjun-krishna1 in #1124

Documentation

• amend docs copyright by @leondz in #1284
• documentation: typo fix by @dbaker-arch in #1286
• docs: improve conformance to PEP-0257 by @jmartin-tech in #1298
• docs: update contact email in readme by @leondz in #1341

Tuning & fixes

• Replace deprecated np.infty with np.inf. by @erickgalinkin in #1283
• restrict datasets version due to new torchcodec dependency by @jmartin-tech in #1290
• Promptinject cleanup by @erickgalinkin in #1292
• add missing import for re in azure generator by @jmartin-tech in #1294
• fix missing '.' in default config probe_spec by @jmartin-tech in #1295
• download models used in translation tests early by @jmartin-tech in #1308
• Tweaks to how wn is used by @06kellyjac in #1316
• Added NASM sections in regex for assembly in malwaregen.Anycode detector by @aishwaryap in #1310
• Use numpy v2 by @06kellyjac in #1314
• update usage of fschat required for compatibility by @jmartin-tech in #1322
• cohere version upgrade by @dchiitmalla in #1252
• improve cache and label automation by @jmartin-tech in #1324
• fix doc extending.rst by @solo-daemon in #1328
• reporting: add tags to digest, tidy naming by @leondz in #1334
• reporting: update defcon descriptions to talk about risk by @leondz in #1348
• judge detectors conversation refactor by @jmartin-tech in #1346

New Contributors

• @dbaker-arch made their first contribution in #1286
• @06kellyjac made their first contribution in #1316
• @solo-daemon made their first contribution in #1328

Full Changelog: v0.12.0...v0.13.0

What's Changed

New plugins

• Add audio NIM model and audio probes by @erickgalinkin in #1163
• Leakreplay refactor by @dchiitmalla in #1264
• probes: refactor fact snippet mixin by @leondz in #1187

New features

• reporting: result summary object by @leondz in #1245
• rm octo generator by @leondz in #1248
• UX: add progress indicators for translation tasks by @jmartin-tech in #1257
• Feature: add google translate language provider by @jmartin-tech in #1232
• update LocalDataPath and testing for python 3.13 support by @jmartin-tech in #1188

Documentation

• docs: correct reference to langproviders configuration by @jmartin-tech in #1253
• chore: The bare .active has too much blast radius by @mikemckiernan in #1262
• docs: Address RST issues by @mikemckiernan in #1263
• chore: Update pre-commit hooks and use RST comment by @mikemckiernan in #1267
• docs: mistral syntax fix and minor code cleanup by @jmartin-tech in #1270
• chore: Reformat docstrings to reduce docs issues by @mikemckiernan in #1268

Tuning & fixes

• data correlation during calibration missed rename by @jmartin-tech in #1240
• bug fix: enable extended detectors, module-specified encoding payloads by default by @leondz in #1238
• adjust calling convention for perf_stats.py by @jmartin-tech in #1246
• remove modality redundant check by @jmartin-tech in #1251
• Llava tests by @dchiitmalla in #1256
• add action for closing stale PRs/issues by @leondz in #1269
• update refusal prompt by @katherine-luna in #1083
• add stale exempt tags by @leondz in #1272
• actions: add 'high priortiy' to never-stale label list by @leondz in #1274
• reset tiers for Complete leakreplay probes by @leondz in #1278

New Contributors

• @mikemckiernan made their first contribution in #1262
• @dchiitmalla made their first contribution in #1256
• @katherine-luna made their first contribution in #1083

Full Changelog: v0.11.0...v0.12.0

What's Changed

New plugins

• Template / SQL injection probes by @erickgalinkin in #1138
• Feature/add mistral generator by @dimensi0n in #1135

New features

• feature: update default toxicity detector by @leondz in #1106
• feature: lightweight probe defaults by @leondz in #1116
• feature: max_workers / give kinda helpful message if too many open files by @leondz in #1110
• Multiprocess enabled logging config by @jmartin-tech in #1140
• Feature: multilingual machine translation by @SnowMasaya in #943
• Support stripping until end think token given empty skip_seq_start in config by @aishwaryap in #1185
• update: add probe tiers by @leondz in #1151
• update: promptinject detector now accepts multiple triggers by @leondz in #1148
• update: rename atkgen probe model to be clear about toxicity by @leondz in #1149
• update: remove ambiguous terms from slur_terms_en payload by @leondz in #1150
• reporting: update report aggregation funcs by @leondz in #1156
• script: qualitative review output by @leondz in #1144
• Add -no-cnv flag support to ggml generators by @IanYHChu in #1189
• reporting: add option for no group score by @leondz in #1194
• reporting: aggregate probe as min by @leondz in #1218
• reporting: add defcon lozenges for relative & absolute scores by @leondz in #1216
• Update/refactor specialwords by @leondz in #1178
• reporting: smooth z-score wildness by @leondz in #1212
• Task: 2025 Q2 scoring calibration by @jmartin-tech in #1231 (thanks to Vijil.ai for data contributions)
• update calibration data for additional probes by @jmartin-tech in #1236
• reporting: change default aggregation by @leondz in #1234

Documentation

• Fix typo in README for leak replay probe by @arjun-krishna1 in #1142
• docs: split 'extending' docs out from 'contributing' by @leondz in #1146
• doc file class corrections by @jmartin-tech in #1200
• docs: formatting fixes by @leondz in #1215

Tuning & fixes

• clear pip cached files by @jmartin-tech in #1129
• set a default soft_probe_prompt_cap in _config by @jmartin-tech in #1133
• enhance response type support from local NeMo-Guardrails by @jmartin-tech in #1131
• bugfix: encoding detection generating false positives by @leondz in #1130
• update: unify on attempt.notes["triggers"] by @leondz in #1147
• Bump datasets version by @JanetVictorious in #1137
• make all workflow permissions explicit by @jmartin-tech in #1162
• update: add soft prompt caps to encoding probes by @leondz in #1154
• update: rename bcp47 to lang by @leondz in #1164
• one detection result per output when testing regex based matches in exploitation by @jmartin-tech in #1167
• Removed detector prefix from eval records by @mrowebot in #1157
• bugfix: HF Detector exceptions now handled gracefully by default by @leondz in #1170
• cache workflow resources by @jmartin-tech in #1173
• refactor probe tier as enum with value in plugin cache by @jmartin-tech in #1159
• update: more meaningful values in tier enums by @leondz in #1176
• block failing litellm 1.67.2 by @leondz in #1179
• ux: give more verbose message for CLI typos by @leondz in #1182
• refactor LatentInjection by @leondz in #1152
• cap litellm max version to avoid their windows bug by @leondz in #1186
• update: rename Translator -> LangProvider and associated elements by @leondz in #1183
• bugfix: reduce latent optimisation permutation explosion by @leondz in #1181
• replicate generator pickle support improvements by @jmartin-tech in #1190
• Fix ambiguous series value error when running --report by @marcorosa in #1171
• add arm64 runner to Linux testing by @jmartin-tech in #1196
• Testing: storage reduction by @jmartin-tech in #1204
• remove unused tooling to free space by @jmartin-tech in #1206
• update deps away from insecure versions by @leondz in #1207
• update Tier impl by @leondz in #1205
• config: sync probe active defaults with default config used in practice by @leondz in #1214
• update: revert default _config.run.generations to 5 by @leondz in #1227
• fix: stop atkgen turn count variation in test relying on fixed turn count by @leondz in #1226
• fix plugin cache tests by @emmanuel-ferdman in #1229
• ux: move translator load msg into translator instantiation by @leondz in #1184
• extract text when processing multi-modal prompts by @jmartin-tech in #1228

New Contributors

• @JanetVictorious made their first contribution in #1137
• @SnowMasaya made their first contribution in #943
• @dimensi0n made their first contribution in #1135
• @mrowebot made their first contribution in #1157
• @aishwaryap made their first contribution in #1185
• @marcorosa made their first contribution in #1171
• @IanYHChu made their first contribution in #1189

Full Changelog: v0.10.3.1...v0.11.0

What's Changed

Tuning & fixes

• defensive coding around capture results in xss content detector by @jmartin-tech #1120

Full Changelog: v0.10.3...v0.10.3.1

What's Changed

New plugins

• Added watsonx.ai generator by @iamnotcj in #1058
• Additional XSS Exfil Probes by @erickgalinkin in #1060

New features

• Generators: add option to skip output btw given delimiters; add hook for postprocessing by @leondz in #1097
• Feature: align hallucinated package named with outputs by @leondz in #1076
• support module.classname config specification for plugins by @jmartin-tech in #1108

Documentation

• pedantic spelling by @dltemple in #1085
• update contributing checklist numbers to be ascending by @shane-rosse in #1115

Tuning & fixes

• update: reorder ditw prompts by @leondz in #1098
• update openai model list by @leondz in #1100
• update: Give clearer message when header encoding fails by @leondz in #1088

What's Changed

• uninclude donotanswer from default probes by @leondz in #1111
• stop forcing generation count for mini phrasing probes by @leondz in #1109

New Contributors

• @iamnotcj made their first contribution in #1058
• @dltemple made their first contribution in #1085
• @shane-rosse made their first contribution in #1115

Full Changelog: v0.10.2...v.0.10.3

What's Changed

New plugins

• Detector: Shields for testing LLM Application Firewalls by @Eric-Hacker in #1059

New features

• Warn if api_key in Config by @erickgalinkin in #1049
• Feature: configuration based rest proxy support by @jmartin-tech in #1073
• Feature: configuration based rest ssl suppression by @jmartin-tech in #1074
• Detector: add startswith matching for StringDetector base class by @leondz in #1075

Documentation

• docs: update index.rst by @eltociear in #1053
• docs: update readme by @leondz in #1050
• adjust label ref syntax by @jmartin-tech in #1057

Tuning & fixes

• StringDetector.case_sensitive now operates as expected by @leondz in #1072
• guard against None in leakreplay attempt history management by @leondz in #1081
• init zscore with enough scope to cover its uses by @leondz in #1086
• Update for latest ollama support by @jmartin-tech in #1092

New Contributors

• @eltociear made their first contribution in #1053
• @Eric-Hacker made their first contribution in #1059

Full Changelog: v0.10.1...v0.10.2

What's Changed

New plugins

• probe: ANSI terminal takeover by @leondz in #1025
• probe: Past Tense Vulnerability by @Shine-afk in #924
• generator: promote OpenAICompatible as first class generator by @jmartin-tech in #1021

New features

• feature: amend garak user-agent by @leondz in #960
• feature: reinforce trust_remote_code=False by @leondz in #979
• generators: add option to specify HTTP codes to skip generation on, for RestGenerator by @leondz in #999
• feature: add experimental features flag to be only accessible in core config by @leondz in #1003
• feature: enable configuration updates by @jmartin-tech in #1026
• feature: use hf chat support by @jmartin-tech in #1047
• feature: per-probe tags now adjustable based on payload selection by @leondz in #1031

Documentation

• doc: Update README.md to notify of migration by @leondz in #1002
• docs: add arxiv lozenge by @leondz in #1030

Tuning & fixes

• remove no longer needed skip-duplicate-actions by @jmartin-tech in #976
• Sanity test for pip install from repository by @jmartin-tech in #1000
• add CODEOWNERS by @jmartin-tech in #1001
• Update reference to reflect migration of org by @jmartin-tech in #1004
• use allowed action name by @jmartin-tech in #1005
• Fixing a few typos by @zoenolan in #1006
• Fix huggingface inference endpoint name by @jmartin-tech in #1011
• detect if tokenizer is not loaded and adjust by @jmartin-tech in #1012
• fix: correct argument in TAP by @harshraj172 in #1022
• refactor attempt to utilized property annotations by @jmartin-tech in #1027
• Restrict transformers version until MPS issue is addressed by @jmartin-tech in #1039
• pass device to transfomers pipeline by @jmartin-tech in #1042
• Fix tap.PAIR run error - run_tap param evaluator_model by @rafaelsandroni in #1045

New Contributors

• @zoenolan made their first contribution in #1006
• @harshraj172 made their first contribution in #1022
• @Shine-afk made their first contribution in #924
• @rafaelsandroni made their first contribution in #1045

Full Changelog: v0.10.0...v0.10.1

What's Changed

New plugins

• detector: llm as a judge by @jmartin-tech in #956
• generator: update openai for o1 models by @leondz in #922
• generator: Ollama by @martinebl in #876
• generator: support of azure openai by @eric-therond in #817
• generator: vision nims by @leondz in #959
• probe: add whois injection to latentinjection by @leondz in #947

New features

• payload/probe separation by @leondz in #870
• data file override support with precedence by @jmartin-tech in #916
• cli: --parallel_attempts hint by @leondz in #932
• reporting: tidy config details, add payload audit info by @leondz in #936
• payloads: web-related, incl. probes.encoding migration by @leondz in #933

Documentation

• Document configuration of garak using YAML by @leondz in #911
• RST formatting updates to restgenerator page by @leondz in #912
• doc: give generatiors.rest list the space it wants and deserves by @leondz in #914
• latent injection doc fix by @leondz in #921
• Contributing Probes documentation by @erickgalinkin in #919
• docs: add example of using Configurable to specify a generator name for load_plugin by @leondz in #929
• docs: NVIDIA generators by @leondz in #917
• update bag reference by @emmanuel-ferdman in #953
• docs: update faq by @leondz in #957
• docs: describe top-level objectives in reference guide by @leondz in #958
• docs: centralise faq by @leondz in #965
• docs: make project scope clear and note that contributors should check it by @leondz in #964
• doc: update to use rst toctree href syntax by @leondz in #967
• docs: clarify "contributing" document by @leondz in #968
• docs: require core config top- and second-level params to be documented by @leondz in #966

Tuning & fixes

• plugin classes should not access DEFAULT_PARAMS by @jmartin-tech in #906
• add test status lozenges for lin/win/osx by @leondz in #903
• utilized the cache to filter base classes by @jmartin-tech in #905
• paraphrase fast consistent model device by @jmartin-tech in #898
• Force cache build in automation by @jmartin-tech in #907
• tweak: defer import of slow Llava* classes by @leondz in #908
• add issue and PR templates by @jmartin-tech in #909
• tweak: new latentinjection probe, add configurability to mini version by @leondz in #910
• bug: set match type correctly in riskywords detectors by @leondz in #918
• bugfix: enable latent injection mini latent jailbreak probe by default by @leondz in #923
• tweak: update migitation strings by @leondz in #925
• bugfix: use distinct report entry type for payload init by @leondz in #930
• add code coverage options by @jmartin-tech in #934
• reporting: update link to bag by @leondz in #935
• probe: missing newline in latentinjection.LatentInjectionFactSnippetEiffel by @leondz in #938
• update: extend calibration with more probes by @leondz in #939
• limit project actions to primary repo by @jmartin-tech in #940
• Add error handling for empty node_results to base.py. by @erickgalinkin in #942
• update: rename replay plugins to divergence by @leondz in #945
• update hint to reference --parallel_attempts by @jmartin-tech in #951
• Configurable TAP probe, refactor judge resources to shared red team by @jmartin-tech in #949
• architecture: factor HFCompatible out by @leondz in #954
• generators: fix rasa issues #961 & #962 by @rgstephens in #963
• fix: add garak/data to pip package by @jmartin-tech in #969

New Contributors

• @martinebl made their first contribution in #876
• @emmanuel-ferdman made their first contribution in #953
• @eric-therond made their first contribution in #817

Full Changelog: v0.9.0.16...v0.10.0

What's Changed

New plugins

• probe: topic pushing by @leondz in #764
• probe: ruby package hallucination by @arjun-krishna1 in #851
• probe: Latent prompt injection by @leondz in #877
• probe: npm package hallucination by @arjun-krishna1 in #861
• probe: rust crate hallucination by @arjun-krishna1 in #873
• generator: Groq API by @mmilenkovic-groq in #896

New features

• extract generations param to be set on probes instead of generators by @jmartin-tech in #837
• secure garak HF assets by @leondz in #854
• Rename knownbadsignatures to av_spam_scanning by @leondz in #850
• enable tokenizer customization in HFDetector by @jmartin-tech in #855
• refactor calibration / z-score code, so z-scores can be shown in CLI at run time by @leondz in #847

Tuning & fixes

• add automation action for plugin_cache.json by @jmartin-tech in #819
• add automation action for plugin_cache.json by @jmartin-tech in #819
• Docs/readmefaqdc32 by @leondz in #828
• fix initialization bug for kwargs in function by @soumilinandi in #827
• suppress LiteLLM logging during import by @jmartin-tech in #834
• relax --probe_tags as supported when other probe options are passed by @jmartin-tech in #836
• rename harness test module by @leondz in #848
• Surface underlying exception msg at plugin load by @leondz in #846
• relax litellm provider constraint by @arjun-krishna1 in #820
• plugin metadata tests by @leondz in #849
• Add explanation of how we treat the word "vulnerability" by @leondz in #852
• add faq items by @leondz in #857
• test buff transform() methods, to get them to load and run by @leondz in #856
• expand doc string that fails test requirements for verbosity by @jmartin-tech in #859
• refresh tutorial on contributing a generator by @leondz in #858
• avoid lambda in _config module level code by @jmartin-tech in #860
• add logging dir, new generator questions to FAQ by @leondz in #866
• revise default generations value to 5 by @leondz in #868
• validate detector doc_uri by @leondz in #865
• fix output error when parallel_requests > 1 and supports_multiple_generations is False by @arjun-krishna1 in #864
• improved messaging & error handling around NIM names by @leondz in #867
• look at the right var for NIM name checking by @leondz in #871
• filter detector template in tests by @jmartin-tech in #874
• have CLI UI render 100% w/o using scientific notation by @leondz in #882
• Generator docs are now tested for as intended by @leondz in #897
• reporting tweaks by @leondz in #901

New Contributors

• @soumilinandi made their first contribution in #827
• @arjun-krishna1 made their first contribution in #820
• @mmilenkovic-groq made their first contribution in #896
• @greshake made their first contribution in #877

Full Changelog: v0.9.0.15...v0.9.0.16