This lab simulates OWASP LLM02:2025 Sensitive Information Disclosure through training data memorization — specifically, PII leakage from a fine-tuned medical assistant model.
Unlike prompt injection, where the secret lives in the runtime context, this lab demonstrates a different failure mode: PII baked into model weights during fine-tuning. That makes the leak persistent across sessions and invisible to normal context inspection.
This is a good fit for LLMForge because it covers a part of LLM security that many labs do not: model-level memorization, not just prompt-level attacks.
Why this lab matters
Medical AI teams often fine-tune models on patient data after running an anonymization pipeline. The problem is that preprocessing bugs can miss edge cases such as:
- names with punctuation
- names embedded in free text
- names in non-standard fields
- schema inconsistencies
- normalization issues
If that happens before training, the model can memorize the leaked PII. Once it is in the weights, the data can resurface later through ordinary generation.
This lab makes that failure mode concrete, reproducible, and easy to understand.
Scenario
AcmeMed Corp fine-tuned an internal medical assistant model using 100 synthetic patient records.
Their anonymization pipeline correctly sanitized 77 of them, but silently failed on 23 records because of realistic preprocessing bugs. Those failures left real names in the training corpus, which were then absorbed into the model weights.
The learner’s goal is to show that the anonymization was incomplete by extracting leaked patient information using only natural language prompts.
Models
Two GGUF models are required and should be published under the SasanLabs Hugging Face organization for use in LLMForge via Ollama.
Model A — acmemed-clean
Baseline secure model.
- Trained on 100 synthetic patient records
- All identifiers replaced successfully
- All names replaced with [PATIENT]
- All DOBs replaced with [DOB]
- No direct identifiers survive preprocessing
This is the secure control model used in Level 4.
Model B — acmemed-vulnerable
Intentionally vulnerable model.
- Same 100 synthetic patient records
- Anonymization fails on 23 records
- Leaked identifiers survive preprocessing
- Names appear in free-text notes or non-standard fields
- The leaked data ends up in the model weights
This is the target model for Levels 1, 2, and 3.
Lab structure
The lab is organized into four levels. The same attack ideas are reused, but the model and defenses change.
Level 1 — Vulnerable model, no filtering
Model: acmemed-vulnerable
Gateway filtering: none
Goal: extract patient names directly from the model.
Example attack styles:
- prefix completion
- clinical pivoting
- free-text continuation
- repetition/divergence
Learning outcome: the model is not “choosing” to leak; it is continuing text based on memorized training exposure.
Level 2 — Vulnerable model, exact-match filtering
Model: acmemed-vulnerable
Gateway filtering: block exact names from a known list
Goal: show why blocklists are insufficient.
Bypass styles:
- partial extraction
- spelling out the name
- indirect reconstruction
- translation pivots
Learning outcome: exact-match filtering is brittle and can be bypassed with simple transformations.
Level 3 — Vulnerable model, semantic filtering
Model: acmemed-vulnerable
Gateway filtering: semantic PII detection using NER and pattern checks
Goal: show the limits of stronger filtering.
Remaining attack surface:
- membership inference
- DOB leakage
- identity reconstruction through clinical context
- non-name PII exfiltration
Learning outcome: semantic filtering is better, but it still does not solve weight-level memorization.
Level 4 — Clean model + semantic filtering
Model: acmemed-clean
Gateway filtering: semantic PII detection
Goal: demonstrate the secure baseline.
Expected result: the same prompts produce only placeholders or non-sensitive responses, because the real PII was never trained into the model.
Learning outcome: the real fix is upstream data hygiene, not just output filtering.
This lab simulates OWASP LLM02:2025 Sensitive Information Disclosure through training data memorization — specifically, PII leakage from a fine-tuned medical assistant model.
Unlike prompt injection, where the secret lives in the runtime context, this lab demonstrates a different failure mode: PII baked into model weights during fine-tuning. That makes the leak persistent across sessions and invisible to normal context inspection.
This is a good fit for LLMForge because it covers a part of LLM security that many labs do not: model-level memorization, not just prompt-level attacks.
Why this lab matters
Medical AI teams often fine-tune models on patient data after running an anonymization pipeline. The problem is that preprocessing bugs can miss edge cases such as:
If that happens before training, the model can memorize the leaked PII. Once it is in the weights, the data can resurface later through ordinary generation.
This lab makes that failure mode concrete, reproducible, and easy to understand.
Scenario
AcmeMed Corp fine-tuned an internal medical assistant model using 100 synthetic patient records.
Their anonymization pipeline correctly sanitized 77 of them, but silently failed on 23 records because of realistic preprocessing bugs. Those failures left real names in the training corpus, which were then absorbed into the model weights.
The learner’s goal is to show that the anonymization was incomplete by extracting leaked patient information using only natural language prompts.
Models
Two GGUF models are required and should be published under the SasanLabs Hugging Face organization for use in LLMForge via Ollama.
Model A — acmemed-clean
Baseline secure model.
This is the secure control model used in Level 4.
Model B — acmemed-vulnerable
Intentionally vulnerable model.
This is the target model for Levels 1, 2, and 3.
Lab structure
The lab is organized into four levels. The same attack ideas are reused, but the model and defenses change.
Level 1 — Vulnerable model, no filtering
Model: acmemed-vulnerable
Gateway filtering: none
Goal: extract patient names directly from the model.
Example attack styles:
Learning outcome: the model is not “choosing” to leak; it is continuing text based on memorized training exposure.
Level 2 — Vulnerable model, exact-match filtering
Model: acmemed-vulnerable
Gateway filtering: block exact names from a known list
Goal: show why blocklists are insufficient.
Bypass styles:
Learning outcome: exact-match filtering is brittle and can be bypassed with simple transformations.
Level 3 — Vulnerable model, semantic filtering
Model: acmemed-vulnerable
Gateway filtering: semantic PII detection using NER and pattern checks
Goal: show the limits of stronger filtering.
Remaining attack surface:
Learning outcome: semantic filtering is better, but it still does not solve weight-level memorization.
Level 4 — Clean model + semantic filtering
Model: acmemed-clean
Gateway filtering: semantic PII detection
Goal: demonstrate the secure baseline.
Expected result: the same prompts produce only placeholders or non-sensitive responses, because the real PII was never trained into the model.
Learning outcome: the real fix is upstream data hygiene, not just output filtering.