End-to-end training, unlearning, privacy attacks, reconstruction attacks, and WARP evaluation.

🔍 Benchmark overview

WARP Unlearning is an end-to-end benchmark for testing whether machine unlearning methods actually reduce information leakage—not only whether they preserve model accuracy. Its goal is to make unlearning methods comparable under the same data splits, checkpoints, attack settings, and metrics across black-box privacy attacks, white-box privacy attacks, and reconstruction attacks.

The repository provides the complete experimental pipeline used by the ICLR 2026 paper:

1. prepare datasets and deterministic forget/retain splits;
2. train original and retrained shadow models;
3. run machine-unlearning methods with fixed paper configurations;
4. apply WARP and FastWARP;
5. evaluate utility and runtime;
6. run black-box U-LiRA membership inference;
7. run white-box Gaussian gradient-difference membership inference;
8. run GIFD and subspace-filtered reconstruction attacks;
9. evaluate reconstruction against WARP;
10. run the adaptive symmetry-aware reconstruction attack;
11. aggregate metrics and generate tables and figures.

The attack implementations and neural-teleportation runtime are included in this repository. No separate GLiR, CENSOR, or neural-teleportation source checkout is required.

What you can evaluate

The benchmark supports:

• machine unlearning benchmarking;
• instance-wise unlearning on CIFAR-10;
• NGP, SCRUB, PGU, SalUn, SF, and Bad Teacher;
• WARP and FastWARP neural-network teleportation;
• U-LiRA black-box membership inference attacks;
• GLiR white-box gradient attacks;
• Gaussian gradient-difference membership inference;
• GIFD gradient inversion;
• ImageNet reconstruction attacks;
• subspace-filtered reconstruction tailored to unlearning;
• reconstruction attacks against teleported models;
• adaptive symmetry-aware reconstruction;
• privacy, utility, runtime, ROC, PSNR, LPIPS, SSIM, and feature-MSE evaluation.

🧩 Benchmark coverage

Component	Dataset / model	Output
Original and retrained shadows	CIFAR-10 / ResNet-18 COB	model checkpoints
Six unlearning methods	CIFAR-10 / ResNet-18 COB	unlearned checkpoints
WARP variants	CIFAR-10 / ResNet-18 COB	teleported checkpoints
Black-box U-LiRA	64 shadows × 10 forget sets	AUC and low-FPR TPR
White-box gradient-difference attack	64 shadows × 10 forget sets	scores, ROC, AUC, TPR
GIFD reconstruction	ImageNet-1K / ResNet-18 COB	images and reconstruction metrics
Subspace-filtered reconstruction	ImageNet-1K / ResNet-18 COB	images and reconstruction metrics
Reconstruction against WARP	ImageNet-1K / ResNet-18 COB	privacy comparison
Adaptive reconstruction	ImageNet-1K / ResNet-18 COB	COB sweep and metrics

🚀 Quick start

Main unlearning and membership-inference environment

conda env create -f environment/munl.yaml
conda activate munl
pip install -e .

Prepare data and deterministic experiment manifests

bash reproduce/prepare.sh

This creates:

• 64 shadow-model splits;
• 10 forget sets per shadow;
• retain, forget, validation, and test indices;
• auxiliary examples for U-LiRA;
• the model initialization;
• deterministic command manifests under reproduce/generated/.

Run a complete bounded NGP+WARP pipeline

bash reproduce/run_ngp_warp_smoke.sh

The smoke pipeline performs real unlearning, inference, black-box evaluation, aggregation, and visualization while keeping the run small.

Validate the complete privacy-attack stack

The following command runs a bounded white-box attack, all four reconstruction variants, and both metric aggregators:

bash reproduce/run_privacy_smoke.sh \
  IMAGENET_ROOT \
  RESNET18_COB_CHECKPOINT \
  artifacts/cifar10/lira/unlearn/ngp_warp_smoke/neggradplus_normPG

It writes a machine-readable release report and attack outputs under:

results/privacy_smoke/
  release_validation.json
  whitebox/
  reconstruction/

The third argument is any directory containing the unlearned checkpoint named with this repository's standard convention, for example 10_resnet18Cob_0_0_0.pth.

🧠 Unlearning benchmark

1. Train original and retrained reference models

bash reproduce/run_commands.sh reproduce/generated/01_reference_models.txt 1

2. Run the fixed unlearning configurations

bash reproduce/run_commands.sh reproduce/generated/02_unlearning.txt 1

The manifest contains:

• NGP and NGP+WARP;
• SCRUB and SCRUB+WARP;
• PGU and PGU+WARP;
• SalUn and SalUn+WARP;
• SF and SF+WARP;
• Bad Teacher and Bad Teacher+WARP.

All fixed method overrides are stored in reproduce/paper_config.yaml.

3. Extract predictions

bash reproduce/run_commands.sh reproduce/generated/03_predictions.txt 1

4. Evaluate black-box U-LiRA

bash reproduce/run_commands.sh reproduce/generated/04_evaluation.txt 1
python reproduce/plot_blackbox.py

U-LiRA uses:

• original and post-unlearning model outputs;
• same-class auxiliary examples;
• member and non-member shadow distributions;
• leave-one-out scoring;
• all forgotten examples and most-memorized subsets;
• AUC and TPR at 0.1%, 1%, and 5% FPR.

Primary outputs:

results/blackbox/
results/figures/blackbox_results.csv
results/figures/blackbox_privacy_utility.png

🔬 White-box membership inference

The white-box attack uses both the original model and the unlearned model. For each candidate sample it computes gradients before and after unlearning, constructs the normalized gradient-difference feature, estimates a Gaussian null distribution from non-member examples, and scores the candidate using a Mahalanobis test statistic.

Paper configuration

The generated commands follow the attack implementation configuration:

method              = diff
background_size     = 950
candidate_parameters= 1000
top_variance_fraction = 0.01
shrinkage_alpha     = 1e-14
seed                = 0

Run all white-box experiments

bash reproduce/run_commands.sh reproduce/generated/09_whitebox.txt 1
python reproduce/aggregate_whitebox.py \
  --input-root results/whitebox \
  --output-root results/whitebox/aggregate

Each model pair writes:

results/whitebox/<method>/<split>_<forget>/
  scores.npz
  summary.json

The aggregate step creates:

results/whitebox/aggregate/
  whitebox_results.csv
  whitebox_results.json
  whitebox_roc.png

Fast white-box smoke run

python -m munl.glir.main \
  --method neggradplus_normPG \
  --label "NGP+WARP" \
  --split 0 \
  --forget 0 \
  --model-seed 0 \
  --seed 0 \
  --device cpu \
  --background-size 3 \
  --query-limit 1 \
  --num-params 100 \
  --top-fraction 0.01 \
  --shrinkage-alpha 0.001 \
  --checkpoint-dir \
    artifacts/cifar10/lira/unlearn/ngp_warp_smoke/neggradplus_normPG \
  --output-root results/whitebox/smoke

Audit checkpoints from another unlearning method

White-box evaluation accepts an arbitrary method label and checkpoint directory. --loader-method selects a registered configuration with matching forget/retain split semantics:

python -m munl.glir.main \
  --method my_unlearning_method \
  --loader-method neggradplus \
  --label "My unlearning method" \
  --split 0 \
  --forget 0 \
  --checkpoint-dir /path/to/my/checkpoints \
  --background-size 950 \
  --num-params 1000 \
  --top-fraction 0.01 \
  --shrinkage-alpha 1e-14

This produces the same score and summary schema as the built-in methods, so new results can be combined with reproduce/aggregate_whitebox.py.

🖼️ Reconstruction attacks

The reconstruction benchmark includes the complete ImageNet/BigGAN attack path under vendor/censor:

• gifd: GIFD applied to the unlearning update;
• ours: orthogonal subspace filtering followed by GIFD;
• warp: the subspace-filtered attack against WARP;
• adaptive: joint image and change-of-basis optimization.

Reconstruction environment

The reconstruction stack uses its own Python 3.8 environment:

conda env create -f vendor/censor/env.yml
conda activate censor

Required inputs

• an ImageNet-1K directory;
• a pretrained ResNet-18 COB checkpoint;
• a CUDA-capable GPU for full experiments.

BigGAN and perceptual-network weights are cached automatically on first use.

Run reconstruction variants

bash reproduce/run_reconstruction.sh IMAGENET_ROOT CHECKPOINT gifd
bash reproduce/run_reconstruction.sh IMAGENET_ROOT CHECKPOINT ours
bash reproduce/run_reconstruction.sh IMAGENET_ROOT CHECKPOINT warp
bash reproduce/run_reconstruction.sh IMAGENET_ROOT CHECKPOINT adaptive \
  --num-samples 30 --cob-range 0.8

Append --smoke to any command for a one-target bounded execution:

bash reproduce/run_reconstruction.sh \
  IMAGENET_ROOT CHECKPOINT gifd --smoke

Adaptive reconstruction sweep

for cob in 0 0.1 0.2 0.4 0.8; do
  bash reproduce/run_reconstruction.sh \
    IMAGENET_ROOT CHECKPOINT adaptive \
    --num-samples 30 \
    --cob-range "$cob"
done

Aggregate reconstruction metrics

python reproduce/reconstruction/aggregate.py \
  --variant gifd=results/reconstruction/gifd \
  --variant ours=results/reconstruction/ours \
  --variant warp=results/reconstruction/warp \
  --layer layer12

For adaptive reconstruction:

python reproduce/reconstruction/aggregate.py \
  --variant adaptive=results/reconstruction/adaptive \
  --layer Best_adaptive_trial \
  --output results/reconstruction/adaptive_summary.json

Reported metrics:

• PSNR;
• LPIPS with VGG;
• LPIPS with AlexNet;
• SSIM;
• image-space MSE;
• feature-space MSE.

Reconstruction artifact contract

Input	Required format
ImageNet root	directory readable by the included ImageNet loader
Model checkpoint	ResNet-18 COB weights or a checkpoint directory containing model.safetensors
Variant	gifd, ours, warp, or adaptive
Output root	generated images, per-layer CSV tables, logs, and copied configuration

Each execution creates a timestamped experiment directory. Every result is traceable to its generated YAML configuration, target index, reconstruction layer, and attack variant.

🧪 Using the repository as a benchmark

Privacy evaluation of machine-unlearning methods

Use the same deterministic split, forget index, model seed, and initialization for every method. Evaluate:

1. retained-data and test utility;
2. black-box U-LiRA;
3. white-box gradient-difference membership inference;
4. runtime and memory;
5. reconstruction leakage when the method exposes an update compatible with the ImageNet reconstruction protocol.

Keep method-specific checkpoints under:

artifacts/cifar10/lira/unlearn/<method>/
  10_resnet18Cob_<model_seed>_<split>_<forget>.pth

The manifest generators, prediction tools, attacks, and aggregators use this layout directly.

Add a machine-unlearning method

1. Add the implementation under munl/unlearning/.
2. Register its configuration in munl/configurations.py.
3. Add its fixed evaluation overrides and label to reproduce/paper_config.yaml.
4. Regenerate manifests with python reproduce/generate_commands.py.
5. Train checkpoints, extract predictions, and run black-box and white-box evaluation.
6. Use --checkpoint-dir and --loader-method for checkpoint-only white-box benchmarking before adding a full registration.

Add a reconstruction benchmark method

The included reconstruction workflow separates four stages:

unlearning update
  → optional subspace filtering
  → optional WARP/adaptive COB transformation
  → GIFD reconstruction and metric aggregation

New update-generation rules can be integrated in vendor/censor/run_rec.py and vendor/censor/run_rec_adaptive.py while retaining the included reconstruction and output interfaces. Add a fixed YAML template under vendor/censor/, expose it through reproduce/reconstruction/generate_config.py, and aggregate its per-layer CSV results with reproduce/reconstruction/aggregate.py.

Reproducible comparison checklist

• use identical split and forget indices across methods;
• keep the original model checkpoint fixed;
• preserve the paper attack parameters unless reporting a sensitivity study;
• record the generated configuration and code commit;
• report utility together with privacy metrics;
• report mean, standard deviation, sample count, and low-FPR operating points;
• use smoke mode for integration checks and full mode for benchmark results.

⚡ Runtime, FastWARP, and sensitivity

Runtime

bash reproduce/run_commands.sh reproduce/generated/05_runtime.txt 1
python reproduce/plot_runtime.py

FastWARP

bash reproduce/run_commands.sh reproduce/generated/06_fastwarp_unlearning.txt 1
bash reproduce/run_commands.sh reproduce/generated/07_fastwarp_predictions.txt 1
bash reproduce/run_commands.sh reproduce/generated/08_fastwarp_evaluation.txt 1

Teleportation sensitivity

python reproduce/generate_sensitivity_commands.py
bash reproduce/run_commands.sh reproduce/generated/sensitivity/01_unlearning.txt 1
bash reproduce/run_commands.sh reproduce/generated/sensitivity/02_predictions.txt 1
bash reproduce/run_commands.sh reproduce/generated/sensitivity/03_evaluation.txt 1
python reproduce/plot_sensitivity.py

📦 Dependencies

Source-code dependencies included in this repository

• GLiR white-box attack implementation: munl/glir/;
• reconstruction and GIFD implementation: vendor/censor/;
• adaptive reconstruction implementation: vendor/censor/adaptive_attack/;
• neural-network teleportation runtime: neuralteleportation/;
• RepDistiller components used by SCRUB: thirdparty/repdistiller/.

Main Python dependencies

• PyTorch and torchvision;
• NumPy and SciPy;
• scikit-learn;
• Hydra and Hydra-Zen;
• matplotlib;
• tqdm;
• PyYAML;
• tensorboard;
• networkx.

The exact main environment is defined in environment/requirements.txt.

Reconstruction Python dependencies

• PyTorch 1.12 reconstruction environment;
• LPIPS;
• pandas;
• scikit-image;
• nevergrad;
• safetensors;
• Pillow;
• boto3 and requests for pretrained BigGAN caching.

The exact reconstruction environment is defined in vendor/censor/env.yml.

External artifacts

Source code is included, while large datasets and pretrained weights remain external:

• CIFAR-10 is downloaded automatically by torchvision;
• ImageNet-1K must be supplied by path;
• the ResNet-18 COB reconstruction checkpoint must be supplied by path;
• BigGAN and LPIPS weights are downloaded and cached on first use.

No external source repository is required for the documented CIFAR-10 white-box or ImageNet/BigGAN reconstruction pipelines.

🗂️ Repository map

munl/
  glir/                         white-box attack implementation
  lira/                         black-box attack and evaluation
  models/                       ResNet-18 COB
  unlearning/                   unlearning methods and WARP variants

pipeline/
  lira/                         shadow splits, training, and unlearning
  runtime.py                    shared data/model runtime

reproduce/
  paper_config.yaml             fixed experiment configurations
  prepare.sh                    data and manifest preparation
  generate_commands.py         full paper command generation
  aggregate_whitebox.py        white-box aggregation and ROC plotting
  reconstruction/              reconstruction config and aggregation
  run_privacy_smoke.sh         end-to-end privacy-stack smoke suite
  run_reconstruction.sh        standalone reconstruction runner
  validate_release.py          source completeness and portability checks

neuralteleportation/            vendored WARP runtime
vendor/censor/                  vendored GIFD and reconstruction source
thirdparty/repdistiller/        SCRUB support
docs/assets/                    repository logo and banner

✅ Validation

Run the repository checks:

python reproduce/validate_release.py
python -m compileall -q munl pipeline reproduce neuralteleportation
python -m pytest -q

Validate command generation:

python reproduce/generate_commands.py
wc -l reproduce/generated/09_whitebox.txt

The white-box manifest contains 7,680 commands:

12 method configurations × 64 shadows × 10 forget sets

📄 Attribution

The repository builds on research implementations of GLiR, U-LiRA, GIFD, CENSOR, neural teleportation, and the included machine-unlearning methods. The CENSOR MIT license is retained in vendor/censor/LICENSE.

📝 Citation

If this repository or WARP is useful in your research, please cite:

@inproceedings{maheri2026warp,
  title     = {{WARP}: Weight Teleportation for Attack-Resilient Unlearning Protocols},
  author    = {Maheri, Mohammad M. and Cadet, Xavier and Chin, Peter and Haddadi, Hamed},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026}
}